Prepared by: Creative Associates International, Inc. Donaldo Hart, Ph.D. Patricia Craig, Ph.D. Daniel A. Morales-Gómez, Ph.D. Prepared for: LAC/RSD-EHR US Agency for International Development May 2001 (DRAFT) The Basic Education and Policy Support (BEPS) Activity Contract No. HNE-I-00-00- 00038-00 Task Order No. 4 Improved Human Resource Policies Evaluation of the Programa de Promoción de la Reforma Educativa en America Latina (PREAL) The Basic Education and Policy Support (BEPS) Activity Contract No. HNE-I-00-00-00038-00 Task Order No. 4 Improved Human Resource Policies Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity ii ACKNOWLEDGMENTS The Evaluation Team is indebted to many parties for their contributions to this research. First and foremost are the numerous interviewees in each country who gave their time. Creative Associates International, Inc., in collaboration with George Washington University and GroundWork, provided important support. In particular, the Team would like to thank Don Graybill and Antonieta Harwood for their hard work and guidance, and Cynthia Prather for her patient review and editing of the document. USAID, especially the Global Bureau and LAC/RSD-EHR, were flexible to the needs of the Team, responsive to its questions, and generous in their support. Our thanks go to David Evans, Susan Morawetz, Cynthia Chassy, and Rebecca Adams of LAC/RSD. Special thanks go to Marcela Gajardo, Carleton Corrales, and Patricia Arregi and their colleagues for their participation in the preparatory work and for their assistance and kindness to the consultants in their countries. At the IAD we especially thank Jeffrey Puryear and Francesca Bosco for assisting in the preparation, providing documents, and responding fully and clearly to all the Team’s questions. We sincerely hope that this report will prove useful to all parties. Donaldo Hart, Ph.D. Daniel Morales-Gómez, Ph.D. Patricia Craig, Ph.D. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity iii CONTENTS Acknowledgments............................................................................................................. ii Contents........................................................................................................................... iii Executive Summary.......................................................................................................... iv I. INTRODUCTION ..................................................................................................... 1 I.1. Background ............................................................................................................ 2 I.2. Synthesis of Findings and Recommendations........................................................... 6 I.3. Future Considerations ........................................................................................... 23 I.4. Methodology......................................................................................................... 24 II. COUNTRY REPORTS .......................................................................................... 27 Overview......................................................................................................................... 28 II.1. Chile .................................................................................................................... 29 II.2. Honduras......................................................................................................... 47 II.3. Nicaragua ........................................................................................................ 57 II.4. Peru................................................................................................................. 69 APPENDICES................................................................................................................... 81 Appendix A. Final Work Plan........................................................................................ 82 Appendix B. Survey Instrument...................................................................................... 96 Appendix C. Survey Results. ........................................................................................ 103 Appendix D. List of Interviewees ................................................................................. 106 Appendix E. Bibliography............................................................................................. 112 Appendix F. PREAL Associated Centers...................................................................... 115 Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity iv EXECUTIVE SUMMARY The Programa de Promoción de la Reforma Educativa en America Latina (PREAL) (translated as Partnership for Educational Revitalization in the Americas) is a regional program working with a network of public and private organizations in Latin America that share a common interest in promoting educational reform in the region. Created in 1995, PREAL is jointly managed by the Inter-American Dialogue (the IAD) in Washington, DC and the Corporación de Investigaciones para el Desarollo (CINDE), in Santiago, Chile. PREAL collaborates with a broad range of public and private sector stakeholders, including governments, NGOs, and international organizations. Through its headquarters office in Washington, DC and the regional office in Santiago, PREAL collaborates with 22 Associated Centers in 14 Latin American countries. The program seeks to improve the quality and equity of education by promoting and encouraging the implementation of better education policies. Mechanisms used to carry out this task include regional Working Groups, conferences and workshops, publications, and electronic communications networks, including internet websites. In 1996, the United States Agency for International Development (USAID) recognized PREAL’s potential contribution to the sustainable development goal and objectives of the Latin America and Caribbean (LAC) Bureau. PREAL was identified as a means by which to achieve the Bureau’s Strategic Objective (SO) #2, “Improved human resource policies adopted in selected LAC countries.” In this light, a five-year cooperative agreement to support PREAL’s various activities was approved. As the five-year period comes to an end, USAID invited Creative Associates International, Inc. (CAII), through its Basic Education and Support (BEPS) Activity, to evaluate the achievements of PREAL during its funding cycle. This report presents the results of that evaluation. Purpose, Methodology, and Scope of the Evaluation The purpose of the evaluation was to address a series of questions posed by USAID in the BEPS Statement of Work. The questions were ranked by USAID in three categories of priority: primary, secondary, and tertiary (see Appendix A, Final Work Plan). The primary questions, which correspond directly to the intermediate results stipulated in the cooperative agreement, were as follows: ß Did PREAL succeed in building public and private sector support for educational reform? ß Did PREAL strengthen the capacity of public and private sector organizations working for educational improvement? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity v ß Did PREAL adequately identify and disseminate best educational policies and procedures? Under the guidance of the BEPS Project Director and the LAC Task Order Activity Coordinator at CAII, three senior consultants carried out research in four selected countries: Chile, Honduras, Nicaragua, and Peru. The country reports presented in Section II reflect both the differences in the ways in which PREAL operates in each country and the general lessons that can be drawn and could be applicable to PREAL as a whole. All fieldwork took place between November 28 and December 19, 2000. The approach to the design and implementation of the evaluation stressed stakeholder participation in the evaluation process and transparency in the manner in which information was handled. Representatives of USAID/LAC/RSD-EHR, the IAD, PREAL staff from the field, and the BEPS team all worked with the Evaluation Team to establish the parameters of the research and to develop the survey instrument. The research methodology included direct personal inquiry and document review. The direct personal inquiry involved one-on-one interviews using a questionnaire that combined closed-end and qualitative questions (see Appendix B for the survey instrument). A combined total of 70 people responded to the survey instrument. Respondents represented a cross-section of public and private stakeholders in each country. The samples in each country were purposively selected and included a large majority of individuals who had participated in one or more events sponsored by PREAL (See Appendix C, Survey Results, and Appendix D, List of Interviewees). The Evaluation Team also held unstructured interviews with several people in the field as well as with USAID and the IAD in Washington. The document review involved a review of a sample of the PREAL publications that were made available to the consultants and relevant materials from USAID/LAC/RSD-EHR (see Appendix E, Bibliography). Synthesis of Findings and Recommendations The findings and recommendations presented in this synthesis reflect those general in nature that emerged from the observations and information gathered in each of the four countries visited. The intention is to present an overview of the main findings and recommendations that are relevant to each of the four countries. Each country report identifies specific findings and recommendations that are primarily relevant to the specific country context, particularly in the case of Honduras and Nicaragua, and to a lesser extent Peru. Two of the country reports (i.e., Chile and Peru) also include findings and recommendations of a general nature that could be considered relevant to PREAL as a whole. This is because an important part of PREAL’s operations is managed from its office in Chile and because Peru is the base both for one of the Working Groups and for one of the members of the Executive Committee. Nevertheless, Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity vi only by inference can the findings and recommendations presented in this report be applied to other countries in which PREAL is active. To set the more specific findings in context, the reader should first consider the following: ß PREAL’s success and impact cannot be measured linearly. The nature of the process of education reform is both complex and long term. In the context of the region, it is unlikely that any single institution, program, or donor could single-handedly have a direct impact on education reforms. ß PREAL’s success and impact are related to its evolution, relationships with donor partners, and the nature of the reform processes within national contexts. The achievements of the Program to date are the result of initiatives that may fall outside the scope of a single donor agreement. Findings 1. Overall, PREAL achieved in all four countries studied the three intermediate results stipulated by USAID: a) building public and private sector support for educational reform; b) strengthening public and private sector organizations working for educational improvement; and c) identifying and disseminating best education policies and practices. 2. PREAL achieved its principal objectives in each of the four countries studied. These accomplishments, however, varied considerably from one country to another. 3. To a large extent, much of the success of PREAL is attributed to the role of key individuals. This reflects the ability of PREAL to identify and select key people that can effectively contribute to meeting PREAL objectives. It also, however, may affect the perception from the field of PREAL as a consolidated network of institutions in the region. 4. The IAD has monitored inputs and outputs appropriately, but needs to monitor outcomes (e.g., policy change) more closely. 5. The complexity of the problem area that PREAL addresses, the broad scope of its agenda and mission, and the resources available to the Program raise concerns about the risk of PREAL exceeding its capacity to manage and address effectively its current program plan. 6. Under present funding and structure, neither PREAL nor the activities it supports are financially sustainable. 7. There is evidence of changes related to education reform policy that are linked indirectly to PREAL’s activities and role. Although research evidence shows no Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity vii official change in educational policy directly attributable to PREAL’s activities in any of the four countries during the period of the cooperative agreement, evidence shows that PREAL has influenced the adoption, modification, and implementation of educational procedures and practices in PREAL-serviced countries. Additional Findings In addition to these general findings, because an important part of PREAL’s operations is managed from its regional office in Chile, the Chile Country Report includes the following general findings that could be considered relevant to PREAL as a whole: 1. There is unanimous consensus that PREAL, since its origin, has made an important and relevant contribution to helping to identify education reform priorities. 2. PREAL publications and events are perceived to be highly effective and of high technical quality. 3. Where access is available, the PREAL internet site and those maintained by some of the Associated Centers are important and useful information tools. 4. PREAL’s approach has been appropriate to its current objectives under the existing conditions for education reform in the region. 5. There is a perception among those interviewed—a minority perception but nevertheless an important one—that PREAL needs to further clarify its role, both in relation to its donor partners as well as to its sponsoring and partner institutions. 6. Prospectively, the success of PREAL is a source of both risks and opportunities. PREAL’s success is likely to add pressure to the Program to expand its scope and be even more responsive to external demands despite its limited resources. At the same time, PREAL’s reputation to date represents a window of opportunity to continue influencing education reform processes in the region. Recommendations Based on the above findings and other information from the research conducted in each of the countries visited by the Evaluation Team, the report presents five major recommendations: 1. Within the context of a given assistance package, define PREAL’s responsibility and parameters of action more narrowly. 2. Within the parameters of a USAID grant with a new and narrower mandate, ask PREAL to revisit and update its vision and a supporting strategy, as these are related to the cooperation with USAID. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity viii 3. Further develop a realistic but demanding performance monitoring plan. 4. Consider shifting some attention and responsibilities from the Associated Centers to the Working Groups and including Working Group representatives in the Executive Committee. 5. Resolve the perceived misunderstandings between the IAD and its client, USAID. The report shows that in the four countries studied, PREAL achieved remarkable success in reaching the objectives stipulated in its cooperative agreement with USAID. The report also shows that at various levels, PREAL needs to take stock of the achievements to date and further strengthen its strategic planning process. PREAL now needs to channel its comparative advantages—excellent networks of senior people, good research and distribution, and the ability to create a vision—to be even more effective in translating policy reform into educational practice—to see, in effect, that its own major recommendations become reality. Finally, PREAL, or the IAD, should not proceed alone in deliberating over future directions and relationships with USAID. There are great potential advantages in the USAID-PREAL collaboration to systematically seek advice from experts and institutions in the region as a means of enhancing their respective potential of impact. USAID has found in PREAL a solid, performing partner for its regional interests. Senior personnel from both organizations should work together to develop new strategies in order to optimize each partner’s resources and achieve the desired results. Future Considerations Future planning should include a discussion of the whole issue of a regional approach to policy reform, as well as to issues of national or sub-regional focus. The report suggests several questions that the Bureau should ponder in order to best resolve these matters: ß Is the Bureau prepared to invest in three more years of similar activities, with very little assurance that policy reform will take place? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity ix ß Is the Bureau willing to continue investment in countries where USAID has neither presence nor program, or even where USAID has presence but no educational strategic objective? ß In countries with bilateral educational programs, how does LAC/RSD/EHR intend to work with the Missions’ Strategic Objectives? ß Would the Bureau consider a more specific, thematic approach to policy reform— one that might prioritize one policy over another, or one set of actions over others? ß Would the Bureau consider focusing on a sub-region, even were such focus to reduce or eliminate support for several countries of historic importance? ß Does the Bureau want greater or less direct involvement in PREAL? Assuming that a future contract of a similar nature would be inappropriate, would it consider other more focussed mechanisms of funding? I. INTRODUCTION Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 2 I.1 BACKGROUND The Context The Programa de Promoción de la Reforma Educativa en America Latina (PREAL) (translated as Partnership for Educational Revitalization in the Americas) is a regional program working with a network of public and private organizations in Latin America that share a common interest in promoting educational reform in the region. Created in 1995, PREAL is jointly managed by the Inter-American Dialogue (the IAD) in Washington, DC and the Corporación de Investigaciones para el Desarollo (CINDE), in Santiago, Chile. PREAL collaborates with a broad range of public and private sector stakeholders, including governments, NGOs, and international organizations. Through its headquarters office in Washington, DC and the regional office in Santiago, PREAL collaborates with 22 Associated Centers in 14 Latin American countries. The program seeks to improve the quality and equity of education by promoting and encouraging the implementation of better education policies. Mechanisms used to carry out this task include regional Working Groups, conferences and workshops, publications, and electronic communications networks, including internet websites. Recognizing its potential contribution to the sustainable development goal and objectives of the Latin America and Caribbean (LAC) Bureau, the United States Agency for International Development (USAID) identified PREAL as a means by which to achieve the Bureau’s Strategic Objective (SO) #2, “Improved human resource policies adopted in selected LAC countries.” In this light, a five-year cooperative agreement to support PREAL’s various activities was approved. As the five-year period comes to an end, USAID invited Creative Associates International, Inc. (CAII), through its Basic Education and Support (BEPS) Activity, to evaluate the achievements of PREAL during its funding cycle. This report presents the results of that evaluation. The Evaluation The evaluation of the PREAL Program was carried out between November 22, 2000 and January 15, 2001. This included fieldwork in the four selected countries: Chile, Honduras, Nicaragua, and Peru. The work was coordinated and supervised by CAII. The senior staff member responsible for the BEPS Activity is Dr. Donald Graybill. The Activity Coordinator for the PREAL evaluation was Antonieta Harwood. A team of three consultants carried out the research and fieldwork, and produced the evaluation report. Dr. Donaldo Hart was the Team Leader for the activity and conducted fieldwork in Honduras. Dr. Patricia Craig was responsible for Nicaragua, and Dr. Daniel Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 3 Morales-Gómez covered Chile and Peru. The fieldwork took place between November 28 and December 19, 2000. Scope of the Assignment and Issues to Be Addressed The purpose of the evaluation was to address a series of questions posed by USAID in the BEPS Statement of Work. CAII, jointly with USAID/LAC/RSD/EHR and the IAD, agreed upon a preliminary work plan and carried out the selection of the countries where the fieldwork was to be conducted. The Evaluation Team reviewed and discussed this work plan with CAII. The work plan set the framework for the evaluation around a set of questions grouped into three priority categories, listed below. The primary questions incorporate the IAD’s intermediate results in its cooperative agreement with USAID. The secondary and tertiary questions contributed to complement the main focus of the evaluation. Primary Questions ß Did PREAL succeed in building public and private sector support for educational reform? ß Did PREAL strengthen the capacity of public and private sector organizations working for educational improvement? ß Did PREAL adequately identify and disseminate best educational policies and procedures? Secondary Question ß Has PREAL contributed to improved quality and equity of education by promoting regional dialogue and encouraging policy implementation? If so, how? Tertiary Questions ß Which of PREAL’s policy reform strategies and/or activities have proven to be the most effective? ß Who is using PREAL’s recommendations or findings? ß Has the structure of PREAL proven to be an advantage or a hindrance to PREAL’s accomplishing its mission successfully? Why? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 4 ß Is there evidence of integration of purposes? That is, has PREAL made efforts to unify or coordinate the mandates of its various funders? ß Is the concept of sustainability pertinent to this activity? If so, how is it being encouraged and is it being achieved? Approach to the Evaluation LAC/RSD-EHR stressed from the outset that as much as possible, all major stakeholders involved in the evaluation should have input into both the country selection and the design of the research. Accordingly, the Team Leader for the evaluation met with the staff at CAII and with LAC/RSD during the planning stage of the research design. He also held a two-hour meeting with representatives from the IAD to review the proposed scope and focus of the evaluation. The Evaluation Team worked over three days at Creative Associates to finalize the design, prepare the research guide, and plan the fieldwork. Much of the first day of discussions included representatives from USAID, the IAD, PREAL-CINDE (of Santiago, Chile), and the PREAL office in Tegucigalpa. Representatives from two of the BEPS subcontractors (e.g., the George Washington University and GroundWork) also were present. The representatives from PREAL’s offices in Chile and Honduras also participated in the preparatory work for most of the three days. Creative Associates facilitated these working sessions with full transparency of purpose and methodology. The research methodology agreed upon during these planning sessions is presented in Chapter I.4. Organization of the Research Synthesis The chapter, “Synthesis of Findings and Recommendations,” presents the major general lessons that, in the view of the Evaluation Team, apply to each and all of the countries studied. The first finding addresses in depth the primary research questions. It offers an overall response to the main question underlying this summative evaluation: Did PREAL achieve the objectives USAID gave it funding to accomplish? Other findings, however, do not follow in the order of the research questions. Two reasons are offered. First, since this evaluation is a composite of four case studies—with four separate survey populations—specific questions are best answered within the context of the country reports. Each country report thus attempts to answer each of the research questions in sequence. Second, in the synthesis chapter, the Evaluation Team made an effort to identify major findings that apply to the four countries. Broader lessons that did not necessarily follow the primary-research-question sequence were extracted from such major findings. The recommendations emerge from the findings, but they are not simply responses to each finding. The Evaluation Team attempted to shape them into a narrative to facilitate further Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 5 relevant discussions and application to the LAC/RSD-EHR strategic planning effort. It was the purpose of the Evaluation Team, as of CAII, to provide a report that is accurate in its data, easy to read, and useful for decision makers in USAID and at the IAD. The reader is reminded that much of the present research has of necessity been qualitative in nature. In formulating their findings and conclusions, the Team stayed faithful to the data. Nonetheless, they alone are responsible for interpretations, findings, and opinions expressed in this report as well as for any inaccuracies that may surface in its review. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 6 I.2. SYNTHESIS OF FINDINGS AND RECOMMENDATIONS This section presents the principal findings and recommendations of the research effort. In most instances, a finding is relevant to all four countries studied; in those instances where it is not, the differences have been noted. An effort was made not to create redundancies by repeating information found in the country reports, which serve as the basis for the findings. The user of the report should bear in mind three important considerations. First, the findings and the subsequent recommendations represent a best judgment call. The findings are subject to varying interpretations. Second, special attention should be given to the fact that the aggregation of quantitative data from the four countries is presented for illustration purposes only, since the seventy respondents to the survey do not represent a single population. Third—and perhaps most important—is the fact that just because a finding or recommendation may be valid for any or all the four countries does not necessarily imply that it is valid for other countries in the region. In addition, to set the more specific findings in context, the reader should consider the following: ß PREAL’s success and impact cannot be measured linearly. The nature of the process of education reform is both complex and long term. In the context of the region, it is unlikely that any single institution, program, or donor could, single-handedly have direct impact on education reforms. ß PREAL’s success and impact are related to its evolution, relationships with donor partners, and the nature of the reform processes within national contexts. The achievements of the Program to date are the result of initiatives that may fall outside the scope of a single donor agreement. Principal Findings Finding 1: Overall, PREAL achieved in all four countries studied the three intermediate results stipulated by USAID: a) building public and private sector support for educational reform; b) strengthening public and private sector organizations working for educational improvement; and c) identifying and disseminating best education policies and practices. The first of these objectives is to build public and private sector support for educational reform. Through pursuit of this objective, PREAL would build broad constituencies and raise the issue of educational reform to the top of the region’s policy agenda. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 7 The Evaluation Team established the following indicators for gauging achievement of this objective: ß Debate created at national level ß Widespread engagement of social sectors evidenced ß Awareness of issues and priorities witnessed among major stakeholder groups ß Collaborative actions undertaken by stakeholder groups ß PREAL’s ability to convene stakeholders established ß Public gatherings held periodically ß Sub-regional task forces operative and productive All of these indicators were fully or largely met in each of the four countries, as stated in the country reports. Both quantitative and qualitative information from the survey offer strong additional evidence that PREAL succeeded in building support by raising the debate about educational reform in the countries studied. Of the 62 respondents who participated in one or more events sponsored by PREAL, 57 gave examples of how these events had promoted discussion of the issues, and 44 (63 percent of total respondents) provided examples that the events had impact on educational practice or policy. Forty-five respondents said that PREAL’s publications have contributed to actions promoting the development of educational policy. Positive responses from Nicaragua were lower than the average but still indicative of impact on building constituencies for educational reform. Other country evidence shows that PREAL succeeded in building support in the public and private sectors and in elevating the agenda of educational reform. Thus, for example, even though PREAL does not carry out country-level activities in Chile, this country offers the region’s best example of educational reform, and indirectly of the impact of PREAL in building regional level support for reforms. In Honduras, however, the survey clearly shows the impact of PREAL on the country education policy changes. Through its work in the commissions and conceptual teams—acknowledged in the documents themselves—PREAL had a major role in shaping the two formal proposals for educational reform that the evaluator reviewed. Based on evidence such as this, it could be argued that PREAL fully achieved its purpose in this objective in the countries studied. The second of PREAL’s intermediate objectives is to strengthen the capacity of public and private sector organizations to work for educational improvement. Through PREAL, the IAD would identify leading institutions and individuals and create partnerships and collaboration. The Evaluation Team approached this objective from two angles. The first was in terms of the Associated Centers—organizations with formal links and in some cases funding from PREAL. The second was in terms of non-associated, stakeholder institutions with which PREAL frequently collaborated. The indicators were as follows: Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 8 ß Associated Centers founded ß Associated Centers established through linkages ß Associated Centers that carried out or promoted PREAL strategies ß Stakeholder organizations aware of PREAL’s major reform objectives ß Stakeholder organizations that received and used PREAL publications Among the four countries, only in Nicaragua was there an Associated Center actually created with PREAL funds, as indicated in the relevant chapter. In Peru and Chile each, there are two Associated Centers, while there is one in Honduras. These Associated Centers are not-for-profit entities that had institutional existence before they joined the PREAL network. Information available indicated that PREAL also has Associated Centers in Brazil, the Dominican Republic, Venezuela, Guatemala, Costa Rica, and El Salvador. In the case of private Associated Centers, there are indications that PREAL strengthened their capacity to advance the cause of educational reform. To varying degrees PREAL also strengthened their overall organizational capacities, as shown by the case studies. The Associated Centers have carried out and contributed to the promotion of PREAL’s strategies. As the country case studies show, the history of Associated Centers is very uneven. Nevertheless, it is beyond question that to varying degrees these Centers have carried out or promoted the PREAL regional agenda. On the side of non-associated stakeholder organizations, the survey categorized four general types: public (28); private for-profit (19); private non-profit (16); and donor (7). During the design stage, a checklist of twelve types of stakeholder organizations was compiled from where the respondents were selected (see Chapter I.4, Methodology), broadening the perspective from otherwise small and narrow samples. As indicated above, high percentages of respondents in each of the countries could discuss the seminar themes, the seminars’ impact on the respondents’ ways of thinking, and their influence on their organizations. The case is similar—with even higher percentages—for publications. Interviewees from ministries and universities, from unions and business associations, from the media and research institutes received, learned from and indicated applying lessons from PREAL’s published materials. Among non-associated organizations, whether public or private, PREAL has strengthened the capacity to promote reform. Educational reform is far more likely to be a direct cause for governments than for private business. There is, however, growing interest among the private sector in education reform, and in many cases their interests and actions overlap. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 9 PREAL’s impact seems to have been greater in the public sector, as it is this sector, (namely the government sector) that enacts policy reform. At the same time, PREAL has been effective in reaching the private sector, and in various cases, key individuals from private organizations working with PREAL have moved to occupy public sector positions in capacities that influence policy. The third of PREAL’s intermediate objectives is to identify and disseminate best education policies and practices in the region. PREAL worked to accomplish this objective by organizing seminars and conferences, distributing publications, and making information available through various websites on the Internet. Indicators for assessing achievement of this objective were: ß Research conducted broadly ß Selection criteria for determining best practices identified ß Documents received regularly by major stakeholders ß Materials designed and packaged effectively ß Published materials of highly professional quality standards ß Website accessible and used Discussions held in Chile and with the IAD in Washington show that research has been broadly conducted and that the process for selecting best practices is proving effective. In Washington, the Evaluation Team examined a selection of the documents produced under PREAL funding and found them to be of exceptional quality in design and production. Materials also appear to be reaching virtually all major stakeholder organizations. An examination of distribution lists found them sufficiently comprehensive. In the sample, 64 of 70 respondents identified by title or by theme one or more of PREAL’s publications. The perceived usefulness of the publications is also noteworthy. Of the 64 recipients of materials, 62 gave examples of what they had learned from the materials, and an almost equal number affirmed by way of example that their organizations used this information. Furthermore, 60 of the 64 who received materials from PREAL also cited materials from other organizations, and 53 of these felt that PREAL’s publications were more useful, considering them more up-to-date or more practical or both. The websites offer more mixed results. In terms of use, Honduras had only one user and Nicaragua two. Thus, of the 19 respondents who had ever visited the site, 16 were from Peru and Chile. This is not an indictment of the websites themselves, however. Rather, it is an indication of how PREAL’s achievements can vary by country, or by sub-region. The PREAL website, which is maintained by the PREAL main site, has received over 18,000 hits. As the Chile reports shows, this site is well constructed and considered highly useful by a select number of researchers who have ready access to the Internet. The differences found in the use of PREAL websites notwithstanding, there is indication that PREAL achieved remarkably well this intermediate objective of disseminating information. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 10 If this evaluation were simply a performance monitoring assessment, the reporting of the findings could end here with the conclusion that the IAD achieved its three objectives in the four countries studied. This implies that at least some of PREAL’s strategies have been effective. This matter will be elaborated further under Finding 5 below. Finding 2: PREAL achieved its principal objectives in each of the four countries studied. These accomplishments, however, vary considerably from one country to another. At first glance, this finding may seem banal. Chile is different from Peru, which is most unlike Honduras, which differs greatly from Nicaragua. None of them represents, for example, the society or politics of Venezuela, much less those of Brazil or Argentina or Haiti. What does this mean for this evaluation? For one thing, areas of uniform achievement, such as the perceived usefulness of the publications, stand out more clearly. For another, the finding raises issues of strategy, management, financial sustainability, and performance monitoring. The impact of PREAL’s recommendations from the Task Forces, for example, has been different from country to country. The Central American Task Force report, Mañana Es Muy Tarde, stirred attention and galvanized serious interest throughout Honduras. In Nicaragua, the recommendations were received with suspicion by some significant sectors. The recommendations of the Task Force report, The Future at Stake, had an impact in cases like Chile, which has long embraced them and to varying degrees acted upon them. In Peru, however, they were not embraced, largely because both government and civil society faced concerns of far greater urgency. A lesson to draw is that the impact of Task Force recommendations is the result of a combination of factors, including local receptivity, perceived quality and pertinence of the recommendations, and follow-up actions of key individuals collaborating with PREAL, among others. The Associated Centers, the structures through which PREAL channels its interventions in the countries, vary greatly in their effectiveness, closeness, the loyalty they feel towards PREAL, and their prospects for financial sustainability. In Chile, CIDE is an independent NGO that has led Working Group activities on behalf of PREAL, while PARTICIPA’s involvement has been limited to the exchange of information and the organization of a seminar in 1997. In Peru, PREAL has located one of the Working Groups in GRADE, but it has unresolved issues in consolidating PREAL partnership with the Foro Educativo. FEREMA in Honduras enjoys active collaboration with PREAL and will almost certainly continue as a loyal, effective partner while funding for activities continues. As the case for Nicaragua reveals, the two Associated Centers, UCA-PREAL and EDUQUEMOS, differ greatly in approach, philosophy, and even loyalty. The differences observed among the Associated Centers reflect to a large extent the nature of the association they maintain with PREAL as part of a broad regional network of institutions. They are not intended to be franchises or program offices of PREAL but members of a loose federation. There is, thus, no single way to describe a “typical” Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 11 Associated Center of PREAL. At the same time, there is not a clear way to unify managing the relationships with them. The issue, however, is one of managing diversity on the basis of a commonly agreed upon operational framework, which includes as appropriate, but goes beyond, sub-grant contract requirements. PREAL’s image, or “brand,” varies in clarity and by theme. In Peru, PREAL is identified primarily with the Working Group on Evaluation and Standards. In Chile, PREAL is identified primarily with the regional Task Force and the fact that the secretarial office is in Santiago. PREAL in Honduras is identified with an individual who works on all fronts in behalf of sweeping, systemic educational reform. In Nicaragua, PREAL is the “ally” or the “enemy,” depending on one’s political affiliation. Based on the diverse experiences of the countries visited, the varying accomplishments of PREAL seem to be less one of degree than of nature, except in the instance of the web pages. From the perspective of each country context, questions such as the following may have different answers: “What is PREAL?” “What is its strategy?” “What exactly has it accomplished?” and “What is its next step?” In part, this may respond to the characteristics of the specific country settings where PREAL operates. It also may be a reflection of PREAL’s own understanding of what constitutes a regional program operating through a loose network of institutions. In some important ways, PREAL could gain from further strengthening its identity. As discussed in Finding 5 below, PREAL has attempted so much in so many different ways, that it is stretched thin. Its purpose and its modus operandi risk being perceived differently in different countries, and even at times fragmented, making it difficult to be considered a regional program. Finding 3: To a large extent, much of the success of PREAL is attributed to the role of key individuals. This reflects the ability of PREAL to identify and select key people that can effectively contribute to meeting PREAL objectives. It also, however, may affect the perception from the field of PREAL as a consolidated network of institutions in the region. In most—but not all—instances according to this evaluation, the IAD has made judicious choices in selecting PREAL staff, who have served PREAL well. In each of the countries visited, however, there was a strong tendency to perceive PREAL as closely related to the individuals who represent PREAL. This may contribute to the differences in perceptions about the PREAL identity itself (i.e., Is PREAL a regional program? Is it a regional network of institutions? Is it an institution? Is it a “donor”? Is it a combination of these?) This finding raises important questions for consideration and further discussion by the Coordinators of PREAL in its future strategic planning process. Should PREAL, which after all is a program, not a legal entity, become institutionalized? Has the IAD any strategies for selecting and retaining especially effective persons? Are there contingency plans to fill human resource gaps? If PREAL is entrusting its operations in the field to Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 12 selected individuals and or institutions (as in the case of the Associated Centers), should the IAD not bring them together with some regular frequency for greater coordination and planning? Should it not impose some standards of performance and verifiable benchmarks of progress as a means to deal with the diversity of the network? Should it have general performance goals as criteria for participation/affiliation with the program? Finding 4: The IAD has monitored inputs and outputs appropriately, but needs to monitor outcomes (e.g., policy change) more closely. Although this evaluation determined that the three intermediate objectives proposed by the IAD to USAID were objectively verifiable as performance benchmarks in the progression towards PREAL’s overall goal, it is not clear that most people working for or with PREAL saw them from such a perspective. This has implications, both for the ways in which PREAL monitors the achievement of these intermediate objectives and the impact of the products produced. In the USAID proposal itself, greater emphasis was placed on the activities to be implemented than on the objectives themselves. PREAL seems to have been put in a position to respond to this demand and became more of a task-focused program, despite the difficulties in “quantifying” the type of results produced by PREAL activities. Thus, for example, a review of the IAD’s early quarterly reports to USAID revealed a vigorous and productive program at work. Through these reports, the IAD provided good monitoring of its inputs (largely financial) and its outputs (the events and publications). An early internal evaluation,1 while not destitute of substantive observations, followed largely the stream of tasks and perhaps inadvertently contributed further to this focus by emphasizing the volume of activities implemented over results achieved. In its proposal, the IAD recognized the importance of monitoring progress, but probably understandably, it seems to have discounted measures of process as “difficult to quantify.” Most of the IAD’s discussion of monitoring in the proposal focused on indicators of final outcome, e.g., improvements in the regional aspects of quality and equity. Then it effectively discounted process monitoring by noting that such improvements may take many years and therefore lie beyond the contractual scope of the agreement. The IAD was right in not assuming performance responsibility for actual policy change. Further steps, however, could have been explored jointly between the IAD and USAID to find more effective ways to monitor progress more directly.2 Finding 5: The complexity of the problem area that PREAL addresses, the broad scope of its agenda and mission, and the resources available to the Program raise concerns 1 Evaluación del Proyecto PREAL, 1997. 2 USAID did establish a set of indicators for the PREAL activity, and the Evaluation Team reviewed them carefully. The Bureau was never wholly satisfied with them and received useful commentary on them. One of the problems with the indicators as a whole was that target and goal-level indicators shared the same space, creating confusion about the grantee’s responsibilities. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 13 about the risk of PREAL exceeding its capacity to manage and address effectively its current program plan. It is not simply a matter of the complexities having grown or the new dimensions of PREAL’s ambitions. Rather, evidence seems to indicate that the complexity of the issues of education reform, the multiple demands from across the region, and PREAL’s own success in addressing some of these issues add new demands on the resources available. To carry out its agenda over the longer term, PREAL lacks the resources—financial and human—to tackle the issues on which it has decided to focus. This is as true in Honduras, where USAID and other donors have aggressive programs for educational change, as it is for Peru, where USAID has no educational objective and where educational issues were far from the top national priority until recently. It also is true in Nicaragua, where political alliances complicate the agenda. Perhaps one of the approaches that has best worked for PREAL thus far is to selectively target and involve senior decision makers. Participants in seminars include top officials from Ministries of Education, leaders of business associations, and presidents of unions. At the Task Force level, the members have represented very senior echelons. In all of the countries under study, most people who have been in one way or another actively involved with PREAL are senior level professionals. To bring about policy reform in developing countries, it is nearly axiomatic that one must sit at the table with the policymakers. PREAL has been there. Two related ways in which PREAL has engaged senior individuals are through nominations to the Task Forces and high profile international conferences. Another of PREAL’s approaches that receives frequent positive comment is its involvement of the private sector. While the Evaluation Team agrees with this strategy in principle, we do not have evidence to describe fully its effectiveness. PREAL’s attempts to involve a broad range of private sector institutions, however, do appear to have been especially effective in building constituencies for educational reform. The third approach—evident from the discussions under Finding 1—is the dissemination of information to an educated lay audience. The findings of the Task Forces have received considerable attention. In reaching so high and so broadly, however, PREAL seems to lack a real strategy for integrating the levels through which it works. On a regional or sub-regional scale, the work of each Task Force becomes a case in point. In the discussion of Finding 2, it was pointed out that the effectiveness of the Task Forces’ findings depends on a combination of factors. The Task Forces convened senior people from the countries represented, and this constituency is appropriate. But the several layers through which each Task Force’s recommendations would have to pass in order to achieve an impact on policy implementation seem not to mesh well. As it is often the case when operating at this level, high-level strategic thinking does not necessarily reach into the administrative layers that would turn recommendations into actual practice or policy. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 14 Acknowledging that the two Task Forces are at different stages of development, PREAL does not appear to have planned fully for follow-through, even though the presentation of the Task Forces’ recommendations in public seminars—most recently of Mañana Es Muy Tarde—is an excellent strategy. As for the private sector’s role, the evaluation did not find in the information available a systematic vision of what the sector can do, or even a discrimination of the components of the sector.3 Much the same can be said of the broad engagement of social sectors in terms, for example, of what groups should be represented, what should be expected of each group, what different sorts of contributions they could make, and how these would mesh. The efforts made to assure inclusiveness do not seem to be systematically included in the program strategy and planning process. The issue is one of strategic choices. Thus, for example, the issues of quality and equity in education demand that attention be given to multiple areas, such as revision of curricula, decentralization, teacher training, testing, evaluating instructors, physical infrastructure and so forth. These are target areas for educational reform. In addition, the organizations promoting reform must negotiate with each other, and organizations promoting reform need to work with the various actors, perhaps mediating between the various interests with ministries of education, private, including corporate, interests, parents’ associations, teachers and their syndicates, and the universities. PREAL has made efforts to galvanized interest, provided information of broad concern, offered limited technical assistance, but it has not followed through deeply in any single area, except perhaps for the efforts of the Working Groups. 3 In Central America, PREAL is taking some action to understand the private sector. It is conducting a survey in four countries, with the objective of providing a typology of how the private sector relates to education. The instruments for the survey are good, but the question remains: how will the results be incorporated into planning? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 15 Finding 6: Under present funding and structure, neither PREAL nor the activities it supports are financially sustainable. The country reports get into some detail on the issue of sustainability in the local context. This section addresses those points that apply to the four countries. Under its USAID grant, the PREAL Program has a specific mandate, and once that mandate is accomplished—as it nearly has been in Honduras—the “raison d’être” for the specific activities carried out under such a grant is over, despite the fact that policy reform in education may well be considered an ongoing need. Because of these facts and insofar as PREAL is or remains a donor-funded intervention in development assistance, it should have a strategic plan for withdrawal. The most obvious avenue for withdrawal, or phase-out, would require transferring responsibilities to local or regional institutions that could continue this type of work on a more permanent basis. This, however, also raises the issue of financial sustainability. In the event of withdrawal, it could be argued that there will always be a need for some kind of coordination if the intent is to maintain a regional presence. Because PREAL depends on donor funding, and because it has no plans to seek and would be highly unlikely to receive endowment monies, a program such as this is difficult to sustain and requires a systematic plan to seek alternative funding. From the perspective of a network such as PREAL, this issue is highly relevant with respect to the Associated Centers. Our research reveals that given the highly competitive environment to obtain donor funding, it is quite likely that for most of the Associated Centers to maintain the type and level of activities carried out with PREAL support, additional funding from other sources will be required, thus raising an important consideration in terms of the long-term planning of the network. The Associated Centers that do not receive funding from PREAL depend in most cases on their own capacity to leverage funds to maintain activities related to the PREAL agenda. Some of the non-funded Associated Centers are sustainable, such as CIDE in Chile or GRADE in Peru. Others, such as FEREMA in Honduras, may prove to be so. For others, however, such as the Nicaragua centers, funding is likely to be a critical issue. In many cases, even if the organizations themselves are sustainable, funding PREAL-related activities remains an issue. Evidence from Peru and Nicaragua suggests that when PREAL’s funding comes to an end, those PREAL-related activities initiated with the centers tends to slow down considerably or stop. To a large extent, this situation makes some of the activities highly dependent upon the capacity of the institution to carry on. Finding 7: There is evidence of changes related to education reform policy that are linked indirectly to PREAL’s activities and role. Although research evidence shows no official change in educational policy directly attributable to PREAL’s activities in any of the four countries during the period of the cooperative agreement, evidence shows that Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 16 PREAL has influenced the adoption, modification, and implementation of educational procedures and practices in PREAL-serviced countries. Although this was an issue explicitly addressed by the Terms of Reference for this evaluation, it must be addressed with caution. “Official change to educational policy” in this context is meant to indicate formal changes in policy carried out by a government act through a ministerial decree or a law passed by the legislature, which enacts a substantive change to educational policy or practice. No such changes were found to be attributable directly to a PREAL-supported activity. This, however, is not surprising, as it is often difficult to identify linear cause-effect relationships in public policy change, and time is a critical factor to consider. The countries visited support this finding. In Chile, for example, this issue has little relevance, since PREAL under the USAID agreement does not have program activities. In Honduras and Nicaragua, no immediate evidence was found that the recommendations made by the Task Forces or by the Working Groups, have been implemented as official policy change. Nevertheless, there are indications that in several instances, the PREAL￾related activities have contributed to actions that represent actual changes in the ways in which policies are assessed or implemented. The best indications of PREAL “impact” on policy changes could be considered at two levels. First, PREAL achieved the objectives for which USAID gave it funding under the agreement. Second, there are indications that donors such as the World Bank, the IDB, and the UN agencies, not to mention the bilateral programs generally, view PREAL’s interventions as ranging from useful to extremely important. Additionally, the fact that even the best efforts of large donors have yielded very few measurable changes to education policy or practice speaks to the difficulty of the undertaking. Thus, a review of the situation is one for the donor community at large, not just for PREAL. Additional Findings In addition to these general findings, because an important part of PREAL’s operations is managed from its regional office in Chile, the following findings that could be considered relevant to PREAL as a whole were drawn from the field visit to Santiago: 1. There is unanimous consensus that PREAL, since its origin, has made an important and relevant contribution to education reform priorities. 2. PREAL publications and events are perceived to be highly effective and of high technical quality. 3. Where access is available, the PREAL internet site and those maintained by some of the Associated Centers are an important and useful information tools. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 17 4. PREAL’s approach has been appropriate to its current objectives under the existing conditions for education reform in the region. 5. There is a perception among those interviewed—a minority perception but nevertheless an important one—that PREAL needs to further clarify its role, both in relation to its donor partners as well as to its sponsoring and partner institutions. 6. Prospectively, the success of PREAL is a source of both risks and opportunities. PREAL’s success is likely to add pressure to the Program to expand its scope and be even more responsive to external demands despite its limited resources. At the same time, PREAL’s reputation to date represents a window of opportunity to continue influencing education reform processes in the region. Recommendations This section presents some general recommendations resulting from this evaluation. The country chapters provide specific recommendations. In most cases, these assume the continuation of PREAL and/or continued funding for PREAL by USAID. The recommendations in this report should be reviewed with the understanding that they are almost exclusively based on information from the four countries visited. It is not possible to say if the findings and recommendations presented here would have been similar or significantly different had the research been conducted in different countries. The intention of the recommendations is to indicate directions rather than present solutions. The recommendations address complex issues that require careful consideration by the parties involved. LAC/RSD/EHR must decide if it wishes to continue support for policy reform, and, if so, it needs to assess if this should be done through PREAL. If the Bureau continues, does it wish to maintain a regional or a sub-regional focus? While the present evaluators are tempted to make their recommendations on this issue, they do not share a unanimous opinion. Even if they did, their opinion would be based on insufficient information, since they have not evaluated the many other countries in which the program has been operative. In Chapter I.3, Further Considerations, the report provides a series of questions for LAC/RSD-EHR that provide further “food for thought” in the process of strategic planning and in the determination of the scope and nature of any future funding to PREAL. Recommendation 1: Within the context of a given assistance package, define PREAL’s responsibility and parameters of action more narrowly. From the perspective of USAID, defining more narrowly PREAL’s focus, would mean identifying a theme(s), and/or deciding to limit the scope of the assistance to one or more countries. If it means the latter, choices still remain: Is it preferable to select countries Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 18 based entirely on their likelihood to enact policy reform and improve educational equity, or quality, or both? Is it preferable to select countries where there is not an identifiable ripe climate for change, or will it prove more strategic to select countries in a sub-region, such as Central America? The IAD, which shares with CINDE in Chile “parental” responsibility for PREAL, need not be alarmed at any such proposal. USAID should recognize in the context of its deliberations, that PREAL is not a program of USAID’s creation, but an independent program that with or without USAID funding could likely continue in some form if alternative funding were to be found. Asking PREAL to conduct certain activities or to achieve certain results that are narrower than the vision the IAD has of the program does not need to compromise that vision. Any funds USAID provides the program further its own strategic objectives. Currently, for example, PREAL concentrates substantially in Central America in response to USAID priorities. Of course in order to do this, the Bureau may have to do more extensive thinking about its future strategy, and we believe that imminent plans are underway for related studies. Should USAID—and the IAD—invest in a more concentrated area of intervention? If PREAL’s scope and ambitions risk exceeding its capacity to carry out and manage a broad range of activities as indicated by this report, and if far higher levels of funding may not be forthcoming from USAID, then only by narrowing scope and ambition can PREAL effectively conduct and manage its mandate. Diminishing the number of countries is not the answer—even though it may be part of the answer. The finding that PREAL’s scope and ambition risk surpassing its management capacity is based on country, not regional, information. Furthermore, the problem is limited neither to PREAL nor to the side of the stakeholders: the host governments suffer from the same indigestion. Even the official proposal for reform in Honduras is so comprehensive that, according to numerous respondents, the Ministry does not know where to begin. Everyone calls for systemic reform, but few set priorities. This is a formula for standstill. Since LAC/RSD/EHR must have a regional vision for its program, it should decide what changes to policy and practice—even if only in selected countries—will advance the causes of improved equity and improved quality. One focus for serious consideration is that of testing and measurement. Transparent results from good tests—which by definition offer valid and reliable data—can be enormously effective in catalyzing the reform process.4 4 Within the region, for example, consider Brazil’s PROVAL program, in which it is applying consistent tests to students and graduates of universities and making the information widely available to the public, ranking universities and faculties on a performance standard. Previously unheard of in Brazil, this process holds more promise for near-term improvement in university curricula and teaching than any other in the last forty years. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 19 The beginnings of this process are already visible. The Working Group on Standards and Evaluation, for example, has produced numerous studies, and these have been presented in domestic and international seminars. Multinational donors have applied international tests to most of the countries in the region, often generating controversy. The call for measurement is heard in each country, and it applies to in-class evaluation of students and teachers, performance monitoring of schools, and national comparisons. PREAL could not address every one of these requirements, but it could go far to addressing some of them. Recommendation 2: Within the parameters of a USAID grant with a new and narrower mandate, ask PREAL to revisit and update its vision and a supporting strategy, as these are related to cooperation with USAID. PREAL has been highly successful in creating and articulating broad visions of educational reform. The two Task Force reports have had a wide impact in creating a sense of urgency and in signaling general directions for action. The Working Groups have provided valuable material. There is a need to revisit the means by which PREAL follows up on the reach and impact of these materials as a means to determine the extent of their impact on policy or practice. USAID and the IAD should leverage the obvious comparative advantages of PREAL in convening high-level decision makers, galvanizing broad interest, and creating both a vision for the future and a sense of urgency. Another comparative advantage has been PREAL’s ability to gain collaboration across various social sectors. Another has been to work with the donor community in the countries. Now PREAL should revisit its results framework to further define practical outcomes within its manageable interest. The highest level of results for which PREAL can reasonably accept responsibility, even if shared with other entities, will be the starting point for defining genuine intermediate objectives. The objectives should be performance￾based and verifiable. Some may be outputs, others results. They should all constitute a path of progress towards the highest level of achievable results. Recommendation 3: Further develop a realistic but demanding performance monitoring plan. PREAL has been consistent in monitoring its outputs but not in following through with monitoring of outcomes. The good news about monitoring plans is that constructing them is logical, clear, and easy if the results framework is realistic, clear and accurate. How does one know if a target or objective has been achieved? The answer lies in indicators. Indicators in the USAID approach suffer undue mystification, however. An indicator is no more than a specification of accomplishment. A monitoring plan can be the ultimate responsibility of the IAD, and it will not prove onerous if the strategic framework is detailed, realistic, logical, and clear. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 20 Placing an emphasis on monitoring spawns other positive consequences. For example, PREAL should begin to state performance outcomes that it desires from its seminars, and it should have the techniques and resources to monitor them and support follow-on activities. Should PREAL elect to give targeted support to the area of measurement, it will already be setting an example in its own program behavior. Monitoring encourages follow-through, so that outputs will not simply resonate in the public voice for a while and disappear in time. Systematic monitoring not only demonstrates achievements but also promotes them. Recommendation 4: Consider shifting some attention and responsibilities from the Associated Centers to the Working Groups and including all Working Group representatives in the Executive Committee. The Working Groups are established by PREAL and have specific mandates. They are largely accountable to PREAL. More than the Associated Centers, they represent unambiguously the interests of PREAL. The Associated Centers are administrative mechanisms, conduits of funds and information, and occasionally lobbying groups. Some of them would barely notice if PREAL disappeared from their sphere. Others are likely to fold immediately. The Associated Centers are independent institutions and share no common identity, except for a loose association to the network. The stronger ones have their own programs of work and agendas of activities. Generally speaking, they are not serving as think tanks or research centers on behalf of PREAL, but are affiliates with their own identity within a broad network structure. In some instances, they are working in coalition with PREAL toward certain common objectives. The advantage is that at least some capacity they know their own country context better. The Working Groups could more nearly fulfill those functions. As the Chile case study observes, the Working Groups have been highly effective in summoning attention to critical issues and in contributing to the quality of national and transnational debate on educational reform. In the case of Peru, people tend to identify PREAL more with the Working Group there than with the Associated Centers. “Working Group” is an appropriate designation. These entities are neither high￾performance teams, nor are they permanent structures. They are accountable to PREAL and the risk of suffering from divided loyalties is low. Their dependency on a single individual is likely to be less than in the case of the Associated Centers, and in any event the risks involved are certainly less great. They can be managed with reasonable effectiveness from Washington, or from any distant center, because their marching orders are clear. Their members, like those of the Task Force, do not even have to be in one place, although they would have to meet from time to time. The objectives of Working Groups can shift from time to time without necessary disruption, though the constituency of the groups might change to accommodate the required technical expertise. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 21 As for the Associated Centers, one must ask on a case-by-case basis, What is the value￾added? The burden is on PREAL, or the IAD, to make clear its expectations of the Centers; it will be no less so for the Working Groups, but the solutions may be more readily forthcoming, and easier to implement and monitor. Working Groups can also focus on sub-regional issues effectively. The regional approach is excellent as a means of facilitating the work of others. At the present time, there is no Working Group constituted in Central America. Central American individuals and institutions do participate in the regional Working Groups, however. Whatever the decision regarding the balance of responsibilities between Working Groups and Associated Centers, Central America should be involved in one or more Working Groups, particularly given that USAID’s regional education portfolio is proportionately concentrated there. PREAL’s Task Forces might also be considered. Recommendation 5: Resolve the perceived misunderstandings and improve communication between the IAD and its client, USAID. The information that the evaluators gathered through their interviews in Washington as well as in the field shows signs of misunderstandings between the IAD and USAID. They have to do with program ownership, indicators, initiative in communicating, and management. While none of the issues have been serious enough to affect the program, none seem to have been resolved. Such misunderstandings are commonplace in development assistance and are readily amenable to solutions. It will be useful for each party to the agreement to remind itself of the other’s point of view. Due to the large proportion of PREAL’s budget that is funded by USAID, the IAD is sensitive to maintaining the Program’s independence from PREAL’s main funder and not project the impression that PREAL is a “USAID program.” USAID is not convinced that its partner has been fully sensitive to its needs. From the evaluators’ perspective, both parties are right. PREAL existed as an active program before USAID chose to invest in it and would probably continue even if USAID withdrew its funding. USAID granted assistance to the IAD precisely because it found in PREAL a vehicle for responding to concerns voiced in the Summit of the Americas. On the other hand, the IAD accepted a cooperative agreement with the LAC Bureau and in doing so acknowledged that the Bureau wanted to remain more closely involved than it would have under a grant. It is generally recognized, unofficial practice for a grantee to take the initiative to contact USAID with regularity even if the operating unit has not made any clear demands in this regard. USAID has required very few tasks of the IAD, which reflects a contemporary performance mentality and approach. The Agency has every right to consider that PREAL is working in its behalf, striving to advance the Bureau’s strategic objective. The IAD should have no objection to this stance, since there has been no requirement that PREAL dedicate any of its other resources to achieving USAID’s strategic interests. As long as the IAD seeks to remain independent of a single donor’s influence, it has only to diversify its resources accordingly. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 22 It is recommended that LAC/RSD/EHR give full reflection to define more precisely what it expects of PREAL in the field and in Washington, and to make these expectations transparent in any new cooperative agreement. We urge the IAD to solicit these expectations if it feels that they have not been rendered explicit. The Evaluation Team has observed no problems, issues, or misunderstandings that are not readily tractable under conditions of good will and open communication. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 23 I.3. FUTURE CONSIDERATIONS Part of the future thinking related to PREAL should involve the whole issue of a regional approach to policy reform as well as to issues of country or sub-regional focus. As it lies well beyond the scope of the present evaluation to address these issues, the Evaluation Team instead suggests some questions the Bureau should ponder in order to best resolve these matters. ß Is the Bureau prepared to invest in three more years of similar activities, with very little assurance that policy reform will take place? ß Is the Bureau willing to continue investment in countries where USAID has neither presence nor program, or even where USAID has presence but no educational strategic objective? ß In countries with bilateral educational programs, how does LAC/RSD/EHR intend to work with the Missions’ Strategic Objectives? ß Would the Bureau consider a more specific, thematic approach to policy reform— one that might prioritize one policy over another, or one set of actions over others? ß Would the Bureau consider focusing on a sub-region, even were such focus to reduce or eliminate support for several countries of historic importance? ß Does the Bureau want greater or less direct involvement in PREAL? Assuming that a future contract of a similar nature would be inappropriate, would it consider other more focussed mechanisms of funding? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 24 I.4. METHODOLOGY The Two Components As a program evaluation with close relevance to decisions about future planning and funding, the present research required document review and direct personal inquiry. The document review encompassed the original proposal and the cooperative agreement, indicators and notes on indicators from LAC/RSD-HER, and PREAL’s written materials made available to the consultants. The Evaluation Team also consulted numerous documents in the field. A list of the documents reviewed is provided as Appendix E. Direct personal inquiry included administration of a survey instrument and unstructured interviews. The Research Population and the Sample The research universe was largely limited to people who had participated in one or more events sponsored by PREAL, who received PREAL’s written materials, or whose senior positions made them aware of PREAL. A reason for this was that for the most part, the public at large was not likely to be familiar with PREAL, or would not be able to identify discrete actions. In each country the IAD, the local USAID missions, and the evaluation team selected a sample that would represent most, if not all, of the following types of organizations: ß The Ministry of Education ß Associated Centers ß Local NGOs ß Research institutes ß Teachers’ unions ß USAID ß Other donors ß Universities ß Private business ß Schools ß Parent or community organizations ß The media Most of the respondents in the four countries had been involved in some of PREAL’s events. In the cases of Honduras and Chile, nearly all respondents were favorably predisposed towards the program. The samples in the four countries can be characterized, Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 25 then, as purposive, representative of a broad range of stakeholders in education, and often strongly and obviously biased.5 The Evaluation Team also conducted several unstructured, open-ended interviews in the Washington offices of USAID and the IAD, both before and after the fieldwork in the region. Method of Inquiry As the sample overwhelmingly comprised individuals who were well informed about PREAL and about educational issues in general, the method of inquiry was one of key￾person interviews. The consultant in Chile and Peru held three small group interviews, and the consultant in Nicaragua held one focus group; all other interviews were one-on-one. The Survey Instrument To ensure consistency, the Evaluation Team prepared a survey instrument before leaving Washington for the fieldwork. The questionnaire was designed to provide reliable data to determine if the IAD had met its three performance objectives for PREAL under the USAID cooperative agreement. The questionnaire also provides data on the secondary and tertiary questions of the Work Plan, use of the Internet web page, and PREAL’s engagement of appropriate sectors. It offers insights into perceived obstacles and solutions to educational reform. Following introductory questions about the demographic characteristics of respondents and their organizations, the questionnaire comprised twenty-six close-ended and five qualitative questions. Of the twenty-six questions with binary (yes/no) or scaled answers, all but one included an open-ended component. This construction provided two important benefits. First, it offered grounded analysis of affirmative responses. For example, if a respondent answered in the affirmative that he had received written material from PREAL, he/she was then asked to cite or to describe one or more of the documents.6 The respondent’s ability to substantiate his or her response reinforces the credibility of that response. Second, the open-ended or qualitative component of the questions allowed for potentially unlimited discussion of the issue at hand. This methodology of data gathering was appropriate for the purpose of the survey. The quantitative aspect allowed for cross-tabulations comparing countries, organizations, and gender with other variables and gave a more solid and illustrative foundation to the findings. The consistency of questions asked in the four countries facilitated analysis and 5 Professional evaluators understand that usefulness of information does not necessarily correlate either positively or negatively with bias in samples. Note also that one group of respondents in Nicaragua held a clearly negative bias towards the program. 6 The method normally followed the pattern of asking for examples: Have you taken any actions or initiatives based on what you learned? (Yes or No). Could you give some examples? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 26 synthesis and allowed the drawing of conclusions representing the four countries overall. The opportunities for respondents to discuss the issues raised by nearly every question opened the path for far broader and deeper information than could have been gathered by a purely quantitative instrument. Analysis The fieldwork yielded seventy valid questionnaire interviews. The breakdown by country is Chile - 12, Peru - 16, Honduras - 24, and Nicaragua - 18. Upon return to Washington, the Evaluation Team carried out the analysis of data and discussed the results from every variable in the survey instrument. This discussion involved comparing frequencies from each country to the others and to overall frequencies, by limited cross tabulation, and by summarizing qualitative responses. The consultants then returned to the Work Plan to ensure that they had covered each item of inquiry. From this analysis they produced a series of principal findings, they discussed recommendations, and they designed a preliminary presentation of results for USAID. The lists of persons interviewed, the survey instrument, and a table of frequencies are found in the appendices to this report. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 27 II. COUNTRY REPORTS Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 28 OVERVIEW The following section includes case studies of four countries: Chile, Honduras, Nicaragua, and Peru. Each includes findings about PREAL’s implementation in that country. Because much of PREAL’s planning and implementation is coordinated in Chile through of CINDE, this section begins with the Chile country report. The Chile country report includes important descriptions: (a) the organizational configuration of PREAL at the regional level; and (b) program management issues related to the primary, secondary, and tertiary evaluation questions that are relevant to the regional program as a whole. Accordingly, based on the observations and interviews in Chile, recommendations at the end of the chapter relate more to PREAL as a whole. The other country reports focus on PREAL’s implementation from a national perspective. In these chapters, a brief discussion of the country context and a synthesis of the main findings is followed by detailed, country-specific discussions related to the primary, secondary, and tertiary questions. Each country report also includes specific findings and recommendations that are primarily relevant to the specific country context, particularly in the case of Honduras and Nicaragua, and to a lesser extent Peru. Because Peru is the base for one of the Working Groups and for one of the members of the Executive Committee, its recommendations more resemble the recommendations of Chile in that they apply more to the program as a whole. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 29 II.1. CHILE The site visit to Chile was conducted from November 26 - December 3, 2000. The evaluator held in-depth interviews with the three staff members of the Santiago office: Marcela Gajardo, Co-Director; Ana Maria de Andraca, Associate Researcher; and Nelson Martinez Berrios, Program Assistant. Twelve additional survey interviews were held with key informants from the Ministry of Education, the private sector, members of local research institutions, universities, and international organizations. Informal in-depth discussions also were held with several of these individuals. Based on these interviews, the evaluator made qualitative observations specific to the “presence” of PREAL-Chile in its regional role. Synthesis of Main Findings To set the findings in context, the reader is reminded of two things: that PREAL’s success and impact cannot be measured linearly; and that PREAL’s success and impact are related to its evolution, relationships with donor partners, and the nature of the reform processes within national contexts. In this light, the following general findings have been identified: 1. There is unanimous consensus that PREAL, since its origin, has made an important and relevant contribution to education reform priorities by: ß promoting dialogue about education reform, ß disseminating current and relevant information to key audiences, ß facilitating a dialogue between practitioners, researchers, and policymakers, ß raising awareness about priorities in education reform, and ß facilitating the articulation of public and private initiatives for educational improvement. 2. There is a perception among those interviewed—a minority perception but nevertheless an important one—that PREAL needs to further clarify its role, both in relation to its donor partners as well as its sponsoring and partner institutions. This refers specifically to: ß a perceived misunderstanding among some in-country partners that the Program has the capacity to allocate funds and play a role as a “donor,” ß the need to clarify its relationships with “Associated Centers” in aspects such as funding relations, representation, and program management, and ß the independence of PREAL as a regional program from its sponsoring institutions (i.e., the Inter-American Dialogue and the Corporation of Development Research) in terms of program planning, programming, priorities setting, and modus operandi. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 30 3. PREAL publications and events are perceived to be highly effective and of high technical quality, in terms of: ß reaching key audiences, both technical and at large, ß raising awareness about education within the education sector, and among public officials and key members of the private sector, and ß maintaining a high level of technical discussion regionally and nationally. 4. Where access is available, the PREAL internet site and those maintained by some of the Associated Centers are important and useful information tools to do the following: ß disseminate up-to-date regional comparative information on education reform, and ß make available information not found in other sources. 5. PREAL’s approach has been appropriate to its current objectives under the existing conditions for education reform in the region. Prospectively, however, there are indications that PREAL would benefit from revisiting two issues: ß its approach to keeping abreast of the challenges posed by the evolving national and sub-regional conditions for education reform, and ß its current operating structure to further increase its effectiveness and focus. 6. Prospectively, the success of PREAL is a source of both risks and opportunities. The success of PREAL is likely to add pressure to the Program to expand its scope and be even more responsive to external demands despite its limited resources. At the same time, PREAL’s reputation to date represents a window of opportunity to continue influencing education reform processes in the region. There is sufficient evidence from the interviews to suggest that PREAL needs to revisit three areas: ß its strategic program planning, programming, and priority-setting mechanisms in a comprehensive manner, ß its program management and decision-making structure, and ß its advisory mechanisms for program planning and delivery. Detailed discussions follow. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 31 Organizational Context PREAL’s Program Organization and Structure As previously stated, PREAL is a regional program co-directed by IAD in Washington, DC and CINDE in Santiago, Chile that works with public and private organizations in Latin America to promote educational reform in the region. PREAL is guided by “executive directors” at CINDE and the IAD and their respective teams in a joint secretariat. There is no general secretariat and regional secretariat distinction. A regional task force advises the joint secretariat and a cross-regional executive committee. The PREAL structure also includes sub-regional coordination or “Link Offices,” Working Groups, and Associated Centers. Executive Committee. The Executive Committee, which was created by PREAL in 1997 following a USAID recommendation, includes the executive directors of the IAD and CINDE, the leader of the Peru Working Group, and the coordinator of Central American activities. The Executive Committee meets approximately every four months to make decisions on planning and programming, project support, and publications. Santiago Office. The Santiago Office has three staff members, including the Co￾Director (the Executive Director of CINDE). The role of the office is to coordinate PREAL’s regional activities. Several activities (e.g., preparation of publications and the Web page, translations, and “Report Card” activities) are outsourced. Link Offices. PREAL has three “Link Offices”: Peru, Honduras, and Brazil. (Originally, the intent was to have a larger number of Link Offices, but costs limited this option.) The role of these Link Offices is to assist the PREAL Coordinators in representing PREAL at the country level and to assist in the dissemination of PREAL information and publications. The activities in the Brazil “Link Office” are coordinated directly from Santiago, as Brazil is not a priority country for USAID and it is problematic to cover for a number of reasons (size, language, etc.). Working Groups. The Working Groups are intended to perform various tasks, including working on strategic issues, with the purpose of gathering information, producing new knowledge through research, proposing policy options, disseminating information, and maintaining links with the Associated Centers (See below). PREAL allocated funds for three Working Groups: FLACSO-Argentina, on Teacher Unions and Education Reform (US$200,000); GRADES-Peru, on Standards and Evaluation (US$437,000); and CIDE-Chile on Politics and Educational Reforms ($151,000), whose activities are now near completion. (The original idea was to have one Working Group per each priority identified by the Task Force, but this was not financially feasible.) Proposals for the Working Groups were selected by the Executive Committee Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 32 based on the recommendations of the Task Force, the Summit of the Americas’ agenda, and specific consultations through meetings. Associated Centers. The mandate of the Associated Centers is to implement program activities at the national level, carry out research, contribute to disseminating publications, and, above all, create a national “space for debate” on educational reform. Under the IADB grant, PREAL established partnership with six Associated Centers in the following countries: Brazil, Colombia, Dominican Republic, Guatemala, Nicaragua, and Peru. Funding was provided at an average of US$120,000 per Center. Under the USAID agreement, support continued for these Centers (Brazil $154,000; Colombia $75,000; Dominican Republic $160,000; Guatemala $75,000; Nicaragua $200,000+; and Peru $68,000). PREAL funding also supported additional centers in Guatemala (3), El Salvador (1), and Honduras (1) with PREAL. Others were included without funding: Chile (1); Costa Rica (1); Ecuador (1); Panama (1); and Venezuela (1). Findings Several observations can be made regarding the Program structure based on the discussions with respondents: ß The original intention of structuring the Program in this manner was that both the Working Groups and the Associated Centers would play a key role in producing policy impact at the national and regional levels. However, this has not been achieved fully, and in some cases it does not exist. A number of reasons for this were found, including the following: a too academic orientation of some institutions; lack of experience among some centers on how to reach policymakers; tendency of some institutions to “protect” their institutional autonomy; and emphases and issues related to the approach of PREAL decision making and funding structure, which is intended to give institutions autonomy as part of a regional network. ß There has been a very high level of production of materials within the topics recommended by the Task Forces and drawn from the Summit of the Americas´ agenda, but a limited number of concrete policy recommendations due to the modalities of work of the Working Groups and Associated Centers. To some extent, PREAL considers that the Working Groups and the Associated Centers are still ideas in progress. ß Overall, the Working Groups and the Associated Centers have worked well. Coordination among themselves and with the PREAL regional office could be improved, however. ß A critical issue to the success of the current PREAL structure and modus operandi is financial sustainability. This issue cuts across the various work modalities (Task Forces, Working Groups, Associated Centers, publications, etc.) Some (i.e., Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 33 Associated Centers) are more affected than others (i.e., Task Forces). Overall, the agenda of PREAL depends to a large extent upon individual grants. ß Although implementers are aware of the difficulties presented by the various modalities, there is not a defined approach to address this except to continue within the cycle of submitting proposals to donors for grants. In part, this could be addressed by giving greater emphasis to developing a “revenue generation plan” as part of the strategic planning of the Program. ß There are indications that some of the difficulties encountered in coordinating the work of the various components of the current structure of PREAL respond to the differences among the various institutions working in partnership with the Program. There are also indications, however, that a more selective and focused approach and a greater emphasis on strategic planning by PREAL could contribute to greater effectiveness in this regard. Building Public and Private Sector Support for Educational Reform A key goal of PREAL is to develop a broad constituency for “fundamental education reform.” Operationally, this translates into two related program objectives: to build public-private sector support for educational reform; and to strengthen public-private sector organizations. The experience of PREAL in Chile has been oriented to use existing technical and institutional capacity in the country to carry out activities regional in scope. No direct investments are made by PREAL in Chile, except for those required to maintain the coordination, produce the Program publications, and support regional initiatives. To build public and private sector support, the regional office in Chile established relationships with two institutions that cooperated as Associated Centers: the Centro de Investigación y Desarrollo de la Educación (CIDE); and the center, PARTICIPA, Educación y Desarrollo. In such a capacity, no funds were allocated to these centers for in-country activities. In addition, PREAL established partnerships with private sector, public sector, and internationally-based organizations. PREAL’s activities with each of these types of organizations is described below. CIDE Until March 2000, CIDE was the institution responsible for coordinating PREAL’s Working Group on “Political Obstacles to Education Reform.” This Working Group concluded with a seminar on the topic and a publication of a book (in press). The seminar was positively received, and government officials identified it as a key PREAL activity. According to information provided by PREAL’s main site, a total grant of US$151,884 was allocated to this Working Group in 1998. Due to resource constraints, however, Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 34 PREAL suspended this financial support. Currently, no further CIDE-PREAL activities are expected. Several signs of success were found in connection with this initiative: ß As a result of the Working Group led by CIDE, there is increased awareness among public officials and private sector groups about the relevance and complexity of the “political dimension” of educational reforms acting upon public policy processes. ß CIDE acknowledges to have further developed its own public opinion presence, nationally and regionally, on the subject as a result of the WG activities. Today, CIDE considers itself better equipped to influence public opinion in the country on this matter. ß Despite the discontinuation of PREAL funding for the Working Group, CIDE expects to maintain a research focus on the subject. Without the involvement of PREAL, however, it is unlikely that the activity will have the same regional impact. PARTICIPA PARTICIPA is a non-governmental organization playing a direct role in preparatory activities for the Summit of the Americas. Successful collaboration took place between PREAL and this private organization around issues related to this event. Beyond this scope, however, current collaboration has been limited to exchanging information and publications, and participating in meetings. No funds from PREAL have been allocated to PARTICIPA except for US$10,760 allocated for the organization of a seminar in 1997. This collaboration highlights the following: ß PREAL has been effective in identifying and collaborating with partners that represent a comparative advantage and add value to the activities of the Program. ß Under this and through other means, PREAL has been able to provide input into high level events such as the Summit of the Americas. Private Sector Partnerships There are indications that PREAL reaches a broad network of both academic and research institutions, both in Chile and regionally. Interviews with faculty members from universities indicated that they seek PREAL publications and materials as a key source of information. In several cases, PREAL books and monographs are required reading for students in undergraduate and graduate programs. Discussions with people from private not-for-profit research centers reinforced this view. PREAL materials and events constitute a key source of regional comparative information Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 35 and technical contacts, both for research and program planning in these institutions. Several of these groups see in PREAL a willing partner and an important link to other institutions across the region sharing similar interests and concerns. In this regard, there is indication that: ß Opportunities exist for PREAL to continue working closely with and broaden its reach to the private sector. ß There is a perception that PREAL has made a considerable contribution to informing public opinion and developing a dialogue among practitioners, researchers, policymakers, and the private sector. ß PREAL has contributed to raising the importance of consensus around priorities in education, facilitating the articulation of initiatives as required by the complexity of national reform processes. ß Among members of the private sector, there is the perception that PREAL produces the “right type” of materials (i.e., reports for technical personnel and researchers, policy briefs, and best practices for decision makers and the business community), as these respond to the needs of different audiences. Public Sector Partnerships Public officials and policymakers at various levels of the Chile Ministry of Education also expressed appreciation for the information and dissemination role of PREAL as well as for the currency of issues in which it focuses its activities. In particular: ß Policymakers found the newsletter format for disseminating information about policy innovations and best practices highly relevant to their needs and available time. ß The seminars and conferences organized by PREAL were also found to be relevant. Particularly important in this regard was the access to current regional comparative information about education reforms. Officials at various levels of the Chile Ministry of Education also expressed their appreciation for the “brokering role” played by PREAL. They found particularly useful materials about “best practices” describing experiences in other countries and briefings (i.e., newsletters) about educational issues that give them easy access to synthesized information. Specific references were made to the following: ß the emphasis of PREAL on teachers and teaching conditions, as this is perceived to be a sector central to the success of any education reform and a sector in need of greater attention; and Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 36 ß the need to reach more directly middle-level public officials and functionaries (at the provincial and local level) within the education sector, as these often play a critical role in the implementation of reforms. Partnerships with International Donors Although only one international organization was contacted in Chile (i.e., OREALC￾UNESCO), several informants commented on the perceived links between PREAL, international organizations, and the “donor community.” In 1997-98, cooperation took place between PREAL and OREALC given their common interest on educational standards and indicators. This added value to UNESCO activities and led to some follow-up initiatives by OREALC. However, this collaboration did not continue, apparently due to lack of funds. UNESCO sees the need to maintain close collaboration with PREAL on issues of educational statistics and indicators, educational innovations and best practices, and educational standards. Other informants reinforced the view that PREAL can play a necessary role as a link between international organizations and donors, and national policymakers and institutions. In this light several observations were made: ß PREAL is perceived to play an important “bridging” role between international organizations and the policy-making community. ß Collaboration with key international organizations in areas of common interest seems to have been ad hoc and could be improved through more systematic planning as a means to maximize efforts and resources and increase impact. ß PREAL’s relationships with the international organizations and the donor community bring associated risks that may affect the perception third parties have about PREAL autonomy from pre-set agendas for education reform. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 37 Strengthening Capacity of Public and Private Sector Organizations Working for Educational Improvement One of the gaps identified in the promotion of education reform is the lack of up-to-date information to feed ongoing research, policy analyses, and public debates in education. By providing important information, PREAL helps to fill that gap. PREAL’s activities are therefore perceived to be directly relevant to strengthening individual and institutional capacities for education reform. PREAL’s approach is perceived to be appropriate for the current conditions of reform in the region and as a means to keep current on specific national educational developments. Both the Working Groups and Associated Centers play an important role. Individuals highlighted several issues related to the Working Group and Associated Center approach used by PREAL: Working Groups. These are highly effective mechanisms for systematically bringing attention to critical issues and promoting national and regional debate. In many instances, people identify PREAL with the issue-area of a particular Working Group. At the same time, informants expect that: - PREAL will strengthen the linkages between the products and results of the Working Groups and concrete policy processes at the national levels by way of closer follow-up on the use of these materials. - PREAL will broaden the scope of the Working Groups as a means to reach the “right” (other) audiences outside of policy-making levels. Associated Centers. Although the experience of the Associated Centers in Chile may not be representative of the approach PREAL uses in other countries in the region, to a large extent, the capacity of Centers to collaborate with PREAL depends on the institutional strength of the Center or its affinity to issues of common interest. Suggestions that were made related to the future consolidation of this approach. - A critical factor in the ability of Centers to collaborate with PREAL is availability of funds. To carry out specific activities most of these institutions depend largely on external (donor) funding. - Acknowledging that PREAL’s financial and human resources are limited, an important role the Program could play is as an “intermediary” between major donors and the Centers. There are indications that PREAL needs to revisit its approach to establishing partnerships with Associated Centers. This implies: Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 38 - clarifying issues of responsibilities, representation, and role of these Centers in the national context, and - setting a clear framework of work to avoid raising expectations about PREAL’s ability to support these relationships over time. Sustainability Directly linked to the capacity building role of PREAL is the issue of sustainability. At least two dimensions were found to be relevant in this regard: the sustainability of PREAL-promoted and/or sponsored initiatives, and the sustainability of the specific forms of institutional partnership. Several informants commented the following: ß PREAL is highly effective at creating an environment for dialogue and establishing partnerships, but there is a need to ensure that both the Program and its partners can maintain the momentum. ß The issue of financial sustainability is central to the effectiveness and success of the activities carried out by the Working Groups and the role of the Associated Centers. ß The sustainability of PREAL initiatives is not only an issue of finance but of capacities created through partnerships. This, however, requires that PREAL have a longer-term vision of the role played by the Associated Centers and the Working Groups and how PREAL can help its partners become more self-sustaining, and thus longer-term partners in education reform. Identifying and Disseminating Best Education Policies and Practices One of the generic activities that in the view of many informants best identifies PREAL’s contribution to the dialogue on education reform is the capacity of the Program to identify and disseminate information on best policies and practices. This activity corresponds directly to one of PREAL’s main objectives. Overall, there is ample recognition that PREAL is perhaps the main current source of regional information in this regard, well beyond what many international organizations currently provide. This opinion is shared by informants from the public sector, private research institutions, universities, and international organizations. Specifically, the capacity of PREAL to identify and disseminate regional and comparative information of best policies and practices is what many consider the most useful role of the Program. Several specific activities were repeatedly mentioned: the PREAL publications; the PREAL sponsored meetings, seminars, and workshops; and the PREAL website. Publications Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 39 The large majority of people interviewed receive and are aware of the publications produced by PREAL. Policymakers and public officials made particular reference to the newsletters as a useful source of synthesized information. The PREAL “Documents” series was also amply known and used, often in relation to the specific areas of interest of the respondent. PREAL books are also known and received by many. Overall, these publications are considered to be current, of high quality and among the best available. Members of the academic community saw them of particular use for teaching and research purposes. In most cases these are available in the institutions’ documentation centers and libraries. Several observations can be made in this regard: ß The PREAL publication and dissemination approach is highly effective, filling an important gap in the availability of up-to-date information from across the region. ß One of the most effective publication tools for a variety of audiences outside the academic community is the Newsletter format. Several suggested expanding this series to include topics such as “current events,” PREAL opinion/position on current issues, and education “news” from across the region. ß A small number of informants commented on what they perceived to be an “academic” overtone of PREAL publications (with the exception of the policy briefs). If the intention is to reach sectors other than researchers, academics, and policymakers—key “actors” in the education reform process such as teachers, parents, the press, and members of the civil society in general—simpler language could be used in these publications. Meetings The activities in which most people report involvement are PREAL-sponsored conferences (national and regional) and national workshops and seminars. Often, respondents associated specific meetings with topics of interest to them and with the work of the Working Groups. Among the meetings most often mentioned were the conferences in Costa Rica and Recife and the meeting organized by CIDE-PREAL in Santiago. Special references were made in this regard to: ß the relevance of these events in terms of their focus on critical education reform issues; and ß the opportunities these events offer to hear from technical experts, learn about other countries experiences, and develop both personal and institutional contacts. Website Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 40 The PREAL website (http://www.preal.cl) is a key instrument in the Program’s efforts to disseminate information and publications. Technically, it is managed by a company hired for this purpose, but decisions on content, maintenance, and design are made by PREAL staff. The site is well put together, easy to navigate, and rich in available information to which access is offered free of charge. To date it has received over 18,000 hits, and it is the regular means by which people from across the region approach PREAL asking for information and publications. The site allows users to contact PREAL directly and to link to other relevant sites. PREAL is in the process of developing an internal database to record the type of requests it receives, setting up a registration site, and introducing more interactive features. Several respondents considered the PREAL website to be one of the most useful tools made available by the Program. Most respondents in Chile have access to the site, and several use it regularly as a source of information. A small number that reported not using the site gave as a reason their lack of time to browse the internet or their lack of internet skills. Two features of the website seem to attract the most attention: the access and downloading of documents, and the database on best practices. The latter offers 85 cases of best practices from across the region, accessible by topic (six main topics) and country (15 countries). Many consider the PREAL site to be one of the best in the region, taking into account the characteristics of the Program and comparing it to other education sites of larger organizations (i.e., UNESCO, WB, the IADB). Several also indicated that the PREAL site has considerable future potential to disseminate a broad range of information on education reform and potentially reach a wider audience. Overall, the website was found to be: ß A highly effective and well managed tool that complements the Program’s regional role in reaching out, networking, and disseminating information; ß A means of bridging the language divide between North and South, as most information is presented in English and Spanish, with the exception of the database on best practices, which is only available in Spanish and will be translated into English in the future. It is worth noting that many of the documents are also available in Portuguese, although not on-line; ß A tool that will require from PREAL continuous attention and resources to incorporate the rapidly evolving information and communication technologies that could allow the following: - ongoing monitoring of its use, Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 41 - the development of effective feedback mechanisms, - the potential use of electronic conferencing and networking with the Associated Centers, and - the development of more interactive features. At PREAL’s own recognition, further steps also need to be taken to standardize the various “PREAL sites.” These are created and managed by some of the Associated Centers in other parts of the region. The fact that many of these sites have been created and are managed by the partner institutions has led to lack of uniformity/standarization in the presentation and formatting of the sites. Some of the respondents also suggested additional features they would like to see included in the PREAL website. These include: ß The more frequent use of an electronic newsletter to inform users about new materials included in the Web Page; ß A section in the Web Page to inform users about PREAL’s presence in the public opinion. This could include a subsite on “PREAL in the Press,” with copies and/or links to newspaper articles reporting on PREAL activities or those of its Associated Centers; ß A “counting” mechanism per subsite to track number of visitors to specific parts of the site that are visited more frequently; ß The creation of an electronic feedback mechanism; and ß The creation of a subsite to present PREAL’s views and positions on key issues, such as “PREAL Opina.” Best Practices PREAL best practices require a special reference, as these came across as one of the features that most respondents associate with both the PREAL publications and website. PREAL best practices are one of the key features of the PREAL site and part of its package of activities. They are selected according to standard criteria developed by PREAL for this purpose. Each best practice includes a brief description and links to documents and/or sites where additional information can be found. The information on BPs also is disseminated through a newsletter. Informants felt that: ß PREAL’s base of best practices fills an information gap about education reform. Although there are other databases (i.e., UNESCO and some of the Banks), these are not as complete and the information is not as clearly presented. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 42 ß The Associated Centers play an important role in the preparation of best practices, providing information and feedback from the national level. Complementing the information made available by PREAL is the REDUC database. Although REDUC is not a PREAL activity (REDUC has existed for over 20 years under the coordination of CIDE), PREAL established an agreement with REDUC to produce research reviews on the issue of education reforms. Access to the REDUC database is available through the PREAL site. Thematic Focus Improving Educational Quality and Equity The central thematic focus on education reform of the Program revolves around issues of quality of education and social equity. The majority of informants saw this as the right focus in the context of education reform. Many considered these issues as the underlining unresolved problems affecting not only educational improvement but development in general. Many others felt that even greater attention should be paid to these issues as they often are relegated to a level of “discourse” and not necessarily addressed in terms of their concrete (practical) dimension. In this regard, most respondents saw the report of the PREAL Regional Task Force, “The Future at Stake,” as a critical document to raise awareness about the problems affecting education in the region. Others, although recognizing the importance and relevance of this report, wondered about the follow-up role of the Task Force after the report was released. Other Thematic Areas Questions about the thematic focus of PREAL and the issues the Program addresses through its various activities generated a wide range of responses. Although most recognized that PREAL is on the right track, several indicated that there are important gaps in terms of issues that need greater attention. This raises the need for careful assessment about how to deal with focus, choices, and priorities in light of the current PREAL agenda and the available resources. Almost unanimously, respondents made reference to the “complexity” of education reform issues and to the need for giving proper attention to a range of issues, since “education reforms are not purely technical processes, as sometimes PREAL seems to convey.” Among the actors, areas, and/or issues that respondents felt needed further attention, the following were mentioned: ß Teachers. This sector is a central vehicle to implementing reform processes and in many instances is one of the main obstacles. Issues of teachers’ salaries, certification, performance, traditional culture, and status are often not properly addressed. PREAL Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 43 could address these issues and target teachers and teacher organizations as a key audience. ß Civil Society Actors and Organizations. One gap in the education reform debate is the participation of the civil society. Issues of local level participation, and community and non-governmental organizations are often missing from the policy debate. . Although PREAL explicitly targets business and political leaders, who are an important part of civil society, PREAL’s approach to targeting the policy making community could allow for more inclusion of civil society in the reform dialogue. Information needs to be targeted to these audiences, and representatives should be included in PREAL seminars/press reports, etc. ß Parents and Students. Despite the emphasis of PREAL on educational quality, not enough attention is given to the role of parents and students as actors in the reform process, particularly in relation to the quality of teaching-learning processes, teacher performance, and the development of monitoring and evaluation mechanisms. ß Middle-level Implementation. Although the policy level is critical in education reform processes, there is widespread consensus that other levels need to be reached, particularly where decentralization is taking place, including local and provincial governments, and schools. In regard to the latter, the issue of “gestion escolar” (school management) was identified as critical and could receive greater PREAL attention. Several identified “weak institutional structures” across the education system as one of the main obstacles to reform. ß Legal Frameworks. This was identified as the “forgotten” area in the education reform dialogue. Although there is recognition that this is a highly sensitive area, it also represents one of the major gaps threatening the feasibility of reform proposals. ß The Political Dimension of Reforms. Reaching out to national and local politicians (and not only policymakers), and the mass media was considered central to PREAL objectives in promoting a vision and a dialogue about education reform. Many thought that more could be done in this regard, including PREAL taking a more open and independent position regarding critical issues in the public debate. ß Poverty and Inequality. Many identified this as a critical underlying issue across the region affecting not only the conditions for reform but also the conditions within the education system itself. Many saw this issue as directly related to the quality of teaching and learning, student performance, school capacities, and accessibility to materials, information, and appropriate school facilities. Recommendations Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 44 The recommendations that follow are based on observations of PREAL operations in Chile, and the findings from the series of interviews with key informants. They may be considered relevant to the work of PREAL as a whole. Reach and Impact Although there is widespread consensus about PREAL’s success in building private and public sector support for educational reform, the Program could further strengthen its reach and impact by: ß Taking a more proactive approach to bringing together policymakers and representatives from the private sector and from the civil society around issues important to the education reform national debates. ß Targeting other key national and regional actors as part of its primary audience in addition to top-level policymakers (e.g., teachers, school officials, and middle-level government officials.) ß Promoting more frequent local seminars and workshops with a broad range of participants on issues complementary to its main thematic priorities. ß Reaching out in a more systematic manner to, and building public opinion through regular use of the press, radio, and television. ß Developing means by which to further its public identity and independent position on issues of national concern regarding education reform processes. Building Public and Private Sector Support There is ample recognition that PREAL from its operational base in Chile has succeeded in establishing various forms of partnerships internationally, regionally, and nationally. The relationships of PREAL with Associated Centers, Working Groups, donors, and international organizations are often cited as examples. There are a number of areas where PREAL could strengthen its position in these relationships as a means of maintaining its independence and its standards of quality control. This could be achieved by doing the following: Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 45 Associated Centers and Working Groups ß Identifying means by which to strengthen and clarify the interaction between the Program and the Associated Centers and Working Groups. This may require re￾defining the terms of reference PREAL uses for each of these modalities and the role of Program coordination in relation to partner institutions. ß Revisiting the standards PREAL applies in its institutional relations with the Working Groups and the Associated Centers. This includes aspects such as representation of PREAL in national settings, and standardization of formats used by institutional partners in publications and websites. ß Developing means by which to consult with and expand the participation of Associated Centers in PREAL decision making processes to strengthen the sense of ownership among partner institutions. ß Addressing up front and in a systematic manner the issue of financial sustainability of the relationships established with Associated Centers and Working Groups to avoid creating unfounded expectations about the funding capacity of the Program. Donors and International Organizations ß Strengthening its independent profile as a hemispheric Program in regard to the priorities of particular donor agencies or international organizations promoting educational reform in the region. ß Collaborating more closely with international organizations sharing similar interests and priorities as a means to optimize efforts and maximize resources. Dissemination of Information PREAL’s high profile in Chile and the region is due in large extent to its success in disseminating current and high quality information on education reform and on best policies and practices. This role can be further enhanced by: ß Assessing on a regular basis the effectiveness and reach of its various publications through the establishment of easy to use feedback mechanisms from its various audiences, ß Finding ways to adapt some of its materials to audiences other than policymakers and researchers, ß Continuing the development of its website, taking into account issues of regional accessibility and connectivity of its users, and Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 46 ß Developing additional electronic means of information sharing to keep audiences current on activities, latest developments, news, and website updates. Program Management and Programming PREAL has maintained a very “Spartan” or carefully controlled approach in terms of management of resources, facilities, and required personnel. There is evidence to suggest, however, that the success of the Program has resulted in an increase in the complexity and scope of the activities that PREAL may not be able to maintain with the resources available. Looking towards the future the Program needs to: ß Revisit its overall program management, planning and programming approach in a comprehensive and prospective manner. ß Develop a strategic plan for a period of 3-5 years, with proposed funding to be obtained independent of the Program’s existing funding sources. This could include specific strategies in areas such as program priorities, research, dissemination, monitoring and evaluation of its activities, public relations, and required resources. ß Develop a proactive revenue generation plan as a means to diversity its funding sources. ß Revisit the way in which the Program allocates funds to institutions and specific activities in order to avoid the perception that PREAL is another donor operating in the region. ß Strengthen its management committee by creating a permanent technical advisory mechanism to the management of the Program which would include representatives from PREAL key partners, including, for example, donors, Associated Centers, Working Groups, TF members. ß Strengthen the Program’s quality control and monitoring capacity as this refers specifically to the institutions and Working Groups operating with PREAL funding, without limiting the autonomy of these partner institutions. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 47 II.2. HONDURAS Educational Context Honduras faces deficiencies in the educational sector similar to those of the Latin American region in general, and of Central America in particular. These include geographic and social inequities, insufficient coverage, obsolete curricula, a teaching community that is inadequately prepared and poorly motivated, and excessive centralization of authority. Partly owing to the centralization of authority, and partly to the nature of the progress sought, all of these deficiencies demand policy reform or—at the very least—changes in practice initiated at and supported by the highest levels of decision making. This is the thorn in the problem in Honduras: reform is stalled at precisely those high levels. Reform is stalled, but it is not quiet. There is a clamor for educational reform today that has not been heard for over a century in Honduras, and that clamor can be heard in some non-traditional quarters such as the industrial sector. As of this writing, four organizations have prepared formal proposals for educational reform, and PREAL has influenced them all.7 PREAL has enjoyed a privileged position in Honduras, for its representative there is the de facto coordinator for PREAL in Central America, and this person has been exceptionally dedicated to the task. The following pages determine what role and effect PREAL has had in promoting change. The following findings emphasize PREAL’s major accomplishments and its weaknesses. Synthesis of Main Findings During the fieldwork for the present evaluation in Honduras, which took place from November 26 to December 7, 2000, the consultant administered the survey instrument to 24 respondents and spoke at length with several other individuals. He interviewed representatives from the donor community, the MOE, NGOs, universities, private schools, teachers’ unions, private industry, the media, and the opposition party. He also held extensive discussions with the PREAL representative and with the senior staff of PREAL’s sole Associated Center in Honduras. ß PREAL has been instrumental in articulating the need for reform, fomenting discussion, and providing insight to needs and solutions. ß PREAL has actively assisted its Associated Center in developing intellectual capacity and in securing resources. ß In being a strong and almost ubiquitous partner, PREAL has nurtured collaboration 7 The proposals have come from FEREMA, FONAC (the “official” proposal), one of the teachers’ unions, and from within the MOE. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 48 among diverse organizations concerned with educational reform. ß PREAL’s efforts have undoubtedly contributed to bringing Honduras to a state of unprecedented readiness for an overhaul to the educational system, although no actual policy reform has yet occurred. ß In Honduras, PREAL’s greatest perceived strength—the respect conceded to its capable and dynamic representative—is its weakness in hiding: perhaps with the exception of FEREMA, PREAL boasts virtually no institutional viability. ß PREAL will continue to be a useful actor in the area of educational policy reform in Honduras, but unless it changes its scope and focus, its usefulness is likely to diminish relative to the past. Building Public and Private Sector Support for Educational Reform The indicators of achievement of this objective are: ß Debate created at national level ß Widespread engagement of social sectors evidenced ß Awareness of issues and priorities witnessed among major stakeholder groups ß Collaborative actions undertaken by stakeholder groups ß PREAL’s ability to convene stakeholders established ß Public gatherings held periodically ß Sub-regional task forces operative and productive Only three years ago, the cry for educational reform sounded like a vox clamantis in deserto (voice crying in the middle of the dessert). No NGO was seriously raising the issues, and the donors were acting independently of each other. Given an exceptionally weak civil society, demand for change from the masses was unlikely to happen and in fact had not occurred. PREAL, however, built up steam rapidly. It established an Associated Center relationship with the newly founded FEREMA, working intimately with and through it. At the same time, PREAL conducted seminars and distributed its publications broadly. Setting no boundaries on its organizational contacts, PREAL worked with all major donors, and participated in MERECE, a donor group for educational change. It collaborated actively with FONAC, a public-sector organization established for conducting comprehensive, national consultation with all sectors of society on a range of pressing issues. PREAL convened representatives from public and private unions, universities and teachers’ colleges, from the government, and private enterprise to participate in national and international seminars. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 49 To what extent did PREAL succeed in promoting an active, widespread debate about educational reform? The first answer comes from the survey. Out of 24 respondents, 20 replied that they had participated in an event (i.e., seminar, conference) sponsored by PREAL. Nineteen out of 20 were able to provide an example of something new they had learned, and 17 of these explained how they had used this acquired knowledge in their professional lives. Thirteen of these 17 explained how their initiatives made a change within their organizations, while 11 gave evidence of some impact on policy or practice in the educational system. The steady decline from 20 who participated in an event to 11 who reported some impact on policy or practice follows an expected logic from any event with an objective of transmitting new knowledge or skills. That over 50 percent could demonstrate to the interviewer’s satisfaction that the events they attended contributed to changes within their organizations is exceptionally high for non-strategic training.8 It is testimony to the quality of presentations at the events. In addition, while only 11 of the respondents could think of ways in which the events they attended might have had some impact on educational practice or policy, 17 responded strongly that these events had contributed to promoting the social dialogue. The quantitative response from the survey is supported by the many explanations and examples the respondents provided. All the donor representatives interviewed, with and without the survey instrument, affirmed that PREAL had been instrumental in raising crucial issues, making them heard, and fortifying them with information from within Honduras and from other countries of the region. Two USAID officers described PREAL as a strong arm in the battle for educational reform and as a catalyst. The dean of one university and one of the Minister of Education’s principal advisors both stressed the prominence of PREAL in the reform campaign. One of the country’s leading business executives, himself active in calling for systemic change, said that PREAL had been the main force behind the struggle. In the person of its local coordinator, PREAL chaired the education commission of FONAC and thus played the major role in determining the content of the proposal for educational reform that will receive the greatest attention at the levels of the minister and the president’s office. Clearly, through its public seminars, literature, and presence, PREAL in Honduras has raised awareness and stimulated debate among a broad range of stakeholders and social sectors. It has inspired active collaboration among donors, the government, and the private sector, and the several formal proposals for educational reform are heavily indebted to the contributions of PREAL. Building Capacity in Public and Private-Sector Institutions for Educational Reform The overall indicators of achievement for this objective are: 8 First of all, we should note that none of these events was actually considered training, even though they have similar objectives. Second, by strategic training we refer to training with very specific objectives in terms of performance outcomes. Strategic training is highly suited to a results-focused environment. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 50 ß Associated Centers founded ß Associated Centers established through linkages ß Associated Centers carried out or promoted PREAL strategies In Honduras, PREAL’s single Associated Center is FEREMA.9 Founded in 1998 with a mandate to work for educational improvement in Honduras, FEREMA became an Associated Center of PREAL in the same year. How much did PREAL help in strengthening FEREMA, and how well has this center carried out PREAL’s strategies? One current FEREMA administrator felt that PREAL had been only somewhat useful in strengthening its management capacities. A proven business executive who provided money, prestige, and competence established FEREMA and acted as its first president, lending both his prestige and his management. But PREAL, in providing technical assistance and guidance on issues and priorities, indirectly improved its overall management. In the area of research, the administrator affirmed that PREAL had greatly helped FEREMA. It offered opportunities for FEREMA staff to participate in domestic and international seminars, introduced the website (the FEREMA administrator was the sole respondent in Honduras who used—or ever had used—the site), and kept it abreast of its own research and documents. PREAL also greatly assisted FEREMA in planning, largely through direct technical assistance in setting priorities in the educational reform strategy. Finally, PREAL was very helpful to FEREMA in fundraising. PREAL provided funds for various programmatic operations and continues with limited financial support. One of the reasons cited for PREAL’s usefulness in this area also speaks to the competent and creative management at FEREMA: since PREAL was the first outside contributor, FEREMA was able to leverage its financial assistance by providing a reference to other institutions. FEREMA now has confidence that it can be self-sustaining without PREAL and is looking for endowment funds. It is clear that PREAL achieved its objective of building organizational capacity through its assistance to FEREMA. FEREMA has used this capacity to conduct various studies on behalf of PREAL and to produce a formal proposal for educational reform that aligns perfectly with PREAL’s agenda. In addition to strengthening FEREMA, PREAL enhanced the capacity of numerous stakeholder organizations to work on behalf of educational reform, especially in the public sector. Through our survey information, we see clearly that stakeholder organizations are 9 FEREMA stands for Foundación Educativa Ricardo Maduro. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 51 aware of PREAL’s major reform objectives and that they have received and used PREAL publications. FONAC acknowledged PREAL’s work in chairing the education commission. Two professors and the dean of the pedagogic university embellished with examples their indebtedness to PREAL for such information as country comparisons, work on standards, and clarity of priorities. A professor at the National Autonomous University felt she had learned—and applied—information on evaluation and certification. The principal advisor to the Minister of Education attended several seminars, read thoroughly Mañana Es Muy Tarde, and was applying this information to an internal ministry proposal for reform. It would be no challenge to cite numerous similar examples—examples in which senior people in public-sector institutions are actually applying information they have gotten from PREAL’s seminars or from its publications. While even in the aggregate such examples fall short of demonstrating actual institution building, they nonetheless give strong evidence that PREAL is succeeding in enhancing capacity within organizations. Whether such enhancement is sufficient to cause actual change is another issue. USAID asked the IAD, through PREAL, to build support and enhance capacity. PREAL has succeeded in Honduras in achieving these objectives. Identifying and Disseminating Best Policies and Practices Because PREAL/Honduras had very little role in the identification of policies and practices for publication and distribution, the discussion here focuses on the indicators of appropriate dissemination. These are: ß Materials designed and packaged effectively for readership ß Documents regularly received by major stakeholders ß Website accessible Out of 24 respondents, 23 affirmed that they had received publications from PREAL. Nearly all respondents were able either to cite or to describe at least one document they had received. Many said they received a variety of publications with some frequency, and several were able to pull documents from their shelves upon the interviewer’s demand.10 Of the 23 who received one or more documents, all gave examples of something new they had learned from them, and 22 said that their organization actually used the information in some fashion. Sixteen provided examples of how the information in these publications may have led to actions favoring the development of new educational policies. 10 One amusing case occurred at the documentation center of the pedagogic university. The chief of the section went to the document shelves, upon the interviewer’s request, but despite her confidence to the contrary, was unable to find any PREAL document. She could, however, produce some from her own desk and talked about them knowledgeably. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 52 In Honduras, the document most cited is Mañana Es Muy Tarde, partly because it is the most recent, partly because it was presented in a seminar, and partly because it is a well￾constructed document that sounds an alarm in a cogent, informed manner about the region. But respondents cited numerous other documents, usually by their theme rather than by their title. The Evaluation Team’s judgment concerning the overall quality of the publications— including their packaging—is very positive. The consultant examined the mailing list in Honduras, which included scores of recipients in the private, not-for-profit, donor, and public sectors—including the unions. There is no question that, in Honduras, PREAL’s publications have reached major stakeholders and have had a substantive, and verifiable, impact on their thinking. The websites are another matter. Here is one instance in which a purposive and positively biased sample has given us the most indicative results possible. It is nearly beyond question that our sample includes those individuals most likely to visit PREAL’s websites: educated, influential, concerned individuals who work in organizations that for the most part have some computer connections. Yet only one of our respondents—the director of FEREMA—had ever used the site. Of the 23 who had never visited it, 9 said they were unaware of its existence and 13 gave another reason—predominantly difficulty of access to the Internet or even to a computer. For the foreseeable future, it is highly unlikely that the PREAL websites, no matter how well designed or advertised, will see significant use in Honduras. They are, nonetheless, accessible. Improving Educational Quality and Equity Has PREAL in Honduras improved the quality and equity of education? The straightforward answer is no. During the PREAL period there has been no demonstrable change in either the quality or the equity of education in the country directly attributable to PREAL. It would, however, be unjust to single out PREAL in this shortcoming. No donor or other program effort has made any improvements in equity or quality if measured purely by end result.11 While equity, largely measured at present in terms of access and enrollment, is a major concern in Honduras, quality seems to hold the upper hand. Responses to interview questions suggest that until the perceived dismal quality of instruction is improved at all levels, resolving issues of equity will be of little national benefit. Quality at present is discussed mostly in reference to curriculum and teachers. If the question that USAID asked of the evaluation is, “Has PREAL contributed to improved quality and equity by promoting dialogue and encouraging policy implementation?,” we should look closely at promoting dialogue and at encouraging policy implementation. The foregoing discussion in the previous sections of this chapter 11 Equity and quality represent the strategic objectives of USAID/Honduras. MERECE, the donor group for education, also has placed both these elements at the top of its results framework. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 53 has demonstrated that PREAL, as much as any organization in country, has accomplished this purpose. PREAL has made a significant contribution to dialogue and a focus on policy. Providing Effective Policy Reform Strategies and Activities PREAL’s strategy in Honduras can best be described as one of raising and maintaining awareness of the importance of educational reform through presence and collaboration with a broad range of stakeholder groups. PREAL has been present in donor meetings, in high-level committees, in the media, and through its widely distributed publications. It has encouraged collaboration with the donors and helped to coordinate their purposes. The Ministry of Education, the universities, and to some extent the unions are using PREAL’s information. PREAL has convened representatives of the private sector (i.e., industry, schools, and the media) and kept them on its distribution lists. In Honduras alone, PREAL has sponsored at least ten seminars. These have taken place in four cities and have directly involved as co-sponsors donors as well as the public and private sectors. The seminars have covered standards and measurement, decentralization, technology and education, the Mañana Es Muy Tarde report, and other subjects of current interest. Fostering Use of PREAL’s Recommendations and Findings by Beneficiaries The survey results indicate that all groups of stakeholders have made practical use of the information offered by PREAL. As indicated above, 17 of 24 respondents demonstrated that they had applied elements of new knowledge acquired from the seminars; 22 of 24 confirmed that their organizations used information from the publications they received. (Fourteen of the respondents judged PREAL’s publications to be of greater usefulness than those they received from other organizations such as UNESCO, UNICEF, the OEI, and so forth.) How effective has PREAL’s strategy been in Honduras? To answer this question requires the researchers to disaggregate PREAL’s influence from that of other organizations and to define the indicators of effectiveness. While respondents point out that this disaggregation is problematic, the survey methodology has accomplished the task of attribution in the best way possible: it has obliged respondents to identify specific seminars and specific publications and then trace their impact through personal action to influence on their organizations. The evidence gathered through survey responses, personal observation, and informal discussion suggests that PREAL has indeed been effective in articulating the main issues of educational reform in Honduras and in garnering widespread support for the cause. Structure and Sustainability Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 54 The structure of PREAL in Honduras looks like this: PREAL does have an office with one administrative person in Tegucigalpa. The individual who is viewed as the director of PREAL in Honduras and is the coordinator of PREAL for Central America, however, is actually a paid (full-time) consultant with neither the title, status, or much of the authority of a chief-of-party or country representative. (He is involved, however, with the IAD in Washington and is responsible for contractual negotiations with EREMA.) Because the success of PREAL’s efforts in Honduras is so largely owing to the dynamism of a single individual, THE IAD’s direct hand with FEREMA provides some protection against the vicissitudes of personnel changes. If the current structure allows for greater control and monitoring from the IAD, with the advantage of a lesser threat of disruption in the event of loss of staff in the field, it also militates against institutionalization of PREAL. If PREAL is to be allowed to fade away, having served well a purpose of galvanizing the dialogue and catalyzing broad interest in educational reform, the structure is a good one. If, on the other hand, PREAL is to continue in similar vein or with a new focus, its organizational status will require clarification. As it stands now, it cannot perform in the most effective manner with either an expanded agenda or with a more targeted, performance-oriented one. To what degree should PREAL in Honduras be a financially sustainable entity? First of all, it is nearly axiomatic that development assistance interventions should be phased out. As for other donor programs, the philosophical justification for PREAL requires eventual transfer of its activities to the host country. In some activities, such as small business lending institutions, one hopes the structures left behind will prove sustainable. In other instances, one seeks specific results through a program, and once those results are achieved, the program and its structures are no longer needed. Such is the case, for example, with traditional non-project assistance in USAID. Where does PREAL fit in Honduras? To the degree in which it is a catalyst, it should “disappear” once it has performed its function. Its raîson d’être is the enactment of systemic change in the educational policy and practice, even if such enactment lies beyond its contractual obligations.12 It is unlikely, however, that improvements in education 12 In USAID jargon, one would say that actual policy implementation is beyond PREAL’s manageable interest. The official enactment of policy would be at the goal or sub-goal level of an operating unit’s results framework. PREAL FEREMA IAD Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 55 policy over the short term will eliminate the need for ongoing monitoring of the status of education from a national or regional perspective. Recommendations The final question on our survey instrument asked respondents what they would suggest that PREAL do in the future to achieve greater impact on educational policy. The responses show less consistency than one might desire. Out of 24 interviews, two call for PREAL to lower the level of its attentions and focus more on mid-level people, including teachers; two call specifically for assistance to teachers; and three ask for emphasis on monitoring, testing, or evaluation. The four suggestions for more events and the four for alliances (always with the respondent’s organization) reveal a general appreciation for PREAL’s usefulness. But the seven suggestions—even calls—for concrete action bear the greatest attention. PREAL may want to solicit more details about the practical assistance that was suggested and consider whether it could support or at least encourage such activities. PREAL in Honduras has been so successful in achieving the program’s three priority objectives that one would be tempted to suggest no change in strategy if the program is to continue as it has in the past. It has built broad support in public and private sectors, has helped strengthen a sustainable NGO, has enhanced the capacity of numerous public institutions to articulate the need for reform, and has helped integrate the purposes of the various donors. The question to be raised here is the one raised in the chapter on synthesis of findings in this report: Is it all right to spend three or more years simply promoting the idea of educational reform? The same question must be asked of the donor community at large, including USAID. There has been no official change of policy in Honduras, and among those interviewed there is skepticism that such change will occur in the near future. When a vast majority of respondents cite the Ministry of Education as the principal obstacle to reform, and many others identify the teachers and their unions as formidable opponents of change, the case for optimism is weak.13 On the other hand, if PREAL is to undertake more concrete actions, it lacks the resources to work on the broad fronts it has addressed thus far. There seems to be a perception that PREAL has tried to be all things to educational reform in Honduras. If this is the case, it can no longer afford to play that role. The evaluator is acutely sensitive to the risks of arrogance, superficiality, and plain error in proposing new directions for a successful and respected program. The following recommendations are based on his discernment of the most solid observations he heard in the field, as well as of his knowledge of donor programs in general. 13 As the representative of one donor agency remarked, The relevant political structures lack coherence, effective relationships, any sort of coordination, and they have no single leadership (trans). Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 56 PREAL in Honduras may soon face challenges, such as diminishing usefulness and lessened effectiveness. Its usefulness may wane as the organizations it has helped gain focus and identify priorities assume the struggle on their own—or in groups. Its effectiveness may lessen as its objectives are superceded by other imperatives, or it exceeds its ability to manage a broadening scope of efforts. PREAL/Honduras should retain strong leadership if it is to guide PREAL and its related organizations as they confront these challenges. The accolades PREAL has received in Honduras for what it has accomplished may tarnish if some concrete results are not forthcoming. How can PREAL achieve such results? It is neither within the scope of this evaluation nor the competence of the writer to propose a precise road map. But what is needed is precisely a road map, and that is what we suggest. PREAL/Honduras, with its access to regional, indeed global, knowledge, should determine through strategic planning what can be accomplished in a determined span of time. It must focus on policy reform and assume some accountability for the effective implementation of such reform. The reform need not represent a complete overhaul, but it should be one essential element in systemic change. Numerous interviewees proposed that evaluation and testing be a strong focus area in the future. If PREAL/Honduras opts to work in the area of measurement, it will benefit from several advantages. It can promote a significant change in educational practice that will precede and promote change in policy. If measurement means applying and disclosing the results from international tests, it will further the call for reform and for actions improving quality. It can work in this area through external technical assistance, available in the region as well as in the donor countries, and it can harness the potential of the Working Groups, thereby lessening its almost total dependence on a single leader. It can work in parallel with the donor efforts, reinforcing them. It need not exceed its limits to manage, and it will create for itself an image, almost a brand, that it lacks at present. Whatever path PREAL opts to follow in Honduras, it should create a road map with performance targets. It should monitor those targets carefully and periodically, and assume responsibility for achievement. The results framework, crafted with the assistance of an experienced person, will prove a useful exercise for the purpose. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 57 II.3. NICARAGUA Educational Context Nicaragua’s educational system demonstrates many of the problems associated with the region’s poorly performing systems. Many of the commonly collected cross-national indicators show it to rank near the bottom of countries in the region in terms of its quality and efficiency. For example, the adult literacy rate is 63.4,14 and only about half of the children who enter primary school will finish the sixth grade. A quarter of the children of primary school age (7-12) and half of those of secondary school age (13-17) do not attend school. In terms of spending per pupil in primary and secondary, Nicaragua (along with El Salvador and Guatemala) trails the rest of Latin America. At the same time, however, it devotes the comparatively enormous sum of six percent of the national budget, or 24 percent of the total education budget, to higher education, a level that two percent of the population will achieve. Linked to low spending is the issue of teacher salaries, which at an average of $60 per month are the lowest in Central America. As in other countries where salaries are so low, this impedes quality improvement because it is difficult to attract qualified teachers. There is an incentive for teachers to leave and find better paying jobs, and their motivation tends to be low. Institutionally, some innovations have occurred. Nicaragua, for example, has gone farther than almost any other country in the region in terms of decentralization. Most secondary and roughly half of primary schools are autonomous, meaning that they are run by school councils who receive resources from the central Ministry. Since the transfer is partially a function of how many students attend, these schools have an incentive to reduce the dropout rate and are making progress compared to their traditionally run counterparts. Systemic reform has been quite slow in Nicaragua, however. This is in large part due to the high level of politicization that characterizes policy making in the education sector and is part of the aftermath of the civil conflict the country suffered in the late 1970s and 1980s. There is no general Law of Education and the legal framework for the school autonomy program has not been passed by the legislature. A National Plan of Education that will lay out the objectives, priorities, and resource needs of the various subsystems of education should be completed this year.15 Synthesis of Main Findings This evaluation was conducted during a twelve-day period in November and December, 2000. It is based on interviews with members of PREAL’s two associated centers, USAID-Nicaragua representatives, leaders of civil society organizations working for education reform, business leaders, policymakers, and Ministry of Education, Culture and 14 Comisión Centroamericana para la Reforma Educativa. 2000. Mañana es muy tarde. PREAL: Washington, DC. p.8. Source: UNDP, Human Development Report 1999. 15 After the research date, the national Plan was approved. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 58 Sports officials. In addition, the evaluation consisted of group discussions and a review of the documents produced in Nicaragua as well as PREAL publications pertinent to the country and region. The two PREAL Associated Centers, IDEUCA16 at the Central American University, and the NGO, EDUQUEMOS, have different audiences, convening power, political orientation, and presence in the national educational debate. IDEUCA is a center that predates the PREAL program, is rather academic in nature, and is housed at the Jesuit University. EDUQUEMOS is a much smaller organization created by PREAL with the mission of promoting the participation of the private sector in the dialogue process. Undoubtedly, both organizations have worked to strengthen policy dialogue that may eventually lead to educational reform by raising awareness of education issues and by participating in policy formulation. These centers have contributed to increasing public support for reform. In that sense, the evaluation found that the PREAL program has succeeded in perhaps the most important of its major goals. Admittedly, unlike some other countries in the region, Nicaragua has not experienced an officially denominated educational reform. Nevertheless, some changes did occur in areas like curriculum and decentralization. The absence of officially denominated educational reform should not be taken as a sign that PREAL has failed to achieve demonstrable results, since the enormity of the task of reform requires a great deal of time and dialogue between different political positions. A second finding is that PREAL has strengthened the capacity of the Affiliated Centers, although neither is as yet self-sustaining in the sense of being able to continue to carry out the PREAL-funded policy research and awareness activities without continued support. The evaluation also found that the dissemination of information produced by PREAL is less effective than it might be. There is a much better network of distribution of the work produced in Nicaragua under the PREAL banner than documents that are distributed from Chile. While certainly some respondents noted that among the benefits of receiving documents from PREAL is that they allow respondents to see what other countries are doing in terms of reform, there was also the sense that for many, PREAL documents consist of IDEUCA work. A number of respondents could only point to the regular Sunday newspaper column written by IDEUCA as an example of PREAL documents. The use of PREAL’s website is extremely low, although this is unsurprising in a country where access to the Internet is highly limited. A fourth finding is that there is a tendency for relatively little coordination or communication across the different levels of PREAL, both with respect to different activities under the PREAL label and with respect to USAID. Although such independence may allow for great flexibility, there also may be opportunities and gains to be realized by closer collaboration. For example, if in the course of writing a book on education reform in Central America, IDEUCA researchers had studied USAID’s model 16 IDEUCA is the name that UCA-PREAL adopted a year ago. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 59 and seen the changes occurring in primary education, they may have had very different findings on primary education reform in Nicaragua, including the role of the Ministry of Education, Culture and Sports in bringing about this reform. Building Public and Private Sector Support for Educational Reform Both IDEUCA and EDUQUEMOS must be credited with promoting support for educational reform. Both institutions have organized numerous in-country workshops and fora, participated in regional events on reform, and worked through the media to influence public opinion on the topic. In addition, through key members of the organizations, they have participated in the National Plan for Education, which is the main effort currently underway for education reform. Each of the Affiliated Centers has a distinct target audience, and the groups that are most involved with each center tend to evaluate the work and contributions more positively than groups or individuals not involved with the centers. This is at least as much a reflection of the continuing divisions in Nicaraguan society as of the actual work being carried out by the centers themselves. IDEUCA’s strategy has been to work with groups in civil society such as unions and community groups, in particular through training courses that focus on education reform and policy formulation. They also have produced or contributed to numerous academic publications about education reform in Nicaragua and in the Central American region. Finally, they also maintain a presence in Nicaraguan society and work to promote support for educational reform through a radio program and weekly news column devoted to education issues. Many of those interviewed greatly value IDEUCA’s participation in the debate, partly because of the great respect commanded by the director and partly because the involvement of an intellectual of such stature helps bring legitimacy to the focus on popular sectors. At the same time, there are critical voices, even from those who think that IDEUCA has made positive contributions. In particular, a common worry expressed was the insularity of the group at IDEUCA and a perceived tendency to not be open to points of view other than their own. The case of EDUQUEMOS is rather different. This is an organization that was founded by PREAL in 1998, with the intention of promoting private sector involvement and support for education reform. It operates on a much smaller scale than IDEUCA and has one administrator and one assistant. Its strategy has been to work with other organizations and to use personal networking to achieve its goals. It has, for example, worked with COSEP, the employer’s association, to establish a subcommittee on education. Early on, it addressed the issue of the tax code and how to create incentives for businesses to support education financially. In addition, EDUQUEMOS has built on the contacts it has made regionally though PREAL-sponsored events to organize study Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 60 trips for members of the private sector to see how technical education is administered in other countries, particularly El Salvador, and to arrange for business leaders in the region to visit Nicaragua. One project that EDUQUEMOS began and then withdrew from highlights some of the general difficulties that are entailed in organizing the business community. EDUQUEMOS was involved in creating a school adoption program jointly with the American Chamber of Commerce (AMCHAM). Members of AMCHAM—businesses that either are American corporations such as Coca-Cola or that have interests in the United States—can work with a local primary school of their choosing and tend to be engaged in activities such as donating equipment, fixing up crumbling infrastructures, or sometimes being role models for students. Initially, the adoption program was jointly organized by EDUQUEMOS and AMCHAM. Disagreements over how to organize the business community’s participation and what role should be played by the Nicaraguan Ministry of Education (MECD), however, contributed to EDUQUEMOS’ decision to withdraw. Strengthening Public and Private Sector Organizations Working for Educational Improvement USAID funding through the PREAL program has had mixed results in terms of strengthening organizations in the public and private sectors. The most notable success has been with IDEUCA. As a direct result of the funding, a number of important civil society organizations working towards educational reform have been created via IDEUCA. One organization, the Movimiento Pedagógico, is a teachers’ organization that was founded by participants in one of IDEUCA’s policy analysis courses. The Foro Educativo is another organization developed out of programs at IDEUCA. Others work with union members. The point, here, is that IDEUCA has been very successful in organizing new groups in civil society to have a voice in the educational reform process. IDEUCA does not appear to be sustainable in the medium run without continued external financing, however. Although they have recently become a university-based educational policy center, the university’s expectation is that the center will be self-financing. This will be difficult. Superficially, it seems that the organization should be capable of garnering additional funds since it has developed and recognized research and training programs. It is doubtful, however, that it or any similar organization in a developing country can find enough international contracts to do research that can sustain the institution. As one respondent noted, although research is a valuable undertaking and one that should be funded, expecting an organization such as IDEUCA to be able to survive on research alone is an unrealistic assessment of the funding environment. He was quick to point out that this is an argument for continued funding. Certainly, however, some funding for research is likely to continue, although it may be for very different topics than those envisioned by the PREAL program. Since there are relatively few research centers in Nicaragua and IDEUCA has a track record and experience in dealing with external agencies and organizations, there is likely to be some continued demand for its services. In research-for-hire situations, the research agenda is that of the funders and may or may Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 61 not include work on education reform. In terms of the training courses and the plans under development at the IDEUCA to create a master’s degree program in policy analysis, there are also limits to viability. Given the likely clientele of the policy analysis program, self-funding would mean that only a very modest tuition could be charged; not one that is likely to cover the costs of the program. There are also, at least in the short run, potential problems with attracting Ministerial contracts to do training of teachers, administrators or staff, who would be a logical source of students and funding. Some in the Ministry of Education, Culture and Sports wondered whether the government and IDEUCA have a shared vision. Without an understanding that they are both working towards the same goals, it may be difficult for IDEUCA to attract state contracts for training personnel. In the case of EDUQUEMOS, the investment has been sufficient to allow it to manage a very small office and to organize activities, albeit on a very limited budget, given its aims. At the time of the evaluation, however, members of the organization felt that it would not survive without continued external funding. The difficulty for this organization is that, at this point, the local business community is neither willing nor able to assume support for it (although such support several years from now is a goal of the members). In addition, they are too small and without the appropriate skill set to compete for money to conduct studies themselves. They do undertake grant proposals to put together studies, where they contract researchers and supervise the production of studies, but the volume of activity they are likely to develop under this model is far too small to allow them to generate sufficient overhead to survive as an organization. One notable omission to the work of strengthening institutions to undertake educational reform in Nicaragua has been work with the National Assembly. A number of respondents felt that because the National Assembly is the body that ultimately passes legislation, it is a key group to receive assistance in order to education them, move their thinking towards how reform should take shape, and inform them about how important it is to increase budget allocations for education. Interviewees pointed to the lack of attempts made to educate and strengthen the capacity of members of the legislature, particularly those on the education sub-committee, as a failing of the PREAL program. They also noted that deputies need such education because they tend not to have expertise in the area and yet often have responsibility because the Education Committee is seen as a low-prestige assignment and members tend to focus their attention elsewhere. Identifying and Disseminating Best Education Policies and Practices As noted above, there has been an emphasis on disseminating the work produced by Nicaragua rather than PREAL as a whole. In general, fewer people among the respondents could point to documents than had participated in activities. In part, this seems to be related to IDEUCA acting as a gatekeeper of information. One group of respondents stood out in terms of not receiving PREAL information: those in the Ministry of Education, Culture and Sports at the policy implementation level, who were much less Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 62 likely to know about or have seen documents from PREAL central offices. Even where they were aware of them, they did not receive them regularly. One issue this raises is the lack of clarity about the mechanism of distribution of centrally produced PREAL documents. If it is not feasible to send them directly from Chile, then there needs to be some follow-up with the Affiliated Centers about how documents are distributed. This could be a task that the regional office in Honduras would want to take up. The responses about information dissemination garnered from interviews also suggest that the production and dissemination of documents is viewed as useful for only a subset of those that PREAL is trying to reach. For example, business leaders generally acknowledged that they had not read documents on best practices; nor were they likely to, given their schedules and lack of time. In addition, some of those familiar with PREAL publications considered them to be less useful than those they received from other sources. In particular, people who were more associated with popular organizations or groups within civil society considered the information they received from PREAL to be too theoretical and not practical enough for it to be useful to them. One respondent, for example, seemed to be asking for more ‘how-to’ types of information, i.e., including with summaries successful reform efforts from other places, specific suggestions and concrete steps about how one could implement those changes in Nicaragua. Those individuals with a more academic background, however, tended to see the information produced by PREAL as both useful and more helpful than other sources, in part for precisely the same reason that some groups criticized it: its global perspective. One top-ranking policy maker, an ex-Minister of Education, also was enthusiastic about the information produced by PREAL. Seeing the document, El Futuro Esta en Juego, early in his tenure as Minister was an important event for him. It crystallized ideas that he already had about the problems of education and was one of the forces influencing his decision-making. Given both the importance that PREAL attaches to influencing policy making and this clear indication of how, under the right circumstances, PREAL information can do this, it is particularly noteworthy that, as mentioned above, most of the interviews with Ministry of Education, Culture and Sports members revealed that they do not receive, let alone read, PREAL’s information about best practices or reform. To the extent that they have the ability to enact policy, the failure of the information to reach them hampers the ability of PREAL to be a notable influence. Finally, the interviews conducted showed that regardless of the person’s sector, there is very little knowledge or use of the PREAL website. Only those who are most closely involved with the associated centers know of it or use it. In part, this is due to the lack of access to the Internet that characterizes much of the country. In this sense, Nicaragua is still at least several years away from a situation where the website could be an effective tool of information dissemination. Even then, however, there is some doubt that it will be as useful as it might since even many of those with access to Internet connections did not express great interest in using the site. Providing Effective Policy Reform Strategies and Activities Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 63 If PREAL’s dissemination of best practices in Nicaragua cannot be considered to be optimal, some of the other activities they have organized to promote reform and dialogue have been quite successful. Within the business community, for example, leaders noted that they had found visits and meetings organized at the level of Central America especially useful. A number talked about visits to FEPADE (Fundación Empresarial Para el Desarrollo Educativo) in El Salvador, a business sector educational foundation, as extremely valuable in terms of giving them ideas about how the private sector could organize technical education. After making initial contacts at a Central American meeting organized by PREAL, some members of the business community visited El Salvador and subsequently there were visits by their Salvadoran counterparts to Nicaragua. Another activity that respondents spoke of in extremely positive terms were courses and workshops organized by IDEUCA, particularly those focusing on policy. Not only did these lead to the creation of new organizations as noted above but they also helped to strengthen the ability of participants to become engaged in the policy reform process. For example, members of some of the organizations who were represented on the advisory groups or working groups of the National Plan spoke of being better able to contribute to the reform process because their capacity to write proposals for reform had improved as a result of having participated in the courses. There are some institutional constraints on PREAL’s ability to offer policy reform strategies. One theme that emerged in many conversations about obstacles to reform in Nicaragua was the short duration of education ministers. Not only is this problematic because of the turnover of officials and the time it takes for new ones to come up to speed about issues, but it also leads to policy discontinuity, even when the ministers are of the same party. As one interviewee observed, all ministers want to put their own stamp on policy so that there is a tendency to change previous policy, sometimes for the sake of change alone. Improving Educational Quality and Equity While there is no direct evidence that PREAL has had an influence on improving quality or equity in the context of Nicaraguan education, much of the work of both associated centers focuses on those topics. As a result, they are laying the groundwork for improvements by putting the issues on the policy agenda. In publications of both centers, there is an emphasis on the linkage between the poor performance of the system and the need to train and pay teachers better, to retain more students, and to make education more relevant to the needs of the economy and communities. One of the great tasks ahead that will likely engage PREAL in the future is creating the political consensus to fund reforms that will improve quality and equity. So far, this consensus does not exist but the discussion could be taken up in the context of the government’s discussions with donors on its poverty reduction strategy. Fostering Utilization of PREAL Recommendations and Findings by Beneficiaries Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 64 One of the areas where PREAL appears to be somewhat deficient in Nicaragua is in its ability to carry out follow-up activities to the meetings and conferences it holds, which diminishes its effectiveness in promoting the use of its recommendations. For example, many of those who had attended regional conferences felt that while those were very valuable, the lack of follow-up afterwards in-country meant that they were able to do little in the way of implementing ideas gotten at the conference. In addition, it appears that mechanisms to implement PREAL recommendations or ideas are lacking. For example, there is overwhelming agreement across all sectors interviewed that a central problem facing education in Nicaragua is the dual problem of low salary and low performance of teachers. Any solution to this will demand participation by the teachers’ unions. Here, it would be useful for the information being produced and discussed by the PREAL working group on teachers’ unions to be disseminated in Nicaragua. However, the lack of vertical integration between the working groups, which are based in Chile and Peru, and the associated centers in Central America is evident since in Nicaragua the latter seemed unaware of the work of the former. Another case where mechanisms have not been found to use PREAL findings effectively is in the case of EDUQUEMOS and the school adoption program by businesses. While EDUQUEMOS knew of and wanted to promote a model of school adoption in the Dominican Republic they had learned about at a PREAL event, they were unable to convince either their partners at the time (AMCHAM) or the business community of the applicability of the model to the Nicaraguan context. Providing a Viable Structure The organizational structure of PREAL in Nicaragua represents trade-offs in terms of the type of presence it can have and the use of scarce resources. In order to evaluate whether the trade-offs have been acceptable, the questions that must be considered are: what are the target audiences to whom PREAL wishes to address its message and advice; are they reaching those audiences; and are there other structures that could achieve the same results more effectively? At the level of the affiliated centers, the trade-offs can be summarized in the following way. In the case of IDEUCA, joining forces with them allowed PREAL to tap into an existing research infrastructure of an organization that has a high degree of ability to convoke and organize important segments of civil society. The costs of establishing such an organization from the ground up would have been impossible for PREAL, and the collateral benefit of being associated with a respected academic has been high. However, As an organization with its own identity and history and one that is particularly linked to a defined position within a highly polarized society, however, IDEUCA does not always necessarily coincide with the views of PREAL and it likely limits the ability to get across the message of PREAL. It is a case of the messenger being unacceptable to certain policy makers, although the efforts of IDEUCA to moderate its image may mean that this is less problematic over time. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 65 In the case of EDUQUEMOS, it was seen as necessary to create an organization to achieve the important task of organizing the business sector since no institutions existed to do that. However, the investment in EDUQUEMOS does not appear sufficient to have been able to generate an organization truly capable of pushing forward the policy dialogue. The experience shows the difficulty of creating an organization from the ground up since funding an office and small staff is not inconsiderable in financial terms but still far from adequate to produce a thriving endeavor. In some ways, it is an organization whose total is less than the sum of its parts. Stated differently, EDUQUEMOS has been involved in several important policy issues, such as organizing and financing technical education, getting education onto the agenda of COSEP, the most important employers’ association;, and changing the tax code to make charitable donations to educational institutions deductible. It appears, however, to have lost some leadership in many of these issues. This is an example of where management assistance from PREAL could be useful in helping them refocus on other issues (including those which it had tentatively set out exploring), and gain credibility as an interlocutor for the private sector. It would be very useful for PREAL to see what strategic assistance the office in Tegucigalpa could offer to EDUQUEMOS that would allow them to devise an integrated package of themes and priorities for working with the private sector. The structure of PREAL’s activities in Nicaragua is such that the Central American Commission has little relationship to the associated centers. The members of the Commission are chosen for their high visibility but, with one very notable exception, the members tend to be fairly inactive. If the Commission had its own budget and its members were willing to use their personal prestige for the promotion of its report, then the lack of integration would be less noteworthy. While the associated centers recently did an excellent job of launching the Commission’s report Mañana es Muy Tarde, better integration of PREAL’s activities might bring more coherence to the task of putting education reform onto the policy agenda. The structure of the PREAL program in Nicaragua also raises questions about the audience it wishes to target. Because of the nature of the associated centers, the focus is on the business community and civil society organizations, and there is at present, less emphasis on high-level policymakers or policy implementers, that is bureaucrats. Certainly, limited resources means PREAL cannot hope to reach all possible stakeholders and decision makers in the policy process in any given county but relying on the model of associated centers means relying on their power to convene stakeholders. Depending on the groups the centers are close to, this strategy may or may not correspond to reaching the most effective groups. Recommendations In keeping with the findings of the country visit, several recommendations can be made. PREAL needs to better define its target audience in Nicaragua. If policymakers are key Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 66 targets, PREAL could consider putting more resources into reaching them. While recruiting high profile individuals to champion the goal of reform is a good strategy, PREAL needs to make sure that they are willing to use their positions in order to exert influence. Therefore, a better selection of individuals is needed and more work with them in terms of devising useful strategies of influence within the country. The Technical Secretariat of the Presidency is another area where PREAL could target policymakers since this body is working on a Poverty Reduction Strategy, where a focus on education reform would make sense and could have some impact. Put more emphasis on working with legislators. Because they are key actors in getting reforms passed, they are an essential element of the process. Especially in terms of increasing their professionalization and helping them to develop expertise in the area of education, assistance in terms of drafting legislation and familiarization with models of legal frameworks for education from other countries, could be extremely useful. Work with parties and candidates in upcoming national elections. One of the lessons of the Nicaraguan case is that turnover in the Ministry is an important impediment to education reform, even when the party of government does not change. It is therefore vital to institutionalize the process. One way to do that is to work to put education reform onto the agendas and platforms of all parties and candidates. Particularly important is the need to develop the political will to increase funding for primary and secondary education, and change the constitutionally mandated six percent of the budget that goes to higher education. Take advantage of a new opportunity to work with IDEUCA on policy reform. The transformation of UCA-PREAL into the IDEUCA center, with its focus on training in educational policy and analysis offers PREAL a chance to influence the policy process by getting involved in training policy implementers and decision makers. PREAL might consider working with IDEUCA in a number of ways: helping IDEUCA develop curriculum for its policy analysis courses, supporting the development of new courses or funding individuals to attend the courses and faculty to teach the courses. Not only would such a strategy help disseminate best practices and results coming out of the study groups to large numbers of people who may help influence the policy reform process but it would also be advantageous to IDEUCA. IDEUCA would benefit from the infusion of resources and the participation of PREAL could make the courses more attractive to a larger number of people who view PREAL as a neutral and well-informed voice on education policy reform. Consider organizing a multi-partisan policy workshop in the United States to create consensus after the national elections. A recently completed evaluation by the World Bank of its education activities in El Salvador showed that one of the most valued and effective events was a high-level policy dialogue meeting held in Cambridge, Massachusetts, which brought together the Minister of Education and members of the governing party, opposition political leaders, members of the legislature, members of the private sector, and NGO leaders. The meeting consisted of presentations by experts on Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 67 key topics in education and on the experiences of other countries undergoing reform as well as professionally facilitated group discussions that sought to develop concrete proposals for moving reform forward in El Salvador. PREAL and USAID could use a similar model for Nicaragua. Hosting such a meeting shortly after the national elections would allow the new administration to shape its education agenda with greater support. Better articulate private sector issues and strategy. PREAL might consider examining the disparate issues and activities that currently characterize private sector involvement in education dialogue such as school adoption, the organization and structure of INATEC (technical education), tax incentives for educational donations, and the role of COSEP and INDE in pushing for reform in an attempt to link these better and more systematically to concrete proposals. PREAL could either work with EDUQUEMOS to manage these issues better or it could consider a more informal structure, such as a committee of actors from the private sector that would meet periodically and interact with counterparts in the region. If PREAL chooses to work directly with EDUQUEMOS, this is an area where the regional office in Honduras should become more active. Relations between the two groups are very good, and management assistance about which issues to focus on would be useful and productive. Rethink the strategy of using the Internet and costly publications. In countries of Central America such as Nicaragua where Internet use lags behind, where there is no tradition of using the internet to gain information, or where the internet is not available for most of the employees in the public sector, other strategies such as field visits might be more appropriate and could be complementary. Because one criticism of visits and conferences was lack of follow-up, these should only be undertaken when a strategy of in-country continuity can be provided. Put publications to work in targeted settings. One of the problems with the PREAL publications is that they are not widely distributed and where they are received, there is not a clear context of how to use them. PREAL might consider working with IDEUCA or other organizations that are offering courses and certificates on education policy to design curricula and exercises based on PREAL publications. This would have the benefit of reaching audiences such as teachers’ organizations, school administrators, and bureaucrats and habituate them to using these documents. In the case of IDEUCA, it could have the added benefit of helping them continue the process of defining their image as an education center. Another area where PREAL might want to focus its document strategy is reaching Ministry of Education, Culture and Sports personnel. Currently, they do not tend to receive PREAL documents. Targeting particular groups in the Ministry and other governmental offices dealing with educational policy for workshops might be one strategy for encouraging the use of PREAL resources but this would need to be combined with a better distribution mechanism of publications if they are to be of ongoing use. Since many in the Ministry do have access to the Internet, some consideration could also be given to working with them to on how to use the website. It might be useful to see how Ministerial personnel in other countries work with the website and take that information as the starting point of a training or user’s guide for bureaucratic staff. This Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 68 latter strategy could be combined with support for IDEUCA’s policy analysis program. At all levels, establish a better dialogue between USAID and PREAL. The associated centers could, with a little effort, improve the flow of information with the Mission by informing them about upcoming activities and projects. Within Nicaragua, increased dialogue between PREAL and the BASE project, which has an important presence within the Ministry of Education, Culture and Sports, could enhance the utilization of PREAL’s recommendations. The regional PREAL representative could play a stronger role in facilitating communication between the associated centers and USAID, not only about upcoming events but also about each center’s goals, objectives, projects, and work plans. PREAL in Washington also needs to recognize that the country-level Mission should be kept apprised directly of its visits to the country and regional-level activities as they affect the country. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 69 II.4. PERU Approach The Peru site visit was conducted from December 4-9, 2000. There are two PREAL Associated Centers in Lima, both of which were visited: the Grupo de Analísis para el Desarrollo, GRADE, which is also the institution hosting one of the Program Working Groups and the center Foro Educativo. In-depth interviews were held with Patricia Arregi, Coordinator of the WG and member of the Executive Committee, and with Veronica Villaran, who is responsible for assisting the coordination and management of the Working Group’s website (http://www.grade.org.pe/gtee_preal/gteepreal.htm). A third member of GRADE, the Research Director and Advisor to the Quality Measurement Unit of the Ministry of Education, Santiago Cueto, also was interviewed. Interviews were also held with members of Foro Eduativo. Ten additional individual interviews and three group interviews were carried out using the survey instrument. Interviewees included members of the private sector, universities, the Ministry of Education, and a donor organization. A meeting was also held with Dick Martin, Kristin Langlykke, and Walter Twanama, representatives of the USAID Mission in Peru. Synthesis of Main Findings Based on these interviews, the evaluator made several qualitative observations specific to the role of the Working Group on Evaluation and Standards and to the “presence” of PREAL in Peru. Background information is presented in the sections below. Other findings are presented under each subheading in point form. As with the other case studies, the reader should set the findings in context by keeping in mind that PREAL’s success and impact cannot be measured linearly and that PREAL’s success and impact depend upon the relationships the Program has developed with its in￾country partners. In this light, the following general findings are identified: 1. There is general consensus that PREAL as a regional Program operating in Peru contributes to raising awareness about issues of importance to the national education reform debate. PREAL does this by: ß informing key actors among the private sector and in the Ministry of Education; ß disseminating current and relevant information about experiences in the region; Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 70 ß maintaining a Work Group on a topic of critical importance to the country’s education system; and ß promoting the involvement of key private sector groups in the debate about education reform. 2. PREAL’s presence in Peru is primarily known in relation to the activities of the Working Group on Evaluation and Standards coordinated from GRADE. This is reflected in: ß the understanding of key informants about critical issues affecting education; and ß the knowledge of key informants about PREAL publications and activities. 3. There are strong indications that in Peru PREAL has made important progress in raising awareness about education reform issues and reaching key members of the private sector. This is reflected in: ß the interest expressed by representatives of private sector organizations (i.e., Peru 2021, Grupo Macro, and IPAE Aporte Empresarial) to collaborate with PREAL around education issues of national importance; and ß the interest in using PREAL/Working Group materials as a key information source. 4. There is indication that in Peru PREAL is making important progress in reaching key actors in the Ministry of Education. This is reflected in: ß ongoing linkages between the Working Group GRADE and the Special Program for the Improvement of the Quality of Peruvian Education (MECEP); and ß the use of publications and materials produced by the Working Group by units within the Ministry of Education 5. PREAL linkages and forms of partnerships with Associated Centers in Peru are uneven. This is reflected in: ß differences in perception about the degree of interest of the Program’s central coordination in maintaining the relationships established with GRADE and Foro Educativo; and ß perceptions of the capacity of the Program to maintain funding for the Associated Centers. An apparent lack of clarity in this regard has created unfulfilled expectations. 6. Current political conditions in Peru seem to offer an important window of opportunity to PREAL to strengthen its role in promoting dialogue around education reform issues. There is evidence suggesting that: Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 71 ß there is a strong perception that the government is now more open to dialogue about education, particularly in aspects related to quality, standards, and improvement of teacher and school conditions; and ß education is one of the issues attracting public attention as part of the agenda of the current government. Building Public and Private Sector Support for Education Reform There is ample evidence that PREAL has made important progress in Peru in reaching its objectives of building public-private sector support for educational reform, and strengthening public-private sector organizations. Limitations, however, have existed due to a number of factors. There is a general opinion that over the past several years, conditions were not favorable for building and promoting dialogue about the education problems affecting the country. To many, this meant a lack of opportunity for debate and a lack of interest among public officials to engage civil society. The socio-political, economic, and demographic makeup of the country also contributed to limiting existing capacities to deal with the magnitude and complexity of the education problems. An added element influencing the effectiveness of PREAL in this context has been its limited resources to operate in Peru beyond the activities of the WG and difficulties in consolidating the Program’s partnership approach with one of the Associated Centers (Foro Educativo). Associated Centers in Peru PREAL identifies two Associated Centers in Peru: the Grupo de Analisis para el Desarrollo, GRADE; and the center Foro Educativo. GRADE is the host institution for the Working Group on Evaluation and Standards. Both are private institutions. The Working Group on Evaluation and Standards is an ongoing activity that produces a number of publications, organizes events, and maintains an internet site. According to information provided by the PREAL-Santiago Coordination, between 1998-2000 a total of US$440,016 has been allocated to this WG in two grants (US$175,016 and US$265,000). (These funds are for activities through April 2002, and include non￾Working Group activities, such as the costs of distributing PREAL documents in Peru, and of PA’s attendance at PREAL ExCom meetings.) Overall, the activities of this Working Group are successfully contributing to filling an important gap in the regional education debate. The Working Group is also increasingly influential in the debate about educational quality in Peru. People interviewed identify PREAL with the activities of the Working Group more than with the Program’s regional activities. This enhances the potential impact of the Working Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 72 Group at the national level but at the same time risks overshadowing the regional work of the Program carried out outside Peru. Among the key informants interviewed, only those related to academic and research institutions were fully aware of PREAL’s regional activities. Several signs of success were found in connection with this initiative: ß The Working Group has been successful in developing an identity with the issue of educational evaluation and standards, which is well recognized and respected among those interviewed. ß The Working Group is well managed and effectively organized in terms of dissemination of information, maintaining a network of contacts both nationally and regionally, and becoming a key source of information on the topic. ß The work of the Working Group is recognized by the quality and currency of the information it disseminates and is likely to face added pressures to become a broader source of information about education reform in the context of Peru’s national education debate. ß The work of the Working Group has contributed to enhancing the role of the hosting institution (GRADE) in participating in the national education debate and identifying new priority issues for its research agenda. PREAL’s relationship with Foro Educativo shows important differences from the partnership established with GRADE. Foro received a grant from PREAL for US$68,800 to carry out activities in promoting education reform. Although with some delays, this activity was concluded satisfactorily in the perception of Foro Educativo. No further collaboration has taken place since then. There are indications that Foro Educativo had expectations about continuing its work with the Program that were not fully met. In part, this seems to have resulted from expectations about further funding and in part from differences in terms of Foro’s own institutional priorities about education in the context of Peru. Specific reference was made to the fact that in 1997-98 Foro was invited to submit a proposal to PREAL for continuing funding. However, for reasons unclear to Foro, the negotiations were not successful. These led to a discontinuation of the collaboration between Foro and PREAL to the extent that the representatives of Foro expressed a lack of awareness about current PREAL regional activities and only general knowledge about the PREAL publications and the regional activities of the Working Group. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 73 Foro sees itself as a key actor in the national education debate, with a strong network of contacts and a solid reputation, particularly given its emphasis on civil society involvement. In this light, it sees itself as a “natural” PREAL partner. The collaboration between PREAL and Foro Educativo highlights the following: ß PREAL’s success in its modus operandi with most Associated Centers depends to a large extent upon the ability of the Program to maintain funding for specific activities; ß There are indications that Foro Educativo saw its partnership with PREAL primarily as one between a funding organization and an executing agency around specific activities; ß There are indications that PREAL unwittingly created funding expectations in Foro Educativo that when not met affected the relationship established with this Associated Center; ß There are indications that the relationship between PREAL and Foro Educativo was further affected by a lack of followup from both parties. ß Given the profile and role of Foro Educativo in the Peruvian context, there is potential and value for PREAL to re-establish collaborative relationships. Private Sector Partnerships The relationship between PREAL and key actors of the private sector in Peru is strong, offering multiple opportunities for further collaboration with important potential impact at the national level. Key informants from the private sector and the university community expressed a great degree of interest in collaborating with PREAL, as they see issues related to the quality of Peruvian education a major area of concern. Representatives from universities saw in PREAL a key source of information, filling a gap in terms of materials available to students and researchers. Particular reference was made to the information the Program makes available about experiences in other parts of the region. Respondents from the entrepreneurial sector were explicit in expressing their need to receive technical advice in education matters and in establishing forms of partnership that could help their organizations to participate in the national education debate by proposing “concrete” proposals for educational change. Several of them acknowledged receiving information from the Program, in particular from the Working Group, and using this information in their own debates. They saw PREAL as a means of bringing together different areas of expertise and facilitating an expanded Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 74 dialogue between the private sector, government officials, and education experts from the region. Relevant findings in this regard are: ß Representatives from private sector organizations expressed their willingness to broaden their collaboration with PREAL around key education issues, particularly educational standards, quality of education, and quality of teachers. ß Most informants receive and use some of the PREAL publications (particularly the newsletters) but feel that PREAL could disseminate information that better addresses their sector’s needs concerning education. ß PREAL can play a more direct role in facilitating dialogue between politicians and representatives from the private sector and pursuing “strategic alliances” to advance an education reform agenda. Public Sector Partnerships Officials from the Ministry of Education saw in PREAL a critical source of technical information and advice not available through other means. They acknowledged using the PREAL publications and information that resulted from events organized by PREAL in the region. PREAL information about evaluation and standards and about best policies and practices were identified as the most frequently used. Ministry officials identified the work of the Working Group as timely and highly relevant to the current education debate about the need to improve the quality of teaching and learning in the country. References also were made to the work of PREAL’s Working Group on Teachers, as this sector was identified as one in urgent need of examination. Giving the changing political conditions in the country, several aspects were found relevant to the future role of PREAL: ß The expansion and consolidation of PREAL relationships with various levels of the Ministry of Education, as there is a great need for technical advice. ß The broadening of PREAL activities in Peru beyond the role of the Working Group on Evaluation and Standards to address other issues such as rural and bilingual education, quality of teacher training, school management and administration, and civic education. ß The creation of a “pool of experts” from other parts of the hemisphere to assist Ministry officials in need of technical expertise. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 75 Partnerships with International Organizations The evaluator interviewed a representative of one international organization (GTZ) in Peru. Several informants, however, voiced their opinion about the role they see PREAL playing in relation to some of the major donors operating in Peru. In general, there is a perception that PREAL plays an important intermediary role between the education agendas of donors and international organizations, and the national education debate. At the same time, a few saw PREAL as “too close to the major players’ agendas” (i.e., the World Bank and the IADB) and without sufficient roots in the problems affecting education in Peru. Overall, however, PREAL was perceived as playing a valuable role in synthesizing information about education in the region and disseminating information about current education issues in the international sphere. Observations made in this regard include: ß The need for PREAL to “maintain a distance” from major donor agencies as a means to protect an independent technical view about educational change; ß The increasing opportunities for PREAL in the current context of ongoing political changes to influence education policy; ß The positive attitude toward collaboration among representatives of donors and international organizations operating in Peru given the work and achievements of PREAL; ß The need to maintain the flow of high quality information but paying more attention to audiences other than policymakers; and ß The need to organize more frequent “middle level” meetings between Ministry personnel and representatives from donors and international organizations (In other words, act as a mediator?). Strengthening the Capacity of Public and Private Sector Organizations Working for Educational Improvement Overall, there was a great degree of consensus about the contribution of PREAL and the Working Group in building technical capacity for education reform. Almost all respondents pointed out the particular needs for technical expertise that exists in the country and the role that PREAL could play in facilitating contacts with other institutions and experts from the region. Many agreed that the greatest potential impact in terms of capacity building of a regional program like PREAL is at the individual rather than the institutional level. This view was often present among informants from the private sector. Although there is agreement that Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 76 PREAL initiatives to create a regional WG and work with selected institutions is a step in the right direction, there is the opinion that this approach is limited in terms of its scope and potential impact given the education problems in the country. One of the major gaps identified across the board by the respondents was the lack of technical expertise within the education sector, including the Ministry of Education. This situation becomes more critical outside the main urban centers. Some of the suggestions about the capacity building role of PREAL included: ß Reaching a broader range of key audiences, particularly outside Lima; ß Targeting the teaching profession and teacher organizations, and facilitating links between these and universities; ß Establishing a series of activities oriented to officials working in the education sector, and representatives from the NGO community on aspects related to policy dialogue and analysis. ß Playing a greater role in facilitating contacts and meetings with experts from other parts of the region and within the country between government, the private sector, and representatives from the civil society. Identifying and Disseminating Best Education Policies and Practices Most respondents know about PREAL through its publications and to a lesser extent through PREAL-sponsored meetings. An important caveat in the case of Peru is that although the majority of the respondents received the publications that PREAL distributes regionally, most reference is made to the materials produced by the Working Group on Evaluation and Standards. This responds in part to the fact that the coordination of the Working Group is based in one of the prestigious research centers of the country, and in part to the fact that some of the materials produced by the Working Groups do not follow the general format of the PREAL publications and are therefore not viewed as publications of the Program. Among the PREAL publications most frequently cited are the Best Practices Newsletter and the Document Series. A smaller proportion of respondents also are aware of and use the materials posted on the Internet site. As with the publications, however, this also tends to be identified with the site of the WG and GRADE rather than with the PREAL site. Among those aware of PREAL regional activities, respondents identified the availability of information about education in other countries as one of the main contributions of PREAL in terms of dissemination. PREAL was perceived to be an important source of information to identify experiences that could be replicated in the context of Peru, a means to identify sources of expertise, and a means to complement other international sources of Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 77 information about education reform. Regional meetings also were identified as an important dissemination activity, although knowledge about specific activities was limited. Several findings are relevant in terms of PREAL dissemination activities: ß Predominantly, PREAL dissemination activities in Peru are channelled through the activities of the WG and the role of GRADE as an AC. This results in a limited awareness about the Program regional activities and publications. ß Not keeping a standard format in some of the dissemination mediums (i.e., some publications, internet site) contributes to the identification of PREAL activities and achievements with those of the WG managed from GRADE. ß The most effective means to disseminate information identified by the respondents was the newsletter format. ß The Internet site of PREAL and the site maintained by the WG were considered to be useful tools but of limited reach outside those areas where there is available connectivity. ß Some respondents identified a lack of PREAL presence in the media as an important gap in the efforts to further educational reform debate. Thematic Focus Improving Educational Quality and Equity Overall, PREAL’s focus on education reform is perceived to be a central topic in promoting educational change. The emphasis on issues of quality of education and particularly on evaluation and standards are considered to be at the core of the current and future education debate in Peru. Without exception, the respondents identified the issue of poor quality education as one of the major obstacles facing the country and a critical issue that needs urgent attention by the government and the society in general. Other Issues Several other issues were suggested as critical to improving education in the country and as possible consideration in the PREAL agenda: ß Teachers. There was consensus that teacher training, qualifications, salaries, and demands must be at the forefront of any attempt to improve education in the country. There is a need to address the conflicting relationship that exists between the government and teacher organizations and the poor conditions and performance of teachers, particularly in the more remote parts of the country. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 78 ß Lack of Political Will and Lack of National Consensus. One of the major historical obstacles to education reform is the lack of political will to acknowledge problems and take leadership in changing the state of education, and the adversarial relationships between government and civil society. Although this was associated primarily with the previous administration, there is still a high degree of uncertainty about the future. ß Centralization and Lack of Flexibility. Many perceive the education system as very centralized and subject to a “culture of government control.” This is reflected in the ways in which the system operates, as most decisions are made in Lima. The results are a lack of information and government incapacity to implement decentralized monitoring systems. ß Lack of an Education Policy and Vision. To a large extent, the failure to improve education is associated with a lack of vision about the future of education and a lack of modern policies to manage the system. ß Lack of Information and Transparency. This is an issue often associated with the previous administration. Many indicated that in order to change education there is a need to know first what the “real problems” are. In this regard, there is a perception that adequate information about the present state of education in the country is not available, and that government officials are reluctant to make information public. ß Financing of Education. There are insufficient resources allocated to resolve the existing problems in education. This area needs priority attention in any attempt to reform the system. ß Rural and Bilingual Education. Several respondents identified the lack of attention to rural and bilingual education as a critical underlining issue affecting quality of education in the country. Sustainability The issue of sustainability is central to PREAL’s role in building capacity for education reform. In the context of PREAL activities in Peru, there is evidence to suggest that the Program needs to: ß Carefully target and focus its capacity building activities in order to maximize its limited resources. ß Develop means by which to build on the achievements of its various forms of partnerships in order not to make these solely dependent on the availability of funding as in the case of Foro Educativo. ß Explore more concrete forms of partnership with the private sector in the form of joint funding of selected activities. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 79 Recommendations Reach and Impact Given the existing conditions in Peru, PREAL has been successful and continues to move in the right direction to meet its objectives of building public and public sector support for educational reform. Under the new political conditions in the country, the Program could further strengthen its reach and impact by: ß Expanding selectively its scope to address other areas of critical importance to the current education reform debate. ß Adopting a more proactive approach in targeting policymakers and key representatives of the private sector and the civil society. ß Promoting more frequent seminars and workshops on issues of national relevance with representatives of different sectors. ß Making better use of its regional comparative advantage to enhance linkages between Peruvian and regional sources of expertise. ß Exploring the possibilities of strategic alliances between the Program and the private sector around issues of mutual concern. Partnerships and Public and Private Sector Support The capacity building role of PREAL could be enhanced through more careful attention to establishing and nurturing partnerships. In this regard PREAL should takes steps to: ß Clarify and streamline its approach to working with local Associate Centers as a means of strengthening collaboration and follow-up of activities initiated by local institutions. ß Address up front and in a systematic manner the issue of sustainability of the relationships established with Associated Centers and Working Groups to avoid creating unfounded expectations about the funding capacity of the Program. ß Collaborate more closely with international organizations sharing similar interests and priorities as a means to optimize efforts and maximize resources. ß Expand its collaboration with key units within the Ministry of Education. ß Develop means by which to consult and expand the participation of Associated Centers in PREAL decision making processes to strengthen the sense of ownership among partner institutions. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 80 Dissemination of Information The Program has been successful in disseminating information about the activities of the WG and to some extent about the regional activities of PREAL. This role could be further enhanced by: ß Improving the coordination of dissemination activities between the central coordination of PREAL and the role of the local Associated Center and the Working Group. ß Finding ways to adapt some of its materials to audiences other than policymakers. ß Further standardizing the format and presentation of the Program materials and Website as a means to increase local awareness about PREAL activities. ß Increasing its presence in public events and national debates in collaboration with government, the private sector, and other institutions working on education. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 81 APPENDICES Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 82 APPENDIX A. FINAL WORK PLAN Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 83 Submitted by: Dr. Donald Graybill, BEPS Project Director Ms. Antonieta Harwood, LAC Task Order Coordinator BEPS Activity November 24, 2000 FINAL WORK PLAN: BEPS/PREAL EVALUATION SUB-TASK ORDER under the LAC/RSD-EHR “Improved Human Resource Policies” BEPS Task Order Submitted to: Ms. Susan Morawetz, PREAL Sub-Task Order Manager LAC/RSD-EHR USAID Washington, DC Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 84 CONTENTS 1.0 BACKGROUND 1.1 BEPS 1.2 LAC Task Order/PREAL Sub-Task Order 1.3 Resources and Limitations 2.0 WORK PLAN 2.1 Target Completion Date: 2.2 Deliverables 2.3 Supervision 2.4 Summary of strategic approach 2.5 Key Design Elements 2.7 Contact and Coordination 2.8 Key Research Questions: 2.9 Countries to be visited: 2.10 Data Sources 2.11 Key Methodologies 2.12 Schedule of Sub-task Order Activities: 3.0 PREAL EVALUATION TEAM 3.0 Team configuration 3.1 Key qualifications, selection criteria 3.2 Consultant team and summaries 4.0 BUDGET Appendix A final draft: SOW for an evaluation of the Partnership for Education Revitalization in the Americas (PREAL) Activity Appendix B CVs of Proposed Consultant Team 1.0 BACKGROUND Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 85 1.1 BEPS The Basic Education and Policy Support (BEPS) Activity is a worldwide, five-year, IQC￾type, basic education project funded by the U.S. Agency for International Development (USAID) and managed by USAID's Global Bureau/Human Capacity Development Center (G/HCD). The BEPS Activity provides specialized professional and technical services to USAID missions and regional bureaus, and carries out educational activities on behalf of USAID in non-presence countries. BEPS consults and collaborates with other USAID global projects, international donors, and organizations. The activity seeks to assist developing and newly independent nations to improve the quality, access, management, and effectiveness of their educational systems, particularly basic education. BEPS is being implemented by Creative Associates International Inc. (CAII), in partnership with CARE, The George Washington University and GroundWork. This team has the expertise needed to respond successfully to the professional and technical needs of USAID in a way that leads to equitable education, human capacity development, meaningful participation, and lasting change. 1.2 LAC Task Order/PREAL Sub-Task Order The Latin America and Caribbean Bureau (LAC/SD-EHR) has a task order titled “Improved Human Resource Policies” with BEPS for provision of technical support and services for a 2.5 year period beginning Sept. 22, 2000. The first activity within this task order is a sub-task order for the evaluation of the Programa para la Reforma Educativa en Latinoamérica (PREAL). The Statement of Work (SOW) which will inform and guide this sub-task order is included as Appendix A to this Work Plan. 1.3 Resources and Limitations ß The PREAL Evaluation is being implemented within the parameters of the following resources/constraints: ß A budget that is set at approximately $100,000, thus binding the evaluation to resources, timelines, and research methods to financial limitations, while at the same time striving for the highest possible level of technical expertise and achieveable results within that budget. ß A tight timeline within which to recruit technical expertise, formulate appropriate strategy and methods, gather and analyze data and develop a final evaluation document. While USAID/LAC had originally suggested the end of December as a target date for completion, BEPS requests an extension to January 15, 2001 for final submission in order to accommodate challenges caused by the holiday season. Informal discussions with USAID/LAC staff have indicated that such a timeline is acceptable. ß Due to the fact that policy development is frequently a very slow process, sometimes taking many years to come to full fruition, BEPS anticipates it may be difficult to Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 86 actually observe attributable results of PREAL’s work. ß The performance of the Inter-American Dialogue (IAD) through PREAL cannot be judged on the basis of actual policy implementation in the beneficiary countries. The IAD’s performance through PREAL will be judged in achieving PREAL’s three stated objectives. ß Since PREAL is funded by more than one donor, it may be that cross-purposes exist in funding goals and, ergo, the shape and nature of activities which flowed from those goals. The PREAL evaluation research will seek to identify if PREAL’s activites have been consistent with goals proposed to USAID and if such cross-purposes exist and diluting of activities has occurred. 2.0 PROPOSED WORK PLAN 2.1 Target Completion Date: BEPS proposes that the final draft of the PREAL Evaluation will be delivered to USAID/LAC/RSD-EHR no later than January 15, 2001. 2.2 Deliverables Consistent with the SOW submitted to BEPS, there are four deliverables to be associated with this sub-task order: (1) the Draft Work Plan, represented by this document; (2) the Final Work Plan, to be established during the pre-field research planning meetings of the consultant team, IAD and USAID representatives; (3) a formal presentation by the research team on results emanating from the field research; (4) the draft of the final evaluation report; (5) the final Evaluation Report. The final Evaluation Report will contain at least the following elements, as presented in USAID/LAC’s SOW: a) Executive summary b) Key Findings and Recommendations c) Review of Issues to be Addressed d) Conclusion e) Summary of Evaluation Methodology f) List of individuals contacted Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 87 2.3 Supervision ß The LAC/PREAL Evaluation Team Leader will be responsible for design, implementation and production of the final evaluation report. He will assume overall supervision of the sub-task order design, implementation, data analysis, presentation, and report-writing. ß As BEPS/LAC Activity Coordinator, Ms. Antonieta Harwood is directly responsible for overall quality and performance of the PREAL Evaluation Team, liaison needs between USAID and the Evaluation Team, and quality assurance of the final report. ß Dr. Don Graybill, BEPS Project Director, will provide general oversight, support and quality monitoring of the sub-task order. 2.4 Summary of strategic approach The strategic approach will use both closed-end and qualitative research methods, interviewing a broad range of key informants and document review. USAID and IAD will be involved in a collaborative capacity with the research team to assist, as needed, in identification of potential data sources, including documents, organizations, and individuals who could be potentially surveyed. In order to preserve the integrity of the evaluation process and ensuing results, the research team will have operational independence as to whether or not to act upon the recommendations of USAID and IAD in strategy development, implementation and analysis. The research will be conducted by a consultant team of three senior experts whose skill areas will be concentrated in the fields of program evaluation, monitoring, social science research, education and social policy. All consultants will be fully fluent in Spanish, hold doctoral degrees, and qualify for senior expert status under BEPS labor category contract obligations. The team leader will be a person with additional expertise in USAID strategic planning systems and processes which will keep the task focused on client needs and context throughout the sub-task order. Within the team will also be significant experience and knowledge of educational policy, issues, trends, and needs represented at both regional and country-specific levels. The evaluation team will work full-time together on project design, methods, data collection, synthesis and draft report development from November 22 through December 15. The first 3 days of full team work, including collaborative support as necessary from USAID and IAD representatives, will focus on finalization of the work plan and research design, including identification of target research informants and sources, selection of data collection methodologies, refining of objective research design tools and methods and final logistics. This will be followed by an intensive approximate two week+ period (Nov. 27 – Dec.11) devoted exclusively to data collection, primarily in the field, but also including one day in Washington, DC. In terms of field research, each individual member Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 88 of the design team will be assigned a minimum of one country to visit within which to implement the research methodology, collecting as much data as possible following agreed-upon team approaches, guidelines, and formats. The research will focus on ascertainment of verifiable results from PREAL’s diverse activities and sub-grants. Those inquiries will be with potential clients of PREAL, to include but not be limited to: ß Ministries of Education, ß Associated institutions, ß Other donors, ß NGOs and government officials, ß USAID Missions ß Other key stakeholders appropriate to the scope of USAID impact evaluation assessment interests and PREAL target interventions. The research will focus on determining to what extent PREAL has achieved its three performance objectives, referred to in their cooperative agreeement as “intermediate results.” It will also attempt to discern if national or regional improvements in educational policy can reasonably be attributed to the work and support of PREAL. Finally, it will look at diverse issues of management, administration, and staffing inasmuch as they affect the performance of PREAL in achieving its purpose. Following data collection, the team will spend four intensive days (Dec. 11-15) in analyzing, organizing, summarizing, and drafting evaluation results for ultimate inclusion in the final PREAL report. A presentation to USAID staff will be made on Friday, Dec. 15 on tentative results produced by the investigation, including also time to identify and discuss relevant questions or issues surrounding evaluation implementation and results achieved. Feedback from those discussions will be folded into crafting the first draft of the evaluation report which will be submitted to USAID by COB on January 5, 2001 and which BEPS will consider to be a near-final version, subject to minor changes, additions or clarifications. In order to meet the BEPS deadlines, USAID will have three days with which to review and provide feedback on the draft report to the Team Leader by COB January 9. This feedback, in turn, will be folded into production of the final PREAL evaluation report, which will be submitted not later than January 15, 2001. 2.5 Key Design Elements A synthesis of the previous overall strategic approach includes the following design elements: ß Clarifying discussions with USAID re: SOW ß Introduction and discussions with IAD re: SOW ß Recruitment and selection of the experts consultant team Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 89 ß Drafting of general work plan and research design ß Refinement of the work plan and research design ß Data collection and analysis; ß Presentation of results to USAID with feedback; ß Report writing. 2.6 Contact and Coordination Contact and coordination with USAID/LAC. Frequent contact and consultation is required in order to assure that the research design, work plan, implementation and reports produced are consistent with USAID/LAC needs and performance expectations. Ongoing consultation will be maintained through adopted, normal channels. The USAID sub-task order manager is Susan Morawetz and the CAII/BEPS sub-task order manager is Ms. Antonieta Harwood. As such, all primary ongoing management concerns and coordination will happen directly between those parties. All products of this sub-task order will be (draft work plan, final work plan, final evaluation report) will be delivered directly to USAID/LAC PREAL Sub-task order Manager for ensuing distribution as she sees fit.. Contact and coordination with IAD. Frequent, ongoing contact and is also required with IAD and PREAL staff during the design and implementation of the task order. BEPS intends the evaluation to be a positive, objectively-based research enterprise which meets the needs of the client (USAID). However, every effort will be made to include, consult and inform, as appropriate, IAD and PREAL staff so that there is a spirit of positive support and collaboration throughout the exercise. BEPS emphasizes, however, that IAD and PREAL input will be strictly limited to informal consultation on an as-needed (or not) and as-appropriate (or not) basis, respecting the lines of independent integrity and decision-making authority that will be established by BEPS and prevail throughout the research period to ensure reliable results. BEPS will not share any drafts of any deliverables directly with IAD or PREAL at any time. Contact and coordination with other key informants. BEPS team members will, as required, assist evaluation team members in making initial contacts with key informants, both in the United States and overseas. 2.7 Key Research Questions The following primary, secondary, and tertiary research questions will be used to guide the research effort: Primary questions: ß Did PREAL succeed in building public and private sector support for educational reform? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 90 ß Did PREAL strengthen the capacity of public and private sector organizations working for educational improvement? ß Did PREAL adequately identify and disseminate best educational policies and procedures? To answer these questions, the Evaluation Team will, in collaboration with the IAD, review its established set of indicators of achievement for each objective and develop additional indicators appropriate to the PREAL evaluation. Secondary question: ß Has PREAL contributed to improved quality and equity of education by promoting regional dialogue and encouraging policy implementation? If so, how? Tertiary questions: ß Which of PREAL’s policy reform strategies and/or activities proven to be most effective? ß Who is using PREAL recommendations or findings? How are they being used? ß Has the structure of PREAL proven to be an advantage or a hindrance to PREAL accomplishing its mission successfully? Why? ß Is there evidence of integration of purposes? That is, has PREAL made efforts to unify or coordinate the mandates of its various funders? What is that evidence? ß Is the concept of sustainability pertinent to this activity? If so, how is it being encouraged and is it being achieved? 2.8 Countries to be visited: BEPS has met with both USAID and with the IAD to discuss issues related to country selection. Selection criteria. The key selection criteria, in discussion with USAID, to be considered are: ß Historical presence of PREAL activities conducted within the country; ß Existence of USAID presence and education programs within the country; ß Country offers positive potential demonstration of reform need and/or accomplishments; ß Country is a key locale for PREAL core activities which are projected to countries which meet the aforementioned criteria; Countries to be visited. Based on the aforementioned criteria and discussions with both USAID and PREAL, BEPS is recommending visits to Honduras, Nicaragua, Chile and Peru. (please note deletions) 2.9 Data Sources Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 91 The following will serve as primary data sources for the evaluation: PREAL ß Funding documents from USAID will be carefully reviewed. Any amendments will receive similarly reviewed. A sampling of PREAL publications will be reviewed. USAID ß Quarterly reports from PREAL to USAID will be scrutinized carefully ß Appropriate USAID Mission personnel in targeted countries will also be interviewed Public Institutions ß The Evaluation Team will interview key officials in visited countries. Some may be intentionally identified prior to visits and others identified opportunistically during the team member’s visit. Private Institutions ß Private institutions will be consulted in countries visited, selected according to the perceived relevance of potential data to the evaluation effort and availability of appropriate key informants. Inquiries will be made in the private sector. Private voluntary organizations (PVOs), community-based organizations and non￾governmental organizations will be included. Associations ß Professional associations, both in the United States and overseas will be included in the data gathering process. Networks ß An examination of electronic networks will be conducted. Informal networks will also be assessed. Think Tanks and Policy/Research Institutes ß Sources of policy expertise such as “think tanks” and policy/research institutes will also be identified and consulted. Press and other Media ß Appropriate press and other media sources will be searched for information that assists Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 92 in addressing the key research questions. Other key informants ß Additional key informants representing a cross-section of past, present and future key stakeholders across local, regional and national levels will be identified and included within the sampling and field research. 2.10 Key Methodologies The following methodologies will be primarily used: Key Informant Interviews – Lists of informants will be drawn up, using USAID and PREAL as initial guides for identification of potential resources. Additional key informants will be opportunistically identified as the project evolves and evaluation team gain additional perspective. Document review – Relevant contractual and technical documents related to assessment of PREAL impact will be studied insofar as their relevance to the overall evaluation effort is established. (Note that the method of “observations” of PREAL activities has been eliminated since, after discussion by the Evaluation Team, it was determined that observation of PREAL activities (with exception of very rare cases) would not contribute to determination of impact. 2.11 Schedule of Sub-task Order Activities Phase Activities Dates Early Development ß Clarification of SOW with USAID ß Early meetings with IAD ß Recruitment and selection of expert consultant team ß Development of early draft Work Plan ß Review and Approval of Draft Work Plan by USAID 10/10 – 10/31/00 10/18 – 11/3/00 10/15 – 11/3/00 10/15 – 11/7/00 11/9/00 Refining of Final Work Plan ß Ongoing consultations with consultant team, USAID and IAD to yield Final Work Plan 11/10 – 24/00 ß Team development and planning; development of final research design, tools and methods ß Washington, DC-based partial research with USAID and IAD 11/22-24/00 Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 93 ß Submission of Final Work Plan to USAID 11/24/00 Research and Data Collection ß Field visits to 4 countries 11/26 – 12/7/00 ß Washington, DC-based data collection and document review 12/8/00 Data analysis and synthesis ß Individual and team organization, analysis and synthesis of tentative results ß Preparation for USAID presentation 12/11-13/00 12/14/00 Presentation to USAID ß Presentation of tentative results to USAID with discussion and feedback on related issues 12/15/00 Final report writing ß Write first draft of final report ß Submit first draft to USAID ß Receive feedback from USAID on first draft submission 12/16 – 1/6/01 1/5/01 1/9/01 ß Submission of final report to USAID 1/15/01 3.0 PREAL EVALUATION TEAM 3.1 Team configuration The Team will be configured with three specialists drawing from three different BEPS partners. The Team Leader (Education Generalist, CLIN 1 A) will be fielded by CAII. The Education Evaluation and Research Specialist (CLIN 7A) will be fielded by GroundWork. The Education Policy Specialist (CLIN 12A) will be fielded by George Washington University. 3.2 Key qualifications, selection criteria Key qualifications for candidate recruitment and identification include: ß appropriate academic preparation and experience in related content areas; ß availability during the time frame during which the evaluation must be conducted; ß no candidate will have had programmatic or financial dealing with PREAL during the last four years; ß demonstrate at least a FSI 3/3 capability in Spanish; ß professional experience dealing with individuals at high levels of government; ß demonstrated familiarity with Latin American educational policies, systems and/or related issues. 3.3 Consultant Team and Summaries Based upon intensive recruitment efforts, BEPS proposes the following PREAL Evaluation Team to USAID/LAC: ß Dr. Donaldo Hart, Team Leader (CLIN 1A). Dr. Hart enjoys Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 94 thirty years of professional experience and has worked as a Team Leader and Chief of Party for USAID-funded activities since 1989. A Brazilian American, he is familiar with Latin American issues in education and has worked extensively in Brazil as well as short￾term in other countries of the region. Dr. Hart is highly experienced in surveys, program evaluations, monitoring systems, and data management and analysis. He has designed and supervised nation-wide surveys and has conducted numerous rapid evaluations of activities in economic assistance, human resources training, and education. From 1997-2000 Dr. Hart was director of one of the largest monitoring and evaluation units in USAID, creating a highly responsive system for monitoring eighteen performance milestones, at three levels, in an education-sector project in Egypt. ß Dr. Patricia Craig, Education Research and Evaluation Specialist (CLIN 7A). Dr. Craig has extension experience in the field of Education Research and Evaluation. Most recently, Craig worked for the Harvard Institute for International Development (HIID) for four years. In that time she worked with the Ministry of Education in El Salvador on the Education Reform as well as IBM’s Latin America Advisory Board to focus private sector’s attention on issues of education and growth. Dr. Craig evaluated medium and long-run possibilities for state and private sector funding of universities of El Salvador to improve quality and performance of higher education sector. Prior to her experience at HIID, Dr. Craig taught political science and sociology at Ohio State University for four years. ß Dr. Daniel A. Morales-Gómez, Education Policy Specialist (CLIN 12A). Dr. Morales- Gómez has had twenty years of project development and management experience with developing countries in education, and assessment of health care services, poverty reduction strategies, safety net programs and community-based services for vulnerable groups. He designed and managed capacity building programs in social policy in support of sectoral applied research, which led to the establishment of 4 regional multi-institutional networks in Latin America, three in Africa and one in South-east Asia involving researchers, practitioners, and policy￾makers from over 30 institutions. Dr. Morales- Gómez has been responsible for monitoring multi-national inter￾disciplinary teams of social policy researchers, practitioners and policy makers in Africa and Latin America in over 40 projects on public policy and improvement of social services in education, health care, safety net programs, and decentralized service provision. He was the principal IDRC education specialist for South America in the 1980’s, and IDRC contact for social sciences capacity building in Southern Cone during transition to democracy. Morales- Gómez has an extensive publication record on international development, social policy, globalization, comparative education, adult education, research ethics, and international aid. Dr. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 95 Morales- Gómez earned is Ph.D. in Educational Planning at the University of Toronto in Canada and has a Master’s in Comparative and International Education from Stamford University. CVs for the aforementioned individuals are included within this draft Work Plan as Attachment C. 1.0 BUDGET The estimated budget proposed for this activity will be $100,000. BEPS anticipates that this total will include: Subtotal personnel = 76,000 Subtotal Travel, Per Diem and ODCs = 24,000 Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 96 APPENDIX B. SURVEY INSTRUMENT EVALUACIÓN DE PREAL CUESTIONARIO Nombre : __________________ Sexo : M F País : __________________ Organización: __________________ Tipo de organización: Pública Privada Sin fines de lucro Tiempo en el cargo: ___________________ Empleo: (si otro) ________________ Diciembre 2000 Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 97 1. ¿Por cuánto tiempo ha conocido las actividades de PREAL? ________ 2. ¿Ha participado usted en alguna actividad de PREAL? ___ Si ¿Cuáles? _______________________ ___ No (pase a la pregunta 10) _______________________ 3. ¿Cuál de éstas le ha sido de más utilidad? _______________________ _______________________ ¿Y cuál le ha sido de menos utilidad? _______________________ _______________________ 4. ¿Hablando de más utilidad, aprendió algo nuevo a partir de esa actividad? ___ Si ¿Podría darme un ejemplo? ___ No _______________________ _______________________ 5. ¿Ha tomado usted alguna acción o iniciativa personal como resultado de lo que aprendió? ___ Si Podría darme ejemplos: ___ No _______________________ _______________________ _______________________ _______________________ 6. ¿Ha tenido esto algún efecto o impacto en lo que hace su organización? ___ Si ¿Podría darme ejemplos? ___ No _______________________ _______________________ _______________________ 7. ¿Ha tenido esto algún impacto en la promoción del diálogo sobre la reforma educativa? ___ Si ¿Podría darme ejemplos? ___ No _______________________ _______________________ _______________________ 8. ¿Ha tendido esa actividad algún impacto en la política o en la práctica educativa nacional? ___ Si ¿Podría darme ejemplos? ______________________ ___ No ______________________ Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 98 _______________________ 9. ¿Hay obstáculos para llevar a cabo lo sugerido por PREAL? (Profundice: obstáculos en el ambiente donde trabaja) ___ Si ¿Cuáles? ___ No _______________________ _______________________ 10. ¿Cómo resultado de esa actividad, ¿considera usted estar más informado de la existencia de actividades que promuevan la reforma educativa? ___ Si ¿Podría darme ejemplos? ___ No _______________________ _______________________ _______________________ _______________________ 11. ¿Ha establecido usted o su organización nuevos contactos como resultado de esta actividad? ___ Si ¿De qué tipo? ___ Personales ___ No ___ Organizacionales ___ Otros (¿cuáles?) ________ ¿Podría darme ejemplos? _____________________ ¿Cómo usa usted estos contactos? ______________ _________________________________________ ¿Ha mantenido estos contactos? ________________ 12. En cuanto a los de menos utilidad por que considera que ha(n) sido de menor utilidad? Para los Centros Asociados: (PARA OTROS, PASE A LA PREGUNTA 17) 13. Su organización ha recibido asistencia financiera de PREAL. Ayudó esta asistencia a fortalecer la capacidad de: a) gestión/administración: mucho algo no mucho (¿Cómo? Cuando la respuesta es ‘mucho’; ¿Por qué? Cuándo es algo o no mucho) b) investigación: mucho algo no mucho Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 99 (¿Cómo? Cuando la respuesta es ‘mucho’; ¿Por qué? Cuando es algo o no mucho) c) planificación: mucho algo no mucho (¿Cómo? Cuando la respuesta es ‘mucho’; ¿Por qué? Cuando es algo o no mucho) d) capacidad de obtener nuevos fondos: mucho algo no mucho (¿Cómo? Cuando la respuesta es ‘mucho’; ¿Por qué? Cuando es “algo” o “no mucho”) 14. ¿Es esta capacidad autosostenible en el futuro sin el apoyo financiero de PREAL? ___ Si ¿Podría explicarme? ___ No (¿qué ocurrira?) 15. ¿Se ha beneficiado su organización de otro tipo de apoyo de PREAL no financiero? ___ Si ¿Podría explicarme? ___No 16. ¿Cómo resultado de este apoyo de PREAL, le ha sido posible a su organización realizar algún tipo de contribución: a) a las políticas educativas Si No b) a las prácticas educativas Si No ¿Podría explicarme? PREGUNTAR A TODOS Un objetivo de PREAL es identificar y diseminar información sobre las mejores políticas y/o prácticas educativas. 17. ¿Ha recibido usted alguna publicación o información producida por PREAL? ___ Si ¿Cuáles? ___ No _________________________ _________________________ 18. ¿Aprendió algo útil de estas publicaciones o materiales? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 100 ___ Si ¿Podría darme ejemplos? ___ No _________________________ _________________________ 19. ¿Obtiene usted información sobre mejores prácticas educativas a través de otros medios diferentes a PREAL? ___ Si ¿Podría darme ejemplos? ___ No _________________________ _________________________ ¿En comparación con otros, encuentra la información de PREAL? ___Más útil ___Menos útil ___Igual de útil 20. ¿Usa su institución la información sobre mejores prácticas educativas que recibe de PREAL? ___ Si ¿Podría darme ejemplos? ___ No _________________________ _________________________ 21. ¿Ha contribuido esta información a acciones que llevan al desarrollo de políticas educativas? ___ Si Podría darme ejemplos? ___ No _______________________ Ahora, hablemos de la pagina del Internet. 22. ¿Ha usado usted la página de PREAL en Internet? ___ Si ___ No ¿Por qué no la usa? ___ No sabe que existe (pase a pregunta ___ No le es útil (pase a pregunta 24) ___ Otra razón ¿Por qué? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 101 23. ¿Qué partes de la página de PREAL le son de la mayor utilidad (indique todo) 24. En su opinión, ¿qué tipo de información podría PREAL incluir en su página que sea de más utilidad para usted? Ahora, hablando en general. 25. En su opinión, ¿cuáles son los problemas en educación más importante que enfrenta el país? 26. En su opinión, ¿ha hecho algo PREAL con relación a estos problemas? ___ Si ¿Cómo qué? ___ No 27. En su opinión, lo que hace PREAL respecto a estos problemas ha contribuido a un cambio oficial de la política educativa? ___ Si ___ No ¿Podría explicar? 28. En su opinión, ¿cuáles son los obstáculos más grandes para llevar a cabo la reforma educativa en este país? 29. ¿Piensa usted que las actividades de PREAL contribuyen a superar esto obstáculos? ___ Si ___ No ¿Podría explicar cómo? Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 102 30. Considera Ud. que PREAL ha involucrado los sectores y actores mas apropiados para influenciar la política educativa en el país? ___ Si ___ No ¿Podría explicar? 31. ¿Qué sugeriría Ud a PREAL para lograr un mayor impacto en los cambios de política educativa? Muchas gracias por su colaboración. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 103 APPENDIX C. SURVEY RESULTS Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 104 Frequency Comparisons Q.# Description Total Hondura s Nica. Peru Chile Respondents 70 24 18 16 12 male/female 45/25 12/12 10/8 13/3 10/2 Public Org 28 12 10 2 4 Private firm 19 2 4 7 1 NGO 16 5 5 5 1 Donor 8 5 1 1 1 2 ha participado 62 20 16 15 11 4 aprendio 62 19 16 15 12 5 tomado accion 51 17 10 13 11 6 tenido impacto 53 13 14 14 12 7 promocion dialogo 57 17 14 14 12 8 impacto politica 44 11 9 13 11 9 obstaculos 42 2 13 15 12 10 mas informado 51 9 14 16 12 11 nuevos contactos 47 9 14 13 11 11a ha mantenido 29 8 8 4 9 13a gestion (mucho) 2 2 13b investigacion (mucho) 6 1 2 2 1 13c planificacion (mucho) 4 1 2 1 13d nuevos fundos (mucho) 3 1 1 1 14 autosostenible 6 1 1 2 2 15 apoyo no financiera 12 1 3 5 3 16a politicas educativas 10 1 3 4 2 16b practicas educativas 9 1 2 4 2 17 ha recebido 64 23 14 16 11 18 aprendio algo util 62 23 11 16 12 19 obtiene de otros 60 23 14 12 19 PREAL mas util 42 14 5 12 19 igual de util 11 8 3 19 menos util 3 3 20 usa su institucion 57 22 9 14 12 21 contribuido a acciones 45 16 7 13 9 22 usado web page 19 1 2 13 9 26 ha hecho PREAL algo 53 18 12 14 27 cambio oficial 38 5 10 13 10 29 PREAL supera 46 14 8 14 10 30 actores mas apropriados 52 20 11 12 9 Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 105 Notes to the Frequency Table The following table comprises nearly all of the close-ended variables in the survey. The first column gives the number of the question the frequencies correspond to, while the second column offers an abbreviation of the question, making it easy to follow the table while referring to the survey instrument. By providing a column for each country, we have effectively shown cross tabulations with of the variables using, in four trials, each country as a constant. The best way to read a frequency table is to look for inconsistencies and patterns. For example, there is a great inconsistency between web page use (Question 22) in Peru and Chile as compared with Nicaragua and Honduras. Or in Question 11a, which asks how many respondents maintained the contacts they established during participation in seminars, we find that the percentages are significantly lower in Peru and Nicaragua than in Chile and Honduras, and this drop-off is to us suggestive of poor follow-up. For examples of patterns, we were pleased to see that the expected logic of consistent decrease in numbers from participating to using to organizational impact, in Questions 2 through 8 generally held. There was a similar decreasing pattern in receiving, learning from, and applying lessons from publications in Questions 17, 18, 20, and 21. We have not provided lists of open-ended responses. The Evaluation Team studied the patterns of these responses extensively in formulating its findings and recommendations, but we have not set them in tabular form. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 106 APPENDIX D. LIST OF INTERVIEWEES Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 107 PREAL EVALUATION LIST OF INTERVIEWEES Washington, DC Rebecca Adams, USAID Cynthia Chassey, USAID David Evans, USAID Susan Morawetz, USAID Jeffrey Puryear, IAD Honduras Public Sector Marina de Aguilar, Director of Research, Universidad Pedagógica Nacional Sonia Aspra, Senior Advisor, FONAC Sajda Burgos, Director, University Education Administration Guillermo Canaca, Chief, Technical Teaching Division, INFOP Coritza Díaz, President, COPEMH (Secondary Teachers’ Union) María del Carmen Díaz, Director of Educational Measurement and Quality, UMCE Mirta Fernandez, Assistant, Technical Division, INFOP Marco Orlando Iriante, Executive Director, FONAC Régulo de Jesús Mancía, Director of Curriculum, EDUCATODOS Normanda Martinez, FONAC Mario Membreño, Professor, Universidad Pedagógica Nacional Aurora Gonzales Muñoz, District Director, MOE Arnaldo Pinto, President, COLPROSUMAH (Elementary Teachers’ Union) Cristobal Rodríguez, Director Educational Planning, MOE Ramón Salgado, Dean of the Universidad Pedagógica Nacional Private Sector Rossana Guevara, Journalist and Television Commentator for Channel 63 Ricardo Maduro, Industrialist and Candidate for the Presidency of the Republic Ana Elsy Mendoza, Journalist and Consultant Leonor Meza, Consultant in Education Carlos Héctor Sabillón, President, FENIEPH (Federation of Private Schools) Not-for-profit Sector María Antonieta Bográn, Executive Director, FEREMA José Carleton Corrales, Regional Coordinator, Central America, PREAL Josefina Gamero, Technical Director, FEREMA Maria Luisa Meyerman, Consultant to CADERH Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 108 Donors Ramón. Calero, Project Director, Spanish Cooperation John Helwig, Education Advisor, USAID Diane Leach, Team Leader, Education Division, USAID Reynaldo Martinez, Vocational Education Specialist, DevTech Systems. John McAvoy, Contracting Officer, USAID Marco Tulio Mejia, USAID Quezada, Country Representative, OEI Teschner Wolfgang. Chief Advisor, GTZ. Jorge Orlando Torres, Sector Specialist, IDB Nicaragua Donors USAID-Nicaragua Loretta Garden David Eggerton Danielle Roziewski IDEUCA Juan B. Arrien Miguel de Castilla EDUQUEMOS Gilberto Cuadra Aurora Gurdian Alicia Langran Veronica Morales Private Sector Fanor Fernandez 3F Gabriel Solorzano GM of Holiday Inn and board member of INATEC Marlen Landero INDE Anastasio Somasrosa Lorena Zamora AMCHAM Civil Society Julian Corrales, Human Rights Prosecutor Sofia Moreno, Coordinator of National Education Movement Julio Hernandez, Secretariat of the National Education Congress Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 109 Jose Antonio Zepeda, Secretary General of FOMCA (teachers’ union) Mario Saenz, Sub-director of Military Hospital Policy Makers/Government Humberto Belli, President of Ave Maria University and former Minister of Education Tulio Tablada, Secretaria tecnica de la Presidencia Brenda Cuadra, Plan Nacional de Educacion Ofilia Sagasa, MEC Myrna Somariba, MEC Chile Public Sector Jose Weinstein. Under Secretary of Education. Ministry of Education, MINEDUC, Chile Cecilia Jara. Chief Education Area. SME. Ministry of Education. Former Head of 900 Schools Program. Ivan Nuñez. Advisor to the Ministry of Education. Teachers Unions. Ministry of Education. Eugenio Diaz Corvalan. Executive Director. Centro Nacional de la Productividad. Private Sector Jose Joaquin Brunner. Director Education. Fundacion Chile. Former PREAL Co-Director. Jose Jimenez. Professor & Researcher. University of Concepcion and University of Santiago. Eugenio Rodriguez. Prof. School of Education. Catholic University and Representative of the OEI in Chile. Orlando Mella. Coordinator Master Program. Coordinator of REDUC. Universidad Alberto Hurtado Ernesto Schiefelbein. Rector (Dean), Universidad Santo Tomas. Robert W. McMeekin. Consultant for UNESCO. Non-Profit Marcela Gajardo. Co-Director PREAL - PREAL, Santiago Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 110 Ana Maria de Andraca. Associate Researcher & Data Base Coordinator Nelson Martinez Berrios. Program Assistant & Web page Coordinator Osvaldo Sunkel. President Corporaciones de Investigaciones para el Desarrollo,CIDES & Director Centro de Analisis de Politicas Publicas Universidad de Chile Sergio Martinic. Director CIDE. Responsible for the Working Group on Political Obstacles to Reforms. Monica Jimenez. Executive Director. Participa. Educacion y Desarrollo Donor Ana Maria Corvalan. Regional Coordinator Information System SIRI. UNESCO – Chile (Representing the Director of OREALC-Unesco). Peru Public Sector Jorge Ferradas. National Director, Programa de Mejoramiento de la Calidad de la Educación Peruana (MECEP), Ministry of Education. Jose Rodriguez Gonzales. Chief, Unidad de Medicion de Calidad Educativa. Ministry of Education. Martín Vegas. Advisor, Member of Parliament & Former Executive Director of Foro Educativo. Private Sector Henry Day. Presidente. Perú 2021. Cesar Fonseca. General Manager. Peru 2021. Alfredo Miró Quesada. Dean Facultad Ingeniería, Universidad Peruana de Ciencias Aplicadas. Lourdes Flores-Nano. Lawyer. Candidate to the Presidency & Member if the Inter￾American Dialogue. Manuel Bello. Dean Facultad de Educación, Universidad Cayetano Heredia . Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 111 Drago Kisic Wagner. Presidente Instituto Peruano de Administracion de Empresas, IPAE & Organizer next Annual Conference of Executives, CADE. Raúl Salazar. Director Instituto Peruano de Administración de Empresas, IPAE / Grupo Macro & Organizer next Annual Conference of Executives, CADE. Hugo Díaz. Area of Education, Instituto de Investigación sobre Defensa Nacional; Consultant & Editor of a Education Newsletter. León Trahtemberg. Director General Colegio Leon Pinelo; Founder member of Foro Educativo; author of a weekly newspaper column & host of radio and TV program. Non Profit Patricia Arregui. Researcher, GRADE & Coordinator, PREAL Working Group on Evaluation and Standards. Santiago Cueto. Research Director, GRADE & Advisor to the Unidad de Medición de la Calidad del Ministerio de Educación (GRADE) Gabriel Ortiz de Zevallos. Executive Director. Instituto Apoyo. Ricardo Morales. Presidente Foro Educativo. Luis Carlos Gorriti. Member of Foro Educativo. Eduardo Bolaro. Executive Director. Foro Educativo. Donors Dick Martin. USAID Representative, Peru. Kristin Langlykke. USAID Representative, Peru. Walter Twanama. USAID Representative, Peru. Wolfgang Kuper. GTZ Representative, Peru. Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 112 APPENDIX E. BIBLIOGRAPHY Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 113 BIBLIOGRAPHY Brünner, José Joaquín. Educación: Escenarios de Futuro. Nuevas Tecnologías y Sociedad de la Información. (Santiago, Chile: Inter-American Dialogue/PREAL, Jan. 2000). Chairman’s International Advisory Council of the Americas Society, Inter￾American dialogue and latin american business council. Socios para el progreso: educación y el sector privado en américa latina y el caribe. Ed. Puryear, Jeffrey M. 2nd ed. (Washington: 1997). Cox. Cristián. La Reforma de la Educación Chilena: Contexto, Contenidos, Implementación. (Santiago, Chile: Inter-American Dialogue/PREAL, Aug. 1997). El Futuro está en Juego: Informe de la Comisión Internacional en Educación, Equidad Y Competitividad Económica en America Latina y el Caribe. (The Inter-American Dialogue, 1998). Evaluación y Reforma Educativa: Opciones Políticas. Ed. Alvarez, Benjamin and Ruiz Casares. (PREAL and Educational Development Academy, 1997). Experiencias de Reforma y Gestión. Comp. Josefina Bruni Celli and Milko Ramsés González. (Foro Educativo Venezuela, 1998). Gajardo, Marcela. Reformas Educativas en América Latina. Balance de una Década. (Santiago, Chile: Inter-American Dialogue/PREAL, Sept.1999). Hanson, Mark E. Descentralización Educacional: Temas y Desafíos. (Santiago, Chile: Inter-American Dialogue/PREAL, Nov.1997). Levin, Henry M. Las Escuelas Aceleradas: Una Década de Evolución. (Santiago, Chile: Inter-American Dialogue/PREAL, Oct. 2000). Myers, Robert G. Educación Preescolar en América Latina. (Santiago, Chile: Inter￾American Dialogue/PREAL, Jul. 1996). PREAL/UNESCO/MINEDUC Chile. Indicadores Educativos Comparados en el Mercosur. (Chile: 1997). PREAL/UNESCO. Financiamiento de la Educación en América Latina. (Bogota: Seminario internacional sobre financiamientos de la educación en américa latina, 1997). Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 114 PREAL/Fundación Getulio Vargas. Evaluación y Determinación en de Estándares en la EducaAmérica Latina. Realidades y Desafíos. (Rio de Janeiro: 1998). Puryear, Jeffrey M. Educación en América Latina: Problemas y Desafíos. (Santiago, Chile: Inter-American Dialogue/PREAL, May1997). Puryear, Jeffrey and Brünner, José Joaquín. Educación, Equidad, y Competitividad Económica en las Americas. vol. II :Estudios de Casos. (Washington D.C.: Organization of American States, 1995). ____________________. Educación, Equidad, y Competitividad Económica en las Americas. Vol. I: Asuntos Claves (1994a). ____________________. Una Agenda para la Reforma Educativa en América Latina y el Caribe. (PREAL 1994b). Puryear, Jeffrey and Olivos, Andrea. Considerando Primero la Educación: Reporte de Conferencia. (PREAL 1995). Swope, John, and Latorre, Marcela. Comunidades Educativas Donde Termina el Asfalto: Escuelas Fe y Alegría en América Latina. (C hile: Centro de Investigación y Desarrollo de la Educación, 1998). West, Edwin. Un Estudio sobre Principios y Practicas de los Vouchers Educacionales. (Santiago, Chile: Inter-American Dialogue/PREAL, Oct.1998). Wilson, David N. Reforma de la Educación Vocacional y Técnica en América Latina. (Santiago, Chile: Inter-American Dialogue/PREAL, Dec. 1996). Winkler, Donald R. and Gershberg, Alec Ian. Los Efectos de la Descentralización del Sistema Educacional sobre la calidad de la educación en América Latina. (Santiago, Chile: Inter-American Dialogue/PREAL, Apr. 2000). Wolff, Edwin G. Evaluación Educacional en América Latina: Proguresos Actuales y Desafíos Futuros. (Santiago, Chile: Inter-American Dialogue/PREAL, 1998). Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 115 APPENDIX F. PREAL ASSOCIATED CENTERS Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 116 PREAL ASSOCIATED CENTERS Argentina FLACSO Brasil Centro de Estudos e Pesquisas em Educação, Cultura e Ação Comunitária (CENPEC) Fundación Getulio Vargas Instituto SER de Investigación Colombia Centro de Investigación y Desarrollo de la Educación (CIDE) Chile Corporación Participa Costa Rica Instituto de Investigación para el Mejoramiento de la Educación Costarricense (I.I.M.E.C.) Ecuador Corporación de Estudios para el Desarrollo (CORDES) Fundación Ecuador El Salvador Fundación Empresarial para el Desarrollo Educativo (FEPADE) Asociación de Investigación y Estudios Sociales (ASIES) Guatemala Centro de Investigaciones Económicas Nacionales (CIEN) Consejo Nacional de Educación Maya (CNEM) GRUPO BASICO, S.A. Honduras Fundación para la Educación Ernesto Maduro Andreu (FEREMA) Nicaragua El Foro Educativo Nicaragüense, EDUQUEMOS IDEUCA (former Universidad Centroamericana (UCA) Perú Foro Educativo Grupo de Análisis para el Desarrollo (GRADE) República Dominicana Technical Assistance to the LAC Bureau’s Office of Regional Sustainable Development in Support of Educational Policy Reform PREAL Evaluation Basic Education and Policy Support (BEPS) Activity 117 FLACSO/Plan Educativo República de Panamá Unión Nacional de Centros Educativos Particulares (UNCEP) Venezuela Instituto de Estudios Superiores de Administración (IESA)