May 2015 This document was produced at the request of the United States Agency for International Development (USAID). It was prepared with the participation of Carlos Rodríguez, Emma Yep, and Eduardo Prochazka, consultants from Partners for Global Research and Development LLC (PGRD) under contract N° AID-527-C-13-00002. Final performance evaluation of the Peru Quality Basic Education Reform Support Program Fotos del Programa SUMA. Actividades en San Martín ii iii FINAL PERFORMANCE EVALUATION OF THE PERU QUALITY BASIC EDUCATION REFORM SUPPORT PROGRAM (SUMA) AID-527-C-09-00001 Date of translation: July 2015 The opinions expressed herein are those of the team that conducted the study, and do not necessarily reflect the views of USAID or of the Government of the United States of America. iv CONTENTS ACRONYMS ................................................................................................................................................. v EXECUTIVE SUMMARY .............................................................................................................................. 7 PURPOSE AND EVALUATION QUESTIONS ............................................................................................12 Purpose of the Evaluation ...................................................................................................................12 Evaluation Questions ..........................................................................................................................12 BACKGROUND ..........................................................................................................................................14 Operating Mechanism .........................................................................................................................15 EVALUATION METHODS AND LIMITATIONS ..........................................................................................17 Evaluation Design Stage .....................................................................................................................17 Field Work Stage ................................................................................................................................. 17 Analysis and Discussion Stage ...........................................................................................................18 Limitations ...........................................................................................................................................18 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS .......................................................................20 FINDINGS ...................................................................................................................................................20 1. Education Decentralization Policies Component ....................................................................................20 1.1. To what extent did the SUMA Program promote specific changes in the education decentralization policy and institutional framework? ...........................................................................20 2. Teaching Quality .....................................................................................................................................35 2.1 To what extent did the SUMA Program promote models and strategies for improving teaching quality that were institutionalized in the sector at the national and regional levels? ...........................35 3. Lessons Learned from the Experience ...................................................................................................48 3.1. Which critical elements (both internal and external) of the SUMA Program favored the achievement of the expected outcomes, and which did not? .............................................................48 3.2 What practical knowledge can be extracted from the experience carried out by the Program? ..56 3.3 How to capitalize on what has been learned in the SUMA Program within the design of new interventions aimed at improving the management and quality of public services in the Amazon? ...60 CONCLUSIONS ..........................................................................................................................................62 RECOMMENDATIONS ...............................................................................................................................64 BIBLIOGRAPHY .........................................................................................................................................66 Annex 1: Statement of Work .......................................................................................................................69 Annex 2: List of interviewees ....................................................................................................................100 Annex 3: Status of public investment projects reported by SUMA ...........................................................102 Annex 4: Table of indicators per contract and indicators reported ...........................................................104 Annex 5: Gender approach in the SUMA Program ...................................................................................108 v ACRONYMS AED Academy for Educational Development (Implementing partner) ANGR Asamblea Nacional de Gobiernos Regionales/National Assembly of Regional Governments AprenDes Innovations in Decentralization and Active Schools CGLDS Comité de Gestión Local para el Desarrollo Social / Local Management Committee for Social Development CIA Círculos de Aprendizaje/Learning Circles CND Consejo Nacional de Descentralización/National Decentralization Council CNE Consejo Nacional de Educación/National Education Council CONEACES Consejo de Evaluación, Acreditación y Certificación de la Calidad de la Educación Superior no Universitaria/Council for Evaluation, Accreditation and Certification of Quality for Non-University Higher Education CONEI Consejo Educativo Institucional /Institutional Education Council COPALE Consejo Participativo Local de Educación/Participatory Council for Local Education COPARE Consejo Participativo Regional de Educación/Participatory Council for Regional Education CRA Centro de Recursos de Aprendizaje/ Learning Resources Center DRE Dirección Regional de Educación/Regional Education Directorate EBR Educación Básica Regular/Regular Basic Education ECE Evaluación Censal de Estudiantes/Student Census Evaluation EIB Educación Intercultural Bilingüe/Intercultural Bilingual Education FHI 360 Family Health International 360 (Implementing partner) GL Local government GRSM Gobierno Regional de San Martín/Regional Government of San Martín IIEE Instituciones Educativas/Educational Institutions ISP Instituto Superior Pedagógico/Pedagogical Institute LGE Ley General de Educación/General Education Act LOF Ley de Organizacion y Funciones/Law on Organization and Functions LOPE Ley de Organización y Funciones del Poder Ejecutivo/Law on the Organization and Functions of the Executive Branch M&E Monitoring and evaluation MEF Ministerio de Economía y Finanzas/Ministry of Economy and Finance MCLCP Mesa de Concertación para la Lucha contra la Pobreza/Round Table for the Fight Against Poverty vi MINEDU Ministerio de Educación /Ministry of Education MGL Modelo de Gobernanza Local /Local Governance Model OAAE Oficina de Apoyo a la Administración de la Educación/Office of Support for the Administration of Education PALMA Plan Anual Local para la mejora de los aprendizajes /Local Annual Plan for Learning Improvement PATMA Plan Anual de Trabajo para la mejora de los aprendizajes / Annual Work Plan for Learning Improvement PCM Presidencia del Consejo de Ministros/Council of Ministers PELA Programa Presupuestal Estratégico Logros de Aprendizaje/Strategic Learning Achievement Budgetary Program PEN Proyecto Educativo Nacional / National Education Project PER Proyecto Educativo Regional/Regional Education Project PIP Public Investment Project PMP Medium term plan POI Institutional operations plan PRONAFCAP Programa Nacional de Formación y Capacitación Permanente/National Program for Lifelong Education and Training SIGMA Sistema de Información y Monitoreo del Acompañamiento/ Integrated Management and Monitoring Tracking System SINEACE Sistema Nacional de Evaluación, Acreditación y Certificación de la Calidad de la Educación/National System for Evaluation, Accreditation and Certification of Education Quality SSII-PER Sistema de Seguimiento e Información- Proyecto Educativo Regional /Monitoring and Information System- Regional Education Project UGEL Unidad de Gestión Educativa local / Local Educational Management Unit UNICEF United Nations Children's Fund USAID U.S. Agency for International Development 7 EXECUTIVE SUMMARY The main purpose of this evaluation is to analyze the extent to which the SUMA program achieved its expected results and to identify key factors, lessons learned and recommendations derived from the program’s implementation. Evaluation questions and methodological approach The evaluation defines two questions directly related to the main components of the program: policy and institutional framework of the decentralization of education, and the introduction of strategies and models for the quality of education that will foster an improvement in the learning process. In addition, the evaluation seeks to identify the elements (internal and external) that affected (positively and negatively) the achievement of expected results and to document practical lessons learned and knowledge that can be derived from this experience and applied in the design of new initiatives in the framework of USAID’s cooperation strategy. The methodological approach is primarily qualitative, combining available secondary information with information obtained from interviewing national and regional actors and the Program technical team. In February and March 2014, the evaluation team interviewed 53 individuals, including representatives from the Ministry of Education, Regional and Local Governments in the San Martin and Ucayali regions, and experts from the National Education Council and civil society. The context Family Health International (FHI) implemented the Quality Basic Education Reform Support Program (SUMA), with an investment of $12.9 million between September 2009 and June 2014. The SUMA Program carried out a national and regional intervention that provided technical assistance to the Government of Peru supporting reform initiatives intended to improve the quality of basic education and student learning performance. During the 2009-2013 implementation period, the decentralization process continued after the establishment of regional governments along with the formal transfer of previously centralized functions, albeit with a marked absence of support for the development of implementation capacity among regional entities. Decentralized entities do not have access to qualified human resources, have limited budgets, and lack the legal, political and economic framework to comply with the government’s results-based public management system. Within the education sector, the National Education Project (PEN) has been approved and adopted as the guiding document for the establishment of education policies. Regional authorities have been able to approve Regional Education Projects following long participatory processes and public debate. Although, the transition from long-term strategic planning to concrete implementation was pending. Concurrently, each year the government publicized learning achievement testing results with diverse consequences: the genesis of a culture of evaluation and transparency accompanied by a public discussion of the responsibilities of the school, the teachers, the family, and the public sector. The National Education Council (CNE) established dialogue and participatory roundtables, and donor organizations and civil society institutions searched for mechanisms to support the implementation of education policies. All the actors recognized that technical assistance at the national and sub-national levels is an urgent necessity. 8 Findings At the regulatory and policy level, SUMA oriented its efforts towards the discussion and preparation of the Law on Organization and Functions (LOF) for the Ministry of Education (MINEDU) and on capacity building for medium-term and operational strategic planning. In order to generate support for the public funding of capacity building processes, SUMA supported the MINEDU, Regional Governments and Municipalities to develop public investment projects (PIP), using the San Martin Regional Education Project, developed with the support of the USAID AprenDes1 project, as a model. To improve the quality of education, SUMA builds on the AprenDes Active Schools experience and improves it in two areas. Inside the classroom through the validation of student Notebooks and Guides, which become Self-Teaching Notebooks. In support of teachers, through the provision of new methodological tools, such as ‘Curriculum Paths’ and the ‘Teacher Compass’, which are abridged versions of the MINEDU tools: Fundamental Knowledge, Progress Maps, and the Learning Paths. The project contributes to the pedagogic dimension of training and professional development through training programs for teachers, school principals, and trainers, which, based on the experience acquired in the regional intervention areas, have fostered improvement in teaching coaches training activity carried out at the national level by the Strategic Learning Achievements Program (PELA). The Local Governance in Education model, carried out at a pilot level in two districts of the San Martin Region, is a contribution of strategic value because it links the decentralization component with the educational quality component and the territorial management approach. This model, based on governance within the educational institution, focusing on learning and student development with the assistance of trained teachers, family and community, is being validated. San Martin is the only region in the country with an educational management model at the district level, which the Regional Government has extended to eleven additional districts. Meanwhile, MINEDU’s efforts have been oriented towards Regional Educational Management models. SUMA Program’s milestones in decentralization policy and at the institutional framework  Support to the Ministry of Education for the formulation of its Law on Organization and Functions (LOF), which by November 2014 had not been approved.  Design of a costing tool to improve public expenditure and provide guidelines for the preparation of Public Investment Projects (PIP). A cost study for rural public education.  Design of the Monitoring System of Regional Education Policies (SSII-PER) for the National Education Council.  Participation in activities of the Roundtable for the Fight against Poverty (MCLCP) to foster the commitment of regional presidential candidates to governance agreements and the National Education Council’s National Common Agenda.  Design and implementation of the Local Governance in Education model (MGL) in two pilot districts of the San Martin region. The model transitioned school governance from the education institution to the whole of the district’s institutions under the leadership of the Local Management Committee 1 Innovation in Decentralization and Active Schools (AprenDes) was a 15.4 million U.S. dollar USAID project, implemented from July 2003 through August 2009. 9 for Social Development (CGLDS). The CGLDS is responsible for preparing the Annual Work Plan for the Improvement of Learning (PATMA).  Technical assistance for the preparation of ten PIP in different regions: three in San Martín, two in Ayacucho, one in Ucayali, one in Amazonas, one in Junín, one in provincial Lima (excluding Metropolitan Lima), and one for the Ministry of Education, for a total of 150 million soles. Three of these PIP have been implemented and three are being implemented. SUMA Program’s milestones in the quality of education component and in teacher development and professional training  Development of a study on identification criteria for Intercultural Bilingual Education Institutions, which was selected as an input in the creation of national standards for the registration of such institutions (RM 008-2012-ED).  Support to the CNE to conduct regional discussion meetings, pedagogical congresses, studies, and the identification of experiences. Support culminated in the elaboration of national standards for the Good Teaching Performance Framework (RM 0547-2012-ED).  Update of AprenDes materials used in the San Martin Region: Self-Teaching Notebooks (how students must learn), Curriculum Paths (what should students learn month by month), and the Teacher Compass (methodological strategies).  Training of regional teams of pedagogic facilitators and coaches for curricula strategies and contents in the areas of mathematics and communication. The training program was an input for the training of coaches of the Strategic Learning Achievements Program (PELA).  Design and implementation of management training programs in pilot districts for teachers, school principals, trainers, and pedagogic coaches. Strengths and Weaknesses At an internal level, SUMA had a team of competent and committed professionals, some of which had participated in the AprenDes program. Among the internal factors that limited attaining major achievements, the SUMA team had difficulties adapting to a different contract mechanism –instead of a Cooperative Agreement- and to numerous contract modifications, which required time and effort to negotiate. The changes in contractor, leadership and focus also contributed to moments of stress and uncertainty for the team. At an external level, the team obtained a good degree of recognition and approval, especially in the San Martin and Ucayali regions, where their intervention and technical assistance was more sustained over time. However, the program suffered from tension and lack of consensus among education sector authorities and functionaries resulting from the pressure to decentralize management, achieve results, improve learning, and move from policy guidelines to effective implementation. Conclusions  SUMA supported decentralization reform processes without inserting new priorities in the sector agenda while responding to the demands of MINEDU and other strategic partners. 10  SUMA sought to align itself with changes in the education sector’s political and institutional context, which required programmatic changes that could not keep pace with the changes in the context.  The program design did not account for the opportunities and threats posed by the implementation context, nor did it have the capacity to manage these risks as they arose.  SUMA carried out a pedagogical model with a package of methodological tools (classroom / school / community) appropriate for rural multi-grade schools, which contributed to the policy of quality and equitable education.  At the district area level, SUMA developed a local governance model that put into practice what education policy proposed in terms of a decentralized management model oriented towards the achievement of learning.  SUMA contributed to the policy of teaching coaching at the national level with its proposal for capacity building in the framework of the Strategic Learning Achievements Program (PELA).  The evaluation team has not find enough evidence to validate the hypothesis that stronger coordination and response capacity from the different government levels contributes to improve learning achievements. Recommendations  During the strategic planning for a policy and regulatory reform intervention, identify and plan to monitor all critical assumptions and risks; for implementation, identify a flexible management approach that allows regular adaptations to an evolving environment.  Improve monitoring of the intervention, defining hierarchical and articulated results, and designing indicators to measure processes, outputs and outcomes.  Ensure that the project timeframe is sufficient to allow for the completion of the different project phases, budgeting time for adjustments as reasonable.  Include a gender focus on interventions that seek to improve learning, given the differences measured between boys and girls in the results of the national learning achievement test.  Link primary education with pre-primary education based on the evidence that early childhood stimulation contributes to reading and writing during the initial years of primary school.  Design specific strategies to improve learning among girls and boys that do not reach level 1, based on an analysis of root causes of poor performance.  Apply knowledge generated from studies and research to consolidate education policies and interventions  Support MINEDU to establish a national observatory of pedagogical best practices, including those generated under SUMA.  In San Martin, accompany the evaluation of the Local Education Management Model in order to complete the validation process that SUMA initiated. Look to articulate the local model within the framework of the Regional Educational Management Model, defining the roles and linkages of the different actors. 11  In San Martín, support and strengthen the implementation of the Local Education Management model in the eleven new districts, engaging dynamic economic actors from the productive sectors (for instance, those of the milk basin, or the coffee basin) 12 PURPOSE AND EVALUATION QUESTIONS Purpose of the Evaluation The evaluation of the Peru Quality Basic Education Reform Support Program (SUMA) seeks to determine whether the program’s design responded to the issues that gave rise to it, whether the outcomes were the expected ones, and whether the manner in which they were accomplished was the most appropriate. This performance evaluation focuses on two key Program components: policy and institutional framework for education decentralization and teaching quality. The Program proposed the following results related to these components:  Participatory and decentralized education management  Improved teaching quality The evaluation has two specific objectives: first, critically analyze the level of achievement of the expected outcomes related to the policy and institutional framework for decentralization in education and teaching quality and, second, identify success factors, lessons learned, and key recommendations derived from the components evaluated, which may be useful in the context of the new USAID Peru cooperation strategy. USAID Peru staff are the main users of the results of this evaluation. Within the cooperation strategy framework (2012 – 2016), said results are expected to be integrated into the design, management, monitoring and measuring processes of new initiatives. MINEDU officials, either at the central or regional levels, are also users of the results. They will be in charge of identifying the benefit of decentralized education management, as well as teacher training and professional development and potential institutionalization of the lessons learned. Members of various networks, such as academic, international cooperation, and civil society networks, are also included within the recipients of this evaluation. The results presented herein can serve as point of reference for their institutional work. Evaluation Questions The objectives mentioned above correspond to four evaluation questions. The first two are directly related to the components of the SUMA program, and the following two address the lessons learned that could be extracted from the components evaluated.2 2 Specific questions derived from each evaluation question are described in the Annex 1. 13 The evaluation questions are the following: 1. To what extent did the SUMA Program promote specific changes in the educational decentralization policy and institutional framework? 2. To what extent did the SUMA Program promote quality education strategies and models that favored improving student achievement? 3. What factors (both internal and external) affected (either positively or negatively) the achievement of the expected results? 4. What practical knowledge can be derived from the evaluation of the SUMA Program and applied to the process design of new initiatives within the framework of USAID Peru’s new cooperation strategy? Each main question contains sub questions related to: the prioritization of issues; the relevance and link of the proposal to prioritized issues; the reason for programmatic modifications; the processes developed by SUMA for strengthening policies and evidence; best practices identified by the Program and the level of ownership by the sector; key contents and processes implemented by SUMA; dialogue, capacity-building and decision-making promoted by SUMA; functionality of the Program’s organizational structure; internal and external factors that contributed or limited the achievement of results and unforeseen decisions made by SUMA. The findings of the evaluation are presented according to the above-described questions and sub questions. 14 BACKGROUND The agreement between the United States Agency for International Development (USAID) and the Government of Peru in 2008 (No. 527-0423), states that USAID will collaborate with the Peruvian Government to “Improve the quality of Basic Education in disadvantaged (geographical) areas.” For this purpose, it will identify key factors for improving the quality and equity of basic education, support the efforts of the Peruvian Government to achieve systemic and macro institutional reforms, and develop capacities at the national, regional, and local levels. The Agreement aims to achieve two results: 1. Strengthening participatory and decentralized education management through an improvement of the institutional framework and policies at the national, regional, and local levels, by supporting and developing planning, organization, management, funding, monitoring, and evaluation skills at the regional/local levels. Areas for strengthening include planning and decision-making information systems; developing strategies to increase investment in education; creating public-private partnerships for public education; supporting initiatives by the Peruvian Government and civil society to promote dialogue on education policy and push for accountability regarding educational processes and outcomes. 2. Improved teaching quality. For this purpose, the improvement of teacher training systems will be supported by systematizing and leveraging successful national and international experiences, designing in-service training programs for multi-grade rural school teachers, and developing educational standards and materials. Likewise, technical assistance will be provided in order to assess innovative practices with potential application on a larger scale as part of national policies, such as teacher coaching strategies, curriculum diversification processes, use of technologies in education, math-teaching proposals, among others. The contract for the development of the Quality Basic Education Reform Support Program (SUMA) was within this framework. The purpose of the SUMA Program was to provide technical assistance services to the Peruvian Government to support reform initiatives aimed at improving the quality of basic education and in particular, student learning outcomes. Initially, the intervention was implemented at the national and regional levels, yet towards the end, it concentrated in a single region: San Martin. Between 2009 and 2013, the Program was the main implementing mechanism of USAID Peru education priorities and reflected the transition to a different intervention style: from direct implementation in schools, to support and technical assistance in policy priorities defined by the counterpart, i.e., the Ministry of Education (MINEDU). It was thus established in both the SUMA Program’s original proposal as well as in its subsequent amendments:3 “… the activities under this contract shall support the government’s basic education plans and priorities. They are aligned with Peru’s strategic and operational plans, such as the long-term National Education Project (Proyecto Educativo Nacional, PEN), Education for All, and the sector’s Multiannual Strategic Plan, as well as the priorities established during the Meeting of Ministers of Education at the 2008 Asia- 3 This reference was taken from the Spanish version of the original contract. In subsequent amendments (made based on program adjustments), the reference concerning this alignment remains unaltered. 15 Pacific Economic Cooperation Forum.” (Contract AID-527-C-09-00001 / Section C.1.4) For the evaluation team, this initiative sought to promote and consolidate policy reforms in Peru’s education sector. The nature of the SUMA Program, in turn was one of political advocacy and validation of proposals, tools, strategies, and materials that could constitute as evidence (empirical demonstrations) that would support the processes of formulation and institutionalization of the reforms underway within the education sector. Unlike the Innovation in Decentralization and Active Schools (AprenDes) program, developed between 2003 and 2009 and which mostly favored teaching and institutional management components inside the schools (Instituciones Educativas, IIEE), the SUMA Program expanded its scope of intervention. It assumed the strengthening of education’s decentralized and participatory management capacity as a work approach, without setting aside the improvement of educational processes at the classroom level. Operating Mechanism According to the global policy established by USAID, the operational implementation of initiatives such as the SUMA Program required establishing a contractual relationship with a North American operator. USAID signed the initial contract in September 2009 with the Academy for Educational Development (AED), and in June 2011 with Family Health International 360 (FHI 360).4 In both cases, these operators recruited local staff (permanent staff and short-term consultants) for the implementation of the activities included in the aforementioned contract. As an operating mechanism, the contract is different in nature from a Cooperation Agreement and defines a different margin of program flexibility and adaptation. In the life cycle of the program that preceded SUMA -AprenDes- the contract mechanism was a cooperation agreement. This type of mechanism favored a flexible adaptation of the program useful for the demands of the changing environment. In the opinion of the local implementers – who came from working in AprenDes under a different operating modality – this flexibility was not evident throughout the life cycle of the SUMA Program. The contract that governed the SUMA Program was characterized by specific activities or requirements, according to the wording thereof, within certain timeframes, and under the assumption that its fulfillment led to certain achievements. The activities were measured through indicators directly related to them, and thus, without strategic and cumulative contribution to estimate the level at which the program achievements were reached. Products became visible by means of deliverables, such as balance sheets, reports, diagnoses, recommendations, all of which constitute inputs. This type of result structure was not different from the one developed in the framework of the AprenDes Program. However, according to the SUMA operators, what was different was the level of rigidity due to it being based on a contract. 4 It is not part of this performance evaluation to explain the reasons for the change of subcontractor; however, as it is mention later, said transition affected the program’s implementation to some degree. 16 From the perspective of those who designed the SUMA Program and determined a structure of activities geared toward the achievement of certain results, everything may have made sense; however, with time, some of the activities ceased to be relevant. This is what the implementing team experienced when it identified that teaching performance evaluation was of no interest to MINEDU, and eliminated it with a contract amendment almost at the end of the period.5 Thus, within the framework of an initiative that sought to support emerging processes of sectoral policy reform and given the (new) demands coming from the daily work with counterparts and allies and the windows of opportunity that opened up in certain circumstances, to the operators, the contract proved to be an inflexible management tool and operating mechanism. That lack of flexibility resulted in long and draining processes to present, discuss, negotiate, and ultimately agree on changes to the program proposal. In a way, the perceived lack of flexibility of the SUMA Program conflicted with one of the principles used to define the initiative —repeatedly ratified with each amendment—. Work around sectoral reform processes requires a rapid response to any changes that could be posed by the counterpart (i.e., MINEDU). “USAID recognizes that reform situations are fluid and that the needs and priorities of the host country and USAID can shift. Within this context, it shall be necessary for technical assistance to remain flexible in order to respond rapidly to the needs of the host country. When political will, resources, and vision all come together, the contractor shall be ready to seize the moment.” (Contract and amendments 06 and 11) 5 Requirement 2.4 defined in the original contract established that the SUMA Program would support the creation of teaching performance evaluation systems. A subsequent amendment eliminated such requirement. 17 EVALUATION METHODS AND LIMITATIONS The purpose of the evaluation was to identify whether the initiative’s design responded to the issues that gave rise to it, whether the outcomes were the expected ones, and whether the manner in which they were accomplished was the most appropriate. The evaluation team applied a mixed method, which combined the analysis of the documentary information available and the qualitative information obtained from interviews carried out in Lima and the regions of San Martin and Ucayali. The program did not have a baseline that would allow identifying the progress made with respect to an initial situation. The evaluation comprised three stages described below. Evaluation Design Stage The team operationalized the questions about the two key components of the SUMA Program –focus of the evaluation–, as well as the policy and institutional framework for decentralization in education and teaching quality. During this stage, the absence of a logical framework and of a cascade of problems in the Program’s design became clear, whereupon the development hypotheses were rebuilt based on the reading of the components, expected results, and operational strategy to achieve them. Likewise, an array of variables, interview guides and questionnaires for fieldwork, the list of key stakeholders to be interviewed, and the fieldwork plan were developed. Field Work Stage Prior to the fieldwork, the evaluation team analyzed the Program’s documents; the contract and its amendments, monitoring plans and its reports; and the technical reports corresponding to the execution stage. Next, semi-structured interviews were conducted with selected informants in Lima, San Martin, and Ucayali. The interviews were recorded with the consent of the interviewees. The main criterion for selecting the informants was their connection to the Program. Likewise, their representation within the public sector and civil society was taken into account. Informants from the following sectors were selected: a) Public sector: authorities and officials of the Ministry of Education, Regional Government, Regional Education Directorate (Dirección Regional de Educación, DRE), Local Educational Management Units (Unidades de Gestión Educativa Local, UGEL), district mayors and their government teams. b) Civil society: representatives of the National Education Council (Consejo Nacional de Educación, CNE) and the Roundtable for the Fight against Poverty (Mesa de Lucha contra la Pobreza, MLCP), both national and regional. Independent consultants and opinion leaders in education matters, as well international cooperation agencies with education projects in the regions covered by the Program were also included. c) Members of the SUMA Program. 18 In total, the evaluation team interviewed, 53 informants 36 of them in the regions of San Martín and Ucayali (see complete list in Annex 2). Table 1: Number of informants interviewed Lima San Martín Ucayali Total MINEDU/DRE 2 7 11 20 UGEL/Network 13 13 Civil Society 10 --- 3 13 SUMA Program 4 2 --- 6 USAID 1 1 Total 17 22 14 53 It is worth noting that in Lima, there were informants that did not agree to be interviewed arguing conflict of interest – two people from MINEDU, three from civil society, and a former Program collaborator. Analysis and Discussion Stage Once the fieldwork was completed, the next stage corresponded to analysis and triangulation of the information obtained from the two sources available: the documentation of the SUMA Program and the interviews conducted. For the analysis, the team transcribed the interviews and organized the contents in matrices by question. The evaluation team leader and the two education specialists used the matrices with the responses to the evaluation questions and the identification of evidences to discuss the initial findings with the USAID Peru team. The results of the Student Census Evaluation (Evaluación Censal de Estudiantes, ECE) served as an input to the evaluation report. The San Martin database was formally requested from MINEDU’s Quality Measuring Unit (Unidad de Medición de la Calidad, UMC), and summary indicators were calculated for the set of schools (IIEE) in the districts where the SUMA Program was implemented. Information on the status of public investment projects (PIP) promoted by SUMA was taken from the National Public Investment System’s (Sistema Nacional de Inversión Pública, SNIP) Project Bank. Limitations There are two sources of limitations for this evaluation: the Program’s design and the difficulties found during work in the field. Regarding the design, the evaluation team did not found documentation on the change theory that gave rise to the project. Likewise, the team did not found Program documentation about the planning (logical framework, problem tree), the intervention proposal, and a contextualized assessment with a strategic view that would consider the potential changes in the decentralization process or in education management priorities. 19 The evaluation period coincided with the closing of the SUMA Program, which limited the access to information and documents. On the other hand, the transition of authorities in the Ministry of Education led to the recommendation by USAID to postpone the interviews of key officers close to the project, whom subsequently, after leaving their posts at MINEDU, refused to be interviewed. Finally, the fieldwork in the regions coincide with school holidays, which affected the availability of informants at the school level. 20 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FINDINGS By focusing on the performance of the SUMA Program, this evaluation seeks to identify objective facts that will either support or deny the fulfillment of the outcomes expected, the manner in which said outcomes were achieved, and the identifications of unforeseen achievements. This section covers three components: education decentralization policies, teaching quality, and the practical knowledge gained from the initiative under evaluation. 1. Education Decentralization Policies Component 1.1. To what extent did the SUMA Program promote specific changes in the education decentralization policy and institutional framework? In its first component, the SUMA Program favored the strengthening of education management capacities at the national and sub national levels, in a context of decentralization. In this topic, SUMA contributed to the implementation of some policies proposed in the National Education Project (PEN) related to an effective and efficient decentralized management, with timely funding; focused on learning achievements (Objective 5 of the PEN), joint budget planning and programming; and coordination among national, regional, and local management levels and instances. Throughout the intervention, SUMA promoted specific changes in education decentralization (detailed below), aligned with regulated policies, and innovative with respect to everyday practices of restricted efficiency (real-world policies). Some of these changes were carried out effectively, and they are described in the paragraphs below. It also promoted important changes in the institutional framework of decentralized education management, some of which were implemented and had significant results, especially in limited local spaces. However, since the bet on institutional changes was focused on the adoption of MINEDU’s Law on Organization and Functions (Ley de Organización y Funciones, LOF), which after four years has not yet been enacted, the set of minor changes was missed out on certain strategic opportunities. 1.1.1: What were the education sector issues, prioritized in terms of education decentralization policy and institutional framework, that the SUMA Program sought to resolve? SUMA’s area of concern was the low quality of education, which causes poor learning results in students. Within this field, the Program addressed the inadequacy of decentralized education management to support services in schools in rural areas. The following are some of the prioritized causes underlying this situation:  Limited institutional and administrative capacity of the central, regional, and local government agencies to exercise their functions within a results-oriented management approach and 21 provision of quality education that will guarantee learning improvement;  The lack of coordination existing between budget planning and programming (at a decentralized level), which results in disordered investment and low contribution toward closing educational gaps;  Lack of clarity in the regulatory framework with respect to the intervention roles of the various sectors at the regional and municipal levels, which limits the exercise of an effective, efficient, and decentralized management;  Inadequate information systems for timely decision-making, oriented toward student learning achievement; and  Low participation and involvement of local authorities, officials from other sectors, parents, and the community in general, in education management. 1.1.2: How relevant and articulated was the SUMA Program’s initial intervention proposal regarding the issues prioritized in terms of the decentralization of education and institutional framework? The SUMA Program focused on bringing education management to its beneficiaries at the different levels in which it acted, i.e. students and parents, local residents, and social organizations and stakeholders at the regional and national levels. At each level, SUMA took steps aimed at improving the relationship between the beneficiaries and the institutions, agencies, and officers in charge of providing public education services. Stakeholders The coordination with the National Education Council (CNE) and the National Assembly of Regional Governments (Asamblea Nacional de Gobiernos Regionales, ANGR) was relevant (it was not continuous with ANGR), but it was strategically insufficient. At the juncture, the CNE had no greater weight on policy decision-making. On the other hand, through the collaboration with CNE, SUMA was able to influence the creation of an agenda that included discussions on several subjects, but particularly orienting towards decentralization of education. The evaluation team’s opinion is that such collaboration could not have any effect on the institutionalization of policies, given that the CNE had no real power, because it depended on its relationship with the political administration of the Ministry of Education (MINEDU). On the other hand, the Ministry of Economy and Finance (Ministerio de Economía y Finanzas, MEF), whom is in charge of approving decentralization initiatives due to their serious consequences in terms of budget management, did not take the CNE proposals into account, because they did not have effective and ongoing support by MINEDU. MINEDU’s Law on Organization and Functions (LOF) There is a constant demand for MINEDU’s LOF within the framework of the conception of the decentralization process presented by the SUMA Program, both in the studies by Chirinos6 and by 6 Chirinos, L. S. (2011). “Status of the education decentralization policies 2010-2011” (Estado de las políticas de descentralización educativa 2010-2011). USAID|PERU|SUMA, Lima 22 Ugarte, Arguedas and Ángeles,7 and in their internal documentation. The expressions are definitive and use words such as “it is necessary,” “it's a factor that hinders the decentralization process,” “its importance is huge,” “it defines the unity of the State and regional diversity…” However, given its repeated absence, it is pertinent to ask ourselves how the decentralization process could have advanced without the approval of the LOF. In the absence of the LOF, and in response to the years of waiting and the coexistence of several legal instruments, such as the General Education Act (Ley General de Educación, LGE) and MINEDU’s Organization and Functions Regulations (Reglamento de Organización y Funciones, ROF) (undergoing continuous revision), have led to the creation of the legal decentralization framework and several others. The new LOF is currently on hold because it interferes politically with the University Act. The LOF version proposed by the SUMA Program did not foreseen the new powers the Act entrusts to MINEDU. It had to be redrafted, not only to include the new institutional framework produced by the regional Governments, but also to adapt to this new higher education regulation. In addition, it must take into account the changes that the new administration will impose on MINEDU. This should mean an emphasis on management and oriented toward meeting spending goals, and the allocation of responsibilities to the regions. Within this complex policy context, the LOF could not be approved, the outcome could be different from desired one, or it could be the desired outcome but not be applicable due to conflicts with the emerging regional legal frameworks. However, its main risk revolves around being updated in relatively short timeframes (2 to 3 years), either during the next administration or by another sectoral administration. The SUMA Program did not take into account this context; without an alternative vision, all solutions were dependent upon the approval of the current LOF bill. Decentralized Management From the emphasis that the SUMA Program put on the development of decentralized education management skills at the micro level, the evaluation team identified that the program sought to bring the management of the education system closer to the local government. This was done under the assumption that better coordination and coordinated responsiveness of governmental stakeholders who influence the provision of education services, would increase joint effectiveness that in turn was likely to improve learning achievements. The main assumption of the SUMA Program was the notion that higher levels of school autonomy (and therefore, local governance) translate into better learning outcomes. It should be noted that efforts were made to illustrate this link, both through conceptual endorsement (clear in the minds of their stakeholders) as well as in the generation of tools to measure it. However, the Program failed to document this relationship. Likewise, the evaluation team believes that the strategic contribution of the SUMA Program consisted of coordinating and pinpointing some policy elements that are present in national and regional regulations, in education institutions, and in local government authorities that are related to the decentralization of education. The SUMA Program acknowledged that the pedagogical model used since the AprenDes intervention required further development that would resulted in effective participatory processes at 7 Ugarte, M., Arguedas, C., & Ángeles, N. (2012). “2009-2009 Education Decentralization Report Card” (Balance de la Descentralización de la Educación 2009 -2009). USAID|PERU|SUMA, Lima 23 the classroom and school level, and that would articulate these processes with decision-making processes in the district. One of the greatest achievements of the SUMA Program is the fact that these local changes have been effectively implemented in two districts, and enthusiastically welcomed and highlighted by their stakeholders. Yet, ensuring its continuity constitutes the greatest challenge. The management skills strengthening process implemented by the SUMA Program constituted an institutionalization at the micro level of the entire educational system’s actions. In other words, in practice, the Program was able to promote the integration of the education sector agencies – from the IIEE and up, including networks, local and regional governments, CNE, and other government stakeholders – within the framework of their responsibilities according to the regulations. 1.1.2: What were the reasons behind the set of programmatic modifications made throughout the program implementation, and what was gained/lost by the SUMA Program from to the expected outcomes? Throughout its lifecycle, the SUMA Program experienced programmatic modifications. Each modification was made through an amendment to the original contract. Between September 2009 and December 2013, there were 15 modifications. While some of them were related to budget modifications, others reflected a variation in program priorities. Other amendments directly redefined the Program’s action plan. In the evaluation team’s opinion, the programmatic modifications were due to the SUMA’s Program exposure and sensitivity to changes in the political and institutional environments in which it was immersed. Figure 1 summarizes these changes: (1) at the national level, two different governments, each with two Ministers of Education. (2) Continuity in the regional governments of San Martin and Ucayali and change in the governments of Ayacucho, Junín and Lima. (3) Changes in the Regional Education Directorates (DRE) – only the case of San Martin is illustrated. (4) Changes in USAID’s education strategy. (5) Changes in the contractor and management team. Figure 1: Changes in SUMA’s political and institutional context President President President President President President President President Minister Education New education strategy Alignment to new education strategy USAID/Peru: Education Office Change of SUMA supervisor (COR) Health & Education Office Merger Change of SUMA supervisor (COR) Initial contractor: AED, Director Palacios Beginning of Contract: September 2009 Acquisition FHI 360 Director Modification Modification 24 In general, it is possible to assert that the Program stood at the junction of four moments of transition, which affected the programmatic changes:  First moment (2009): disagreement with respect to government priority. The initial design of the SUMA Program was not consistent with the government’s provisions with respect to decentralization. This led to some early adjustments to the direction of the Program.  Second moment (June 2011): change of American contract operator. When the contract’s technical and administrative management passed from AED to FHI 360, there was a momentary rupture in the Program’s execution.  Third moment (August 2011): The change of government in Peru in July 2011 resulted in a “third generation” of programmatic changes. New opportunities arised from the fact that the new officials already had a favorable view of SUMA (such as the Vice Minister of Institutional Management, Fernando Bolaños, or the Director of Elementary Education, Flor Pablo) or of some of its key counterparts (Minister Patricia Salas before the CNE). New sectoral priorities were proposed, to which the SUMA Program had to answer.  Fourth moment (June 2012): Following the redefinition of USAID’s global education strategy, a long discussion on the phonological approach and the textual communicative approach on student learning opened up in the SUMA Program. This debate led to an investigation into literacy in the country, which in turn had an impact in the educational materials produced by the Program: the Self￾Teaching Notebooks (Cuadernos de Autoaprendizaje), the Curriculum Paths (Tramos Curriculares), and the Teacher Compass. (Brújula Maestra). Amendment No 6, effected in mid-2012, was a key moment in the programmatic redefinition process of the SUMA Program. Amendment No 11, of May 2013, constituted the final adjustment of the terms of reference of the SUMA Program. Comparing the two amendments with the original contract of September 2009, allows us to identify both, programmatic continuities as well as changes (Table 2). In amendment 6, some of the commitments assumed by the implementing team were clarified. However, it also represented a moment of internal, long, and extended debate about the approaches to daily work, after which there were changes in managerial staff. In turn, amendment 11 did not represent a substantive programmatic change, but it specified the focus of the work in San Martin in the last year. The first change identified (at the indicators and deliverables level established in Requirement 1.3) formalized the reduction of the areas of intervention, which, in practice, began to occur after the change of regional authorities. The Program withdrew from Junín in May 2011, when it was not able to address the regional literacy priority due to it being outside of its mandate. Likewise, the Program withdrew from the Lima Region in June 2012, after several changes in DRE authorities, and ongoing disagreements between the President and Vice President with respect to the education issue. The program left Ayacucho in October 2012, due to context difficulties that required a greater amount of resources that were ultimately concentrated in one region. In all cases, removing the regions from the program was mandated by USAID. 25 Table 2: Changes in the SUMA Program’s outcomes structure Original Contract (September 2009) Amendment 06 (June 2012) Amendment 11 (May 2013) Achievement 01 Requirement 1.1 Strengthen educational decentralization policies and institutional framework There was no change There was no change Number of indicators 4 There was no change There was no change Number of deliverables 1 There was no change There was no change Requirement 1.2 Review best practices for decentralized management There was no change There was no change Number of indicators 2 There was no change There was no change Number of deliverables 1 There was no change There was no change Requirement 1.3 Strengthen education management systems and processes It changed There was no change Number of indicators 5 No number of regions is specified Reduction in the number of IIEE supported Work focused on San Martín Number of deliverables 3 Reduction in the number of IIEE supported There was no change Requirement 1.4 Enhance budget allocation and quality of expenditures for education There was no change There was no change Number of indicators 3 Reduction in the number of regional governments supported (PELA) There was no change Number of deliverables 2 There was no change There was no change By reducing the number of regions, there was a decrease in the number of demonstrative educational institutions (IIEE) where the achievements in learning were going to be measured, to match the number of IIEE in the districts where the Program was carried out. This decision involved mapping the number of observation units, which favored a better concentration of efforts and resources by the SUMA Program. Later, this requirement is redefined once again through amendment 11 by focusing the last year of operation of the SUMA Program on the San Martín Region. The second change (at the indicators level established in Requirement 1.4) was related to the reduction in the number of regional governments supported in the implementation of the Strategic Learning Achievement Budgetary Program (PELA). Again, this reduction resulted in a greater capacity to offer specialized technical assistance. All these programmatic changes – formalized through amendments to the original contract – involved long, complex discussion, negotiation, and agreement processes among all the parties involved: the 26 national implementation team, the American operator (AED or FHI 360), and USAID/Peru.8 Generally, the evaluation team believes that the programmatic changes that took place during the implementation of the Program – and which were recorded in the amendments to the contract9 – could be the result of an ambitious design that did not stem from a critical assessment of the actual dynamic of the country’s education sector.10 With some of those changes, the Program gained a higher level of programmatic precision, in other words, an adjustment and alignment according to the sectoral agenda (at the national and regional levels). This happened, for instance, with the elimination of the requirement related to the improvement in teacher training. On the other hand, the Program also gained operational capacity by desisting from undertaking some of the tasks initially laid down, thus being able to concentrate on others that better fit the realities found in the field and in the institutional environment. 1.1.3: What did the strengthening process of the policy and institutional framework for decentralization in education implemented by the SUMA Program consist of, and what objective evidences could be identified as outcomes? SUMA contributed in several ways to the strengthening of the policy and institutional framework for decentralization in education by: a) Generating knowledge: At the beginning of its implementation, the SUMA Program performed an analysis of the current situation and the gaps in decentralization policies and practices, based on studies conducted by specialists and by the development of a consultation and discussion process with key MINEDU staff and experts in education from civil society and regional governments. Years later, the reflections generated and proposals made were published,11 constituting elements of reference for policy-related discussions. 8 The national implementation team was made up of locally recruited staff and was responsible for the direct implementation of activities that would allow the fulfilment of the requirements. The team had direct and permanent relationship with the counterparts; they gathered all demands and needs for technical assistance in the education sector (at the national and regional levels). The American operator (at first AED and then FHI 360) was responsible for institutional representation and technical￾administrative accountability before USAID/Peru. 9 The programmatic changes identified herein correspond to the formal level of the initiative, which were reflected in official documents and had contractual validity. However, there were other changes that corresponded to daily operations, i.e., to the implementation of the specific strategy aimed at achieving the requirements, which were not documented. 10 The evaluation team believes that the Program’s design made assumptions of linearity in fundamentally political process, such as the decentralization process, which in practice has shown progresses and setbacks that constitute a threat to the efforts of any project. Likewise, it assumed that the stakeholders’ power relationships would remain after the electoral process, such as, for instance, the CNE’s power to exert influence. In addition, it assumed a relationship between decentralizing efforts and improvements in learning, which has not been confirmed. Finally, even though the issues in education management and quality were identified, there was no in-depth analysis of the factors that explain their status. 11 The following documents contain the reflection and discussion generated on that regard: “Descentralización de la gestión educativa [Decentralization of Education Management] (Chirinos, 2012)”, “Estado de las políticas de descentralización educativa 2010 [Status of Educational Decentralization Policies] (Chirinos, 2012)” and “Balance de la descentralización de la educación 2009 – 2012 [2009-2012 Educational Decentralization Balance] (Ugarte et al, 2013).” All these publications, in their digital version, are available at: http://www.sumaeducacion.pe/documentos1.php 27 b) Providing technical assistance to national (MINEDU, CNE) and regional agencies in the implementation of their education priority agendas: Based on the coordination with various national units and offices of MINEDU, the SUMA Program answered to a specific request: to provide technical assistance in the drafting process of the preliminary version of MINEDU’s Law on Organization and Functions (LOF). At the time, said law was identified as a key component of the educational decentralization process, and therefore, the technical assistance requested was correctly framed within the range of actions that it could carry out.12 The specific contribution of the SUMA Program was to stress the implications of decentralization in the operation of the education sector, especially with regard to the coordination of different levels of management in the strategic and budget planning fields. Eventually, all discussions around MINEDU’s LOF ceased, since it stopped being a priority in the context of decentralization. Within the framework of the National Education Council (CNE), SUMA was also involved in and supported the preparation of the 2011-2016 National-Regional Common Agenda on Education Policy, where one of the points for reflection was decentralized education management.13 It developed, in collaboration with the CNE, a monitoring and information system for regional education projects (SSI￾PER), to give an account of progress and constraints in the implementation of regional education policies. To date, 25 regional governments have updated their data to 2013, including data in the areas of planning, budgeting, program and project management, institutional organization, professionalism, and citizen participation. c) Promotion of Policy dialogues: Through the promotion of regional dialogue around educational development policies, the SUMA Program sought to create and strengthen spaces for discussion about educational decentralization. In the electoral context (right in the middle of the project’s life cycle), these dialogues served to establish Governance Agreements14 among the candidates, and to ensure the fulfillment of the commitments after the election. Candidates of regional government pledged first to develop policies that will allow comprehensive attention to early childhood, with focus on human development and intercultural. Second, to implement policies that would contribute to the comprehensive education of students of regular basic education (EBR) and intercultural bilingual education (EIB), with sustained quality learning on intercultural, cultural identity, equity in gender relations, and environmental responsibility. The creation of discussion forums championed by the SUMA Program is part of the program strategies that the cooperation promotes to support sectoral reform formulation and implementations processes. 12 During the administration of García, the effective transfer of functions from the central level to the regional level provided for in the Decentralization Bases Law, required the drafting of the LOF, which was an important condition to define what responsibilities, functions, and powers would remain with MINEDU and which, consequently, would transfer to the regional Governments. 13 For the SUMA Program, the CNE was an important advocacy platform and reflexive echo around the decentralization of education management. See the following link: http://www.cne.gob.pe/images/stories/cne-publicaciones/ AGENDA%20COMUN%20politicas%20para%20el%20quinquenio%20julio2011.pdf 14 http://www.mesadeconcertacion.org.pe/elecciones/cuadros/metas/ 28 During the government change process (2011), the SUMA Program participated in the identification and formulation of priorities that needed to be addressed by the new MINEDU administration. The formulation of these priorities was an opportunity to return to the agenda of education decentralization, as well as other topics of interest. And so it did, along with other institutional stakeholders invited who managed to define some guiding documents that served as a reference for the team that was assuming functions in MINEDU, under the Minister, Patricia Salas. The technical assistance provided by the SUMA Program during the government change process consisted of policy recommendations aimed at improving public spending, decentralized management, and funding, in addition to the preparation of a brief study on the costs of rural public education. I was a sign of a programmatic openness and adaptation to context demands – in the face of the initial priorities of teaching evaluation and municipal management. However, all these efforts around strengthening the policies and institutional framework for decentralization in education were diluted by the lack of willingness of the government to continue with the decentralization process. 1.1.4: What were the decentralized education management best practices identified and debated by the SUMA Program, and what was the level of ownership of said practices by the education sector? The SUMA Program developed several studies about decentralized education management experiences at the national and international levels. Cases from South Africa, Nicaragua, and Colombia, as well as experiences in the San Martin, Arequipa, and La Libertad regions of Peru, were analyzed. The content of these studies were widely discussed in various forums and published starting in 2012.15 The SUMA Program thoroughly fulfilled the requirement of reviewing, evaluating, and sharing set of best practices in education decentralization with involved stakeholders. Based on international best practices, it made a critical analysis of the local governance model that had been implemented in San Martin. The analysis found that bottleneck experiences were related to conditions necessary for intergovernmental coordination at the local level, especially the clear definition of the role of each stakeholder and the inter-sectoral work assumptions. MINEDU has continued the discussion on decentralized education management and between 2013 and 2014 has published a series of "working papers" on the subject, which present the roles of the different levels of government, inter-sectoral coordination, management tools, and other elements that the SUMA model has put into place during its implementation. 15 The following studies give an account of the review of experiences: “Decentralized Model of Education in South Africa” (Crouch, 2013), “School Autonomy in Nicaragua: Politics, Education, and the Social Contract, 1993-2007” (Arcia, 2012), “Decentralized Education Management: The Case of Bogotá, DC” (Niño, 2012), “Systematization of Educational Management Models of the Regional Governments of San Martin, Arequipa, and La Libertad” (Manrique, 2012), “Analysis of the Implementation of Decentralization and Modernization Reforms in Colombia” (Meade et al, 2012). All of them can be downloaded from the following web page: http://www.sumaeducacion.pe/documentos1.php 29 1.1.5: Which key contents and processes were implemented by the SUMA Program in strengthening of education management systems and processes framework, and what objective evidence could be identified as outcomes? Application of the Local Governance Model to Education In the districts of Pajarillo and Cuñumbuque, the SUMA Program applied a pilot experience of the Local Governance Model (Modelo de Gobernanza Local, MGL). This model not only consists of teaching strategies, educational materials, and assessment tools, but it also integrates a management strategy for work in education. Following the lead of the local authority, and in coordination with the education sector, specific targets are established (based on the analysis of previous evidence), progress is measured (based on the application of instruments), and success is celebrated (as incentive for the effort).16 This experience constitutes an inter-sectoral coordination effort in a territorial space, where local government, representatives from the various governmental areas, and the community make up the Local Management Committee for Social Development (CGLDS), analyze the issues, make decisions, and implement actions aimed at improving student learning. In pilot districts, the members of the Committee clearly expressed that even though this is the result of SUMA’s support, it is led by the Mayor and has been incorporated into the organizational structure of the Municipality, for which purpose there have been modifications to the regulations, organization and functions manual. From the decisions of the Committee, a budget has been assigned, both for inter-sectoral actions in support of the educational community, as well as to arrange training and support for teachers. At the beginning of the Model, these actions were carried out within the framework of a joint plan that later incorporated cross-cutting goals, and is now considered one single document, which is the Local Annual Plan for Learning Improvement (PALMA). The CGLDS meets regularly to monitor progress towards the achievement of the Plan’s goals. They identify when the progress is poor and take actions to overcome these deficiencies. Even though the Model’s design includes vertical coordination among the various management levels – educational institutions, Networks, Local Educational Management Units (UGEL) and Regional Education Directorates (DRE) – that coordination does not yet fully work in the application of the pilot project in Pajarillo and Cuñumbuque. On the other hand, the Regional Government’s Organization and Functions Regulations (ROF) does not include the networks as formal DRE agencies; however, it does include “promoting the creation and operation of educational networks” as one of the UGEL functions, which it ultimately does, by making an UGEL employee responsible for the network’s management functions. In the Local Governance Model, the SUMA Program put together an improved pedagogical proposal of active schools with a local management component, based on the assumption that it was necessary to have a management support linked to the pedagogical aspect in order to produce improved student learning. In San Martin, the DRE values the Model, assuming it as part of the Regional Educational Management Model. They have begun replicating the experience in 11 additional districts. 16 The pedagogical component of the Local Governance Model includes educational materials (curriculum paths, teacher compass, Self-Teaching Notebooks, and teacher training program in math and communication), and progress evaluation tools. The management component includes management tools (PATMA, CGLDS, and joint plan). This performance evaluation does not analyze each of these tools, but it does analyze their relationship taking into account the purpose of the SUMA Program. 30 Tools for Regional and Local Management Based on the experience jointly accrued with UNICEF in Ayacucho and Ucayali, SUMA contributed to the coordination of the long-term and medium-term planning of the Regional Education Projects (PER). It participated, in coordination with MINEDU’s Office of Support for the Administration of Education (Oficina de Apoyo a la Administración de la Educación, OAAE), in the preparation of the technical guide for the formulation of Medium Term Plans (Planes de Mediano Plazo, PMP), and also contributed to result￾oriented institutional operations planning (POI). In support of the regional/local planning and management processes, SUMA provided specialized technical assistance in the design of a system to manage key information at the sub-national level, called “Strategic Information for Education Management at the Regional and Local Levels.” Certainly, the management planning tools mentioned above are mandatory in public administration at all levels of government. In San Martín, stakeholders appreciate SUMA’s promotion of participatory planning processes and its contribution to the 2013-2016 Mid-Term Plan. But they also recognize DRE’s leadership in coordinating the Institutional Operations Plan (POI) with the Mid-Term Plan, and in generating a block chain from the educational institutions’ PATMA with the Networks’ POI, these with the UGEL’s POI, and in turn, these with the DRE’s POI. Conversely, in Ucayali, SUMA’s support was not able to come to fruition at the DRE’s institutional operational level due to regional administrative bureaucracy obstacles. In this sense, SUMA contributed to capacity building in management systems and processes. However, the evaluation team has not been able to find any evidence with respect to the operational definition or list of capabilities that were developed —in relation to education systems and process management in the context of decentralization, neither in the documentation reviewed nor in the statements from the informants consulted. Likewise, it has not been able to identify the difference between these capabilities and those promoted by MINEDU in the rest of the country. 1.1.6: What were the processes promoted by the SUMA Program related to increased public investment, and what objective evidence could be identified as outcomes? San Martin’s Regional Education Project (PER), approved in 2006, stems from a Public Investment Project (PIP) deemed “pioneering” due to its focus on improving the quality of education. The implementation of PER was supported by AprenDes with the preparation of the new PIPs, some of which were resumed by SUMA at the beginning of its operations. The SUMA Program provided technical support to the design and viability of nine PIPs in the various regions where it worked, three of which were designed during the AprenDes period. The themes of these PIPs were different in each region, but they were generally all oriented towards improving the quality of education, whether by improving teaching-learning processes, developing teacher capabilities, or developing management capabilities (see table 3). In 2012, based on the experience gained, SUMA also supported the formulation of MINEDU’s PIP for decentralized management, which it seeks to socialize and establish, among other things, new education management structures in 24 DREs and 72 UGELs nationwide. The specific contribution of the SUMA Program was the drafting of PIP’s technical note. The approximate total amount of investment resources that the PIPs facilitated by SUMA were expected to generate was 150 million new soles. However, from the 10 PIPs developed, nine were declared 31 feasible; of those nine, two are in the investment phase, three in the implementation phase, and three have already been implemented, mobilizing 16.5 million. While a public investment project must be feasible prior to being implemented, a feasible project does not necessarily mean that it already has resources for its implementation. The different authorities of each sector or government level and their management bodies define what project will be implemented and when it will be implemented once it has been declared feasible. Such is the case of MINEDU’s PIP, worth 104 million soles, which has been in the investment phase since March 2013, but to which no resources have been allocated to date.17 The three projects already implemented amounting to 12 million soles (San Martin, Ucayali, and Amazonas) were declared feasible prior to 2009, and implemented between 2009 and 2013. The three projects that are currently being implemented were declared feasible between 2009 and 2011 amount to 14.7 million soles, but to date, only 4.2 million of that amount has been implemented. Of the latter, one is being executed at the provincial government level (Huanca Sancos, Ayacucho) to promote an active schools approach. The PIP support and management championed by the SUMA Program has been important because it sought to leverage the funds required to make and maintain the decentralization model and make the active schools methodology feasible through the development of workshops and learning circles (CIA) for facilitators, teachers, and school principals. In addition, the technical assistance rendered to the formulation and/or implementation of the PIPs has been very well appreciated both by stakeholders in regional and local governments, as well as by the technical staff of MINEDU. However, SUMA’s contribution is minimal compared to the practice already carried out by the regions in the preparation of PIPs. In San Martin, between 2009 and 2014, 516 PIPs in education were declared feasible, and 581, worth 276 million new soles have been implemented. Of those 276 million, 153 million have been implemented at the local level.18 While certainly most of these PIPs are for infrastructure and equipment improvement purposes, the MEF is also encouraging value-added PIPs focused on intangible19 factors, and even solely on intangible factors, specifically in the regions and in the education sector. Likewise, in this same note, MINEDU has worked on various budget initiatives. In this sense, the contribution of the SUMA Program with regard to the design and implementation of the PIPs was, in nature, technical within the framework of a national policy. The national treasure investment mechanism was already defined. The specific contribution consisted of making feasible the use of said mechanism to favor the education sector. 17 The project’s final study was approved and a Vice-Minister Resolution of MINEDU authorized its implementation in January 2014. In the second quarter of 2014, 10 million soles were committed to its implementation, for 15.2 million soles committed throughout the year. 18 Source: MEF, Monitoring of budget execution. http://apps5.mineco.gob.pe/transparencia/mensual/default.aspx?y=2014&ap=Proyecto 19 MINEDU considers feasible the following investments: curricular innovation initiatives, cultural adaptation of the educational material, teacher training, and measures geared toward improving the management capabilities of the educational service. The Guía Simplificada para la Identificación, Formulación y Evaluación Social de Proyectos de Educación Básica Regular, a Nivel de Perfil del MEF (Simplified Guide for the Identification, Formulation, and Social Evaluation of Regular Basic Education Projects at the MEF Profile Level, 2011). 32 Table 3: Level of implementation of the Investment Projects facilitated by SUMA SNIP Code Date profile approval Period of Implementation Record Status Region Profile Amount Allocated Amount Implemented Amount % Allocated % Implemented 34904 (*) 2006 2009-2010 Implemented San Martín S/. 1,956 S/. 1,695 S/. 1,695 87% 100% 69189 (*) 2007 2009-2012 Implemented Ucayali S/. 5,702 S/. 5,683 S/. 3,857 100% 68% 76010 (*) 2008 2009-2013 Implemented Amazonas S/. 5,871 S/. 7,297 S/. 6,676 124% 91% 123350 2009 2011-2014 Under implementation Ayacucho S/. 5,987 S/. 5,596 S/. 3,266 93% 58% 131794 2010 2011-2014 Under implementation Lima Provinces S/. 4,997 S/. 171 S/. 139 3% 81% 151844 2010 Feasible Junín S/. 5,876 S/. - S/. - 156154 2010 Investment Phase San Martín S/. 5,211 S/. - S/. - 169746 2010 Observed San Martín S/. 5,947 S/. - S/. - 185286 2011 2013-2014 Under implementation Ayacucho (Huanca Sancos) S/. 3,680 S/. 879 S/. 849 24% 97% 208856 2012 Investment Phase National S/. 103,988 S/. - S/. - S/. 149,214 S/. 21,321 S/. 16,482 14% 77% (*) PIP designed with technical assistance from AprenDes. Follow-up and implementation with technical assistance from SUMA. SUMA continued reporting it. Source: Annex 3 33 1.1.7: What were the internal and external factors that contributed / limited the achievement of the outcomes of the SUMA Program related to the educational decentralization policy and institutional framework? SUMA counted with the following internal factors contributing to the achievement of its outcomes:  The existence of a proposal and a qualified technical team that came from the previous AprenDes experience. The team was complemented with temporary staff for additional tasks, within the framework of a satisfactory organizational structure;  Funding availability and;  A flexible framework to respond to requests for technical assistance. Some of the internal factors that limited its outcomes include:  SUMA did not have a robust assessment linking its objectives with each other, in a causal manner and based on evidence;  The change in operator (AED – FHI 360) and;  The time taken by negotiations between the operator and USAID to amend the contract, to the extent that they caused instability in the project team. The external factors that allowed for the achievement of the outcomes were:  A process of decentralization (even with limitations in design) already underway in the country and on the agenda of educational priorities (PEN);  Measures underway such as the Strategic Learning Achievements Program (PELA);  The possibility of having public funding mechanisms for educations, Public Investment Projects (PIP);  At the regional level, the political will of the highest authority, in the case of San Martin;  The existence of regional education agendas (PER) and;  The willingness shown by the CNE to include the participation of cooperation organizations. Some of the external factors that limited the achievement of SUMA’s outcomes, both nationally and regionally, were the changes in authorities because of the scheduled elections. In both cases, there were changes in sectoral management and technical officials. At the regional level, the impact was greater, because they had to deal with situations such as those of Junín, Lima Region, and Ayacucho, which ultimately forced an early closure of the Program in those regions. It is no coincidence that the Program’s intervention had greater stability and continuity in San Martin and Ucayali, where regional Presidents were elected for a second term administration. 1.1.8: What unforeseen decisions did the SUMA Program have to make and what were their practical implications? One example of an unforeseen decision that took place after a shift in government priorities was the elimination of requirement 2.4 regarding teacher training. In the original contract, it was established that the SUMA Program would support accreditation and certification processes for teachers that graduated from Pedagogical Institutes (institutos superiores pedagógicos, ISP). To that end, it worked with the organization in charge of accreditation: National System for Evaluation, Accreditation and Certification of Education Quality (Sistema Nacional de 34 Evaluación, Acreditación y Certificación de la Calidad Educativa, SINEACE. However, the obstacles encountered and the redefinition of MINEDU’s priorities, forced a change in program that was formally expressed in amendment 06: adopting teacher performance evaluation as a new work axis, within the time context of the last stages of the administration of Alan Garcia. In this framework, the SUMA Program assumed the commitment of contributing to the design of the teacher evaluation system and the corresponding development of capabilities at the regional level. Sometime later, after a change in administration (beginning of the administration of President Humala Tasso) and the formation of a new management team in MINEDU, sectoral priorities changed once again, this time in favor of the new law on the teaching career, reason why the proposal for teacher performance evaluation ended up being truncated. In the evaluation team’s opinion, these programmatic changes reflect an adaptation effort by the SUMA Program with respect to the governmental and sectoral agendas. While the modification of the work axis did not yield the expected outcome, the SUMA Program did what it was set out to do: to provide specialized technical assistance according to the (changing) agenda of priorities of its main counterpart (MINEDU). From an efficacy point of view, resources were invested without achieving any results. 35 2. Teaching Quality The second axis of analysis of the performance evaluation is related to teaching quality. To the SUMA Program, improved education quality meant applying methods and strategies aimed at achieving student learning and improving teaching performance, taking the experience of Active Schools and AprenDes, and providing specialized technical assistance in teacher training and professional development. 2.1 To what extent did the SUMA Program promote models and strategies for improving teaching quality that were institutionalized in the sector at the national and regional levels? SUMA promoted a model of intervention called comprehensive pedagogical proposal (propuesta pedagógica integral), based on the experience of active schools and AprenDes, developed in the 2003- 2009 period in San Martin and Ucayali. SUMA was expected to validate AprenDes’ strategies and tools, and to develop new methodological and management tools, as well as to place this pedagogical proposal within the architecture of the Local Governance Model. The active schools methodology seeks to respond to the challenges posed by rural multi-grade, single￾teacher schools, and focus the intervention on improving learning achievements in comprehensive communication and mathematics. The reality of rural schools is characterized by a lack of methodology and relevant materials for teachers and students of different grades who share a single classroom, poor teacher preparation to respond to the needs and expectations of their students, lack of technical support and resources, and great difficulties to implement the school curriculum in the classroom. AprenDes’ experience with the Active Schools methodology was developed and validated in 275 elementary schools and Spanish-speaking rural populations in 11 provinces and 51 districts of San Martin and Ucayali. According to an outside evaluation of the experience20, some of the highlights included: (a) Contribution to the policy of educational equity, by offering a methodological design with educational materials and teacher training programs for rural, single-teacher, multi-grade schools; (b) Contribution, together with the Andean Center of Excellence for Teacher Training (Centro Andino de Excelencia para la Capacitación de Maestros), to the creation of the strategic learning achievements program (PELA); (c) Technical support to obtain funding for education through public investment projects (PIP); and (d) Contributions to medium-term planning. Following up on the previous experience, SUMA kept the intervention in Spanish-speaking populations, focusing on two districts of San Martin: Pajarillo and Cuñumbuqui.21 In addition, it provided support in the form of training and technical assistance to schools at a national level, including educational institutions in San Martin served by the PELA and in the last year to educational institutions of the emergency plan, which pertains to the same regional government. The SUMA program resumed the Active Schools methodology. SUMA updated the concern it had for curricular matter and it focused its attention to responding to the needs expressed by the teachers: what should the students learn? What and with what? What should the teachers learn? What should the 20 Evaluation of USAID/PERU Education Programs: AprenDes and Andean Center. Summary Evaluation Report, April 2010. Marcia Bernbaum, José Rivero Herrera, Ernesto Schiefelbein. 21 Districts known as educational decentralization laboratories. Pajarillo is located in the province of Mariscal Cáceres and Cuñumbuqui in the province of Lamas. 36 teacher coaches learn? Faced with the same questions, the Ministry of Education designed and launched the Progress Maps, Fundamental Knowledge (Aprendizajes Fundamentales, AF), and Learning Paths. At the same time, the Student Census Evaluations (evaluaciones censales de estudiantes, ECE) reported student-learning outcomes for UGELs, teachers, and parents. In response to the same concerns, and by way of implementation of MINEDU’s proposals, SUMA: a) Generated knowledge through studies and research22 that, as explained below, contributed to MINEDU’s Good Teaching Performance Framework; b) Created training programs for principals of educational institutions, teachers, and teacher coaches; c) Created curriculum tools, during the development of classroom:  Curriculum Paths in the areas of Mathematics and Communications for 1st through 6th grades (What should the students learn?);  Teacher Compasses in the areas of Communications and Mathematics (How should the students learn?) and;  The Self-Teaching Notebooks materials for 1st through 3rd grades in the areas of Communications and Mathematics (With what should the students learn?). d) Created management tools for classroom work, for the educational institution and for the district:  Annual Work Plan for Learning Improvement (PATMA);  Local Annual Work Plan for Learning Improvement (PALMA) and  Joint District Plan. e) Incorporated classroom evaluation cards and a monitoring and management system; f) Provided technical assistance through training programs and workshops for principals, teachers, trainers and coaches. Organized learning circles (CIA) for principals, teachers, trainers, and coaches, as well as for school Municipalities advisors. The study “Report Card on Reading and Writing in Peru: a Reflective Approach about the Criticism, Institutional Framework, and Implementation” provided input for the Self-Teaching Notebooks, Curriculum Paths, and school compass, as well as for training programs for teachers, teacher coaches and trainers. This tools created by SUMA is incorporated into MINEDU’s Fundamental Knowledge, Progress Maps, and Learning Paths. The officials of San Martin’s DRE interviewed explained that these 22 SUMA carried out the following studies and research: “Teacher Training and Development Policies: Report Card and Critical Issues” (Políticas de formación y desarrollo docente: Balance y temas críticos), “Better Teachers? 2006-2011 Report Card on Teaching Policies” (¿Mejores maestros? Balance de políticas docentes 2006-2011), “Systematization of the Teacher Performance Evaluation Pilot Plan” (Sistematización del Plan Piloto de Evaluación del desempeño docente), “Report Card on Reading and Writing in Peru: a Reflective Approach about the Criticism, Institutional Framework, and Implementation” (Balance sobre la lecto-escritura en el Perú: un acercamiento reflexivo sobre la crítica, el marco institucional y la implementación), “Findings and Recommendations from the National Program for Lifelong Education and Training (PRONAFCAP) in Basic Intercultural Bilingual Education” (Hallazgos y recomendaciones del Programa Nacional de Formación y Capacitación permanente (PRONAFCAP) en educación intercultural bilingüe básico). 37 tools23 facilitate the work of the teachers in the classroom by helping them implement the materials received from MINEDU. SUMA’s proposal – “day of achievement”24 – was assumed by MINEDU and has been institutionalized nationwide in the framework of the Mobilization for Learning “We Can All Learn, No One Left Behind” and the Good Start to the School Year. The learning circles (CIA) organized for facilitators reinforced the initial active school methodology, mainly in two aspects: in the development of capabilities in communications and mathematics, and in teacher coaching methodological strategies. Likewise, there were CIAs developed for monthly meetings that included all teachers, and for same grade teachers who met on a monthly basis. The methodological proposal, which focuses on learning achievements, has been assumed in the creation and implementation of PIPs in regions such as Ayacucho, Amazonas, Lima provinces, and in MINEDU’s decentralized management PIP. In conclusion, the proposal implemented by SUMA is relevant to the context and schools, since it responds to the needs and expectations of teachers as well as to the needs and expectations of students and local residents. It focuses on addressing student learning, principal and teacher performance, and community involvement. 2.1.1: What were the main issues detected related to the teaching quality of the SUMA Program and with respect to those issues to which it offered an alternative outcome? Since the low quality of Basic Education was the main focus of the issue revealed in limited student learning achievements, SUMA approached the situation from these angles: a) The disjointed management of the education system; b) The limited professional development of teachers; and c) The limited social involvement in education. In the field of educational management, it identified critical issues related to:  Disjointed planning tools and processes in sectoral agencies, among government levels (national, regional, and local), and amongst student learning-oriented educational planning tools;  High fragmentation of interventions in educational institutions, with no connection to each other and with no link to educational goals and outcomes;  Dysfunction of the sector’s agencies related to the new institutional framework and lack of capacity to provide technical assistance to schools in a result-oriented management;  Educational organizations’ weakness to focus on student learning due to the weight of administrative tasks. 23 The evaluation team was able to see in the field, photocopied versions of SUMA’ tools for validation, since they had yet to be printed in their final version. 24 The Day of Achievement (Día del Logro) is a milestone in the Learning Path, where the children orderly show, in a public act, the activities they have developed and their results. MINEDU: The Day of Achievement, methodology guide, undated. 38 Related to the professional development of teachers, the main critical issues addressed were:  Teachers’ limited initial training to respond to the needs, interests, and expectations of students of multi-grade schools; Training not designed to improve the learning achievements of students with different realities;  Limited methodological preparation and very few supporting educational tools that facilitate classroom performance;  Lack of educational management mechanisms to learn from one’s own practice or to learn from similar experiences;  Limited teacher coaching to support their performance in the classroom by those charged with providing technical assistance (specialists from UGEL and DRE);  Lack of relevant materials for multi-grade and/or single-teacher schools in rural areas, both for the teacher as well as for the students (textbooks, learning notebooks, support cards, laboratories, libraries, play centers, resources for productive projects). Related to social involvement in education, the following issues were identified:  Little parental participation in educational institutions, either due to lack of notice and/or due to family issues;  Limited participation in education by Local Government authorities, due to ignorance of their roles and functions and their potential to intervene in the development of children, adolescents, and young people;  Limited efforts by the principals of educational institutions to promote a rapprochement with parents and local authorities, and encourage them to participate in the education process;  Distancing of educational institutions from parents, whether voluntary or involuntary, due to various factors or causes (parental illiteracy, task overload, absence of one of the parents);  Difficulties to consolidate an effective participation in agencies such as the Institutional Education Council (Consejo Educativo Institucional, CONEI) due to lack of knowledge of their roles and functions. Faced with these issues, SUMA developed a comprehensive pedagogical proposal with several components:  Organized, teacher training geared toward student learning, in the areas of communication and mathematics, curriculum planning, and work strategies for multi-grade classrooms. SUMA provided technical assistance to teachers and coaches through training workshops and learning circles, and created the following programs: Coaches Training, Specialization of PELA Trainers, Teacher Training and Partial Distance Training Diploma for Leaders in Initial Literacy.  Related to curricular tools and educational materials, SUMA innovated or improved upon those developed by AprenDes. This resulted in a set of pedagogical tools or technology packages that included: Self-Teaching Notebooks (1st to 3rd grades, in communication and mathematics), Curriculum Paths for mathematics and communication for 1st to 6th grades, Teacher Compass in two areas, classroom evaluations for 1st to 6th grades, and management tools: Local Plan 39 (PALMA) and Work Plan (PATMA). Figure 2: SUMA’s response to the issues identified in the national policy framework 2009 2010 2011 2012 2013 MINEDU Priorities: Beginning of the public teaching career. Approval of the PEN 2007, with emphasis in infrastructure 2011-2016 Policies: (1) quality learning: communication, mathematics, citizenship, (2) < 5 years have access to quality educational services, (3) rural areas overcome gaps, (4) ethnic groups overcome gaps, (5) teachers are trained and perform in public career, (6) institutions strengthened￾decentralized management, (7) higher education institutions SUMA PROGRAM Regional dialogues on teacher performance (CNE). 1 st pedagogical congress. Guidelines for Good Teaching Performance Regional Dialogues 2 nd pedagogical congress. R2.1 Study on teacher training & development policies R2.a Study on teachers and services of EIB IIEEs. Teaching Policies report Card R2.1 Identification and registration of IIEE (RM 008/2012) R2.1 Good Teaching Performance framework (RM 547/2012) Compilation of national and international teacher training experiences R2.1 Technical assistance to PRONAFCAP R2.4 Assistance to CONEACES R2.2 Support to MINEDU on curricular framework R2.4 Systematization of the teacher performance evaluation pilot R2.3 Teacher coaches training program, specialization of PELA trainers. Comprehensive pedagogical proposal of active schools for rural multi-grade classrooms: Learning notebooks, Curriculum Paths, Teacher Compass, school municipality, CONEI Without seeking to delve into the country’s curriculum issue, we must mention that the debate on the National Curriculum Design (Diseño Curricular Nacional, DCN) and Regional Curriculum Designs is not yet resolved. Mainly with respect to the range of flexibility and what they can adapt to a local reality, without losing the national vision. Another major issue is the alignment between the tools provided by MINEDU for learning improvement that guide the education action in schools, such as Fundamental Knowledge (AF), Progress Maps, and Learning Paths. Different teams have developed each of these tools and therefore, they present technical-methodological alignment issues at the curriculum organizer level. They even employ different approaches inside the curricular areas and use vague terminology. By way of example, in some cases, the organizers are competences and capabilities (Fundamental Knowledge and Paths), while in others the organizers are performance domains, aspects, levels, and indicators (Progress Maps). The inconsistencies arose during the elaboration process and could have been remedied with a general technical management of the issue and a better coordination between political decision-makers and technical cadres, as well as with horizontal communication between teams and within each team. MINEDU’s proposal25 consists of moving toward a National Curriculum Framework, common to all Peruvians that could really be feasible in the classroom. The widely shared proposal consists of a curriculum focused on complex learning and skills. Having that said the adaptation of MINEDU’s tools by the SUMA team and the design of the Curriculum Paths and Teacher Compass, updated in 2013, is a contribution that facilitates the work of teachers. The 25 National Curriculum Framework. Proposal for Dialogue. Second Version. MINEDU, 2014 Local governance model for improved learning 40 results, however, are yet to be evaluated. The Learning Paths define the knowledge that each student must have gained by the end of the school year. SUMA’s Curriculum Paths present these learnings in monthly periods, which supposedly lends more clarity to the teacher on the progression of student learning development by month-to-month and grade-by-grade. The Teacher Compass created by SUMA, supports the classroom’s curricular program, with strategies suggested for each month of the year, as well as resources and time necessary. These strategies are linked to the anticipated learnings in the curriculum paths. The Self-Teaching Notebooks in Communication and Mathematics for students from 1st through 3rd grades, created by SUMA, are an improved version of AprenDes’ Notebooks. Finally, there are classroom evaluation kits, which are used by the teachers to identify the extent to which their students are achieving the skills planned for each two-month period of the school year, and thus provide feedback to their curricular practice. As presented by the SUMA team itself, what followed next was: What should the teachers and coaches learn? This question led to the teacher-training program and the coaches training program, within the framework that teacher training is a process of reflection, exchange and interaction, and pedagogical coaching is a key process for improving teacher performance. A look at the set of tools and processes developed by the SUMA Program shows that we are before an innovative and systemic educational intervention, since all of its elements are interlinked. The purpose (improving the quality of the student learning) is related to the components of teachers and coaches performance improvement, training programs, and the package of materials for use in the classroom (Notebooks, Paths, Compass) for school support (Networks and Resource Centers). In addition to this, we can mention stakeholder involvement in the school municipality and CONEI, and management support to the UGELs, as well as to local and regional governments. From the point of view of the outcomes obtained, SUMA gradually built this proposal as the intervention progressed in response to teachers needs observed in the field. Since the end of the implementation coincided with the completion of the Program, SUMA was not able to evaluate and identify the direct impact of these methodological tools on teacher and student learning. In the medium term, it will be advisable to evaluate on site (a) The use and management of the Curriculum Paths and Teacher Compass, from the observation of day-to-day classroom work and the testimony of teachers themselves; (b) The quality of these tools in achieving the learning outcomes; and (c) Whether the use and management of these tools helped teachers generate the ability to create or recreate new materials or strategies for new groups of students, which is what the Active Schools methodology seeks – not just the mechanical application of certain strategies. One of the main risks involved in having an external institution provide the aforementioned tools is the teacher dependency that could be created around those tools, a risk of falling into a mechanical approach to the detriment of learning to produce their own materials for new students each year. We infer this potential situation from the response of a teacher about SUMA’s work: “They have not yet 41 come this year. They have not sent us the cards; we are still waiting for the notebooks.” 26 This risk, which is not unique to the SUMA Program case, is a common risk when teachers become implementers of a specific or single type of methodology. 2.1.2: How coherent and functional was the proposal implemented by the SUMA Program before the issues with the quality of education, and in what aspects was it different from other ongoing initiatives? The methodological proposal applied by SUMA in two districts of the San Martin region, was coherent and functional because it responded to the problems inherent to multi-grade schools in the areas of intervention. Schools in rural areas lack direct attention with resources and methodologies adapted to the needs, interests, and expectations of students, families, and authorities. In such context, it was very well received the arrival of an outside agent such as SUMA, which facilitates resources, offers teacher and principal training, materials, and tools, and energizes participation processes (regulated by Law but seldom practiced) focused on learning achievements. SUMA’s specific feature was the combination of two basic education components. The first, the pedagogical aspect, with an improved technology package, active schools, and education management and the second, related to educational institutions and districts and the involvement of stakeholders in the Local Governance Model framework. Other educational initiatives in that same period had special and more emphasis on the pedagogical aspect (Basic Education Improvement Program, with the support of the Canadian cooperation), TAREA in boarding schools (Canchis, Cuzco), and UNICEF (intercultural bilingual education). At the national level, the Fe y Alegría experience uses the strategy of educational networks, coaches, teacher training, materials, and adds investment in infrastructure. 2.1.3: In what aspects did the proposal for quality education experience changes as a result of programmatic modifications applied to the SUMA Program? The first change – in the 2.1 Requirement deliverable – provided for amendment 06, involved a more realistic scheduling of drafting reports on teacher trainings and professional development policies. Compared to the deadlines for the deliverables established in the original contract and subsequent amendments, this modification expressed greater openness to the inherent complexity of the analytical processes in the preparation of the document. The second change – in the 2.3 Requirement– consisted of the prioritization of teacher training around reading. This change was also expressed in amendment 06. The introduction of the phonological approach to teacher training assumed the implementation of the global push that USAID had formulated for its global educational intervention, compared to the textual communicative approach assumed by MINEDU. This change in approach revealed a programmatic inconsistency with respect to the alignment that the program should have regarding the options assumed by the Peruvian Government in the curricular area of communication. The third change – in the 2.3 Requirement and corresponding indicator levels – expressed in amendment 11 was related to the redefinition of the resources involved in teacher training and 26 Response given by a teacher in San Martin, during the systematization of the active schools experience, carried out within the framework of the Systematization of Successful Experiences undertaken by the National Fund for Peruvian Education (Fondo Nacional de la Educación Peruana, FONDEP) in 2013. 42 eliminated the information and communication technologies that were part of said processes. The rural conditions in which the capabilities promoted by SUMA were implemented, did not allow for their use. Their inclusion within the original version revealed a certain level of ignorance of the reality of the place where the intervention was applied. The fourth change – in the 2.4 Requirement and respective indicators and deliverable levels – implied the elimination of an entire line of work that had been included in the initial stage: the technical assistance for the identification and development of evaluation and accreditation systems for teacher training. This radical change was expressed in amendment 06. The modification reflected the SUMA program adaptation with respect to the change shown in the sectoral agenda of a new administration. The concern around the initial training and performance evaluation of teachers ceased to be of relevance. Table 4: Changes in the SUMA Program’s outcomes structure Original Contract (September 2009) Amendment 06 (June 2012) Amendment 11 (May 2013) Achievement 02 Requirement 2.1 Strengthen teacher training and professional development There was no change There was no change Number of indicators 4 There was no change There was no change Number of deliverables 1 Redefinition of deliverable deadlines There was no change Requirement 2.2 Review best practices in teacher training and professional development There was no change There was no change Number of indicators 2 There was no change There was no change Number of deliverables 1 There was no change There was no change Requirement 2.3 Improve teacher training systems Focus training on reading IT component eliminated Number of indicators 4 There was no change IT component indicator eliminated Number of deliverables 1 Focus training on reading There was no change Requirement 2.4 Improve initial teacher training Requirement eliminated Number of indicators 3 Indicators eliminated Number of deliverables 1 Deliverable eliminated 43 2.1.4: What milestones were implemented by SUMA that were part of the strengthening process of the policy and institutional framework for teacher training and professional development, and what objective evidence could be identified as outcomes? The SUMA Program identified that the first step to strengthen the policy and institutional framework for teacher training and professional development was to generate knowledge about it. For that reason, in a coordinated effort with MINEDU, it developed a study focused on three critical areas: initial training, teacher performance evaluation, and teacher support in the classroom (pedagogical coaching).27 The results were discussed in various national and regional forums; it allowed the SUMA Program to present before the National Education Council (CNE) a synthesis of expected teaching performances, i.e., those that result in the improvement of learning achievements, thus contributing to the public discussion agenda. Based on the strategic relationship established with the CNE, the SUMA Program contributed to the definition and execution of events to reflect on the role of teachers. The events included regional dialogues, educational conferences, and systematizations, aimed at collecting experiences from teachers and students of public and private institutions regarding Good Teaching Performance. In addition, it managed a study on the situation of training and professional development policies.28 At the same time, the SUMA Program provided technical assistance to the National Program for Lifelong Education and Training (PRONAFCAP) and carried out a study on in-service teacher training, especially for teachers linked to intercultural bilingual education (EIB). It was due to the warnings of existing information gaps in EIB,, that the SUMA Program sought to characterize educational institutions with a proposal for EIB identification criteria and EIB educational services,29 from which a technical standard, applicable nationwide, was developed in 2011. Since then, the SUMA Program was invited to be part of MINEDU’s Technical Committee on EIB. SUMA offered a technical assistance plan to the Council for Evaluation, Accreditation and Certification of Quality for Non-University Higher Education (Consejo de Evaluación, Acreditación y Certificación de la Calidad de la Educación Superior No Universitaria, CONEACES) to create an ISP accreditation methodology. It was only partially implemented due to the changes in national priorities, in turn reflected in a Contract modification. At the request of the Ministry of Education, SUMA supported training on pedagogical coaching within the framework of the Strategic Learning Achievements Program (PELA), and provided technical assistance to over 23 teacher trainers and coaches.30 It participated in the design of regional in-service teacher training systems, and created a technical team to train teacher trainers in Ucayali. Teacher trainers and teacher networks were prepared based on various PIPs, the PELA, and active school 27 The text “Teacher Training and Development Policies: Report Card and Critical Issues” (Políticas de formación y desarrollo docente: balance y temas críticos) (Ugarte et al, 2011) gives an account of the outcomes. It can be reviewed in the following: http://www.sumaeducacion.pe/pdf/Doc_Ugarte_FINAL.pdf 28 The document “Better Teachers? 2010-2011 Report Card on Teaching Policies” (¿Mejores maestros? Balance de políticas docentes 2010-2011) (Cuenca, 2012) contains the main evidences of said study. It can be reviewed in: http://www.sumaeducacion.pe/pdf/mejores_maestros.pdf 29 The study is “Intercultural Bilingual Schools: Contributions to Guarantee a Quality EIB Service” (La escuela intercultural bilingüe: aportes para garantizar un servicio de EIB de calidad) (Burga et al, 2012). It can be reviewed in: http://www.sumaeducacion.pe/pdf/EIB.pdf 30Institutional Memory, MINEDU 2012. 44 programs. Within the framework of the programmatic implementation of the phonological approach, the SUMA Program conducted an applied research on the factors that affect reading. Its results were used to discuss and propose some technical recommendations on literacy strategies to MINEDU. The evaluation team notes that the SUMA Program made strenuous efforts to provide useful inputs to strengthen the policy and institutional framework related to teacher training and professional development. It generated knowledge on the subject and established strategic alliances so that the program’s contributions would reach the relevant policy makers. It participated proactively in different discussion and technical proposal spaces. Given that the purpose of the SUMA Program was to advocate in the definition and implementation of sectoral policies, the evaluation team warns that one of its main contributions was the generation of knowledge, i.e., gathering evidence and discussing said evidence in spaces close to educational policy￾makers. SUMA’s most important contribution under this heading was the support and contribution provided to the development of the Good Teaching Performance Framework. The framework was approved by MINEDU in 2012, upon the country’s need to have a guiding document on good professional performance within the National Education Project (PEN) policy. 2.1.5: What were the best practices identified and promoted by the SUMA program on teacher training and professional development, and what was the degree of institutionalization by the education sector of such practices? The Good Teaching Performance Framework (Ministry Resolution Nº 0547-2012-ED) SUMA and other institutions supported the work of the CNE’s Teacher Development Roundtable and Roundtable for Good Inter-Institutional Teacher Performance. The Roundtable proposed to contribute to the building of a national consensus on the meaning of good teaching performance. With 1,400 teachers, principals, parents, students, it opened public debate and dialogues across 21 regions. SUMA and other institutions in the Roundtable contributed directly to these processes over a two-year period to develop, in a participative manner, good teaching performance criteria and teacher training and evaluation guidelines. In December 2011, the CNE gave MINEDU the Good Teaching Performance Guiding Framework proposal. In March 2012, MINEDU called a panel of experts and reorganized the CNE’s proposal, which was finally approved. Now it guides good teaching performance toward responsibility for outcome achievement. The Framework is the first step in the process of building and implementing public policies to strengthen the teaching profession. It is strategic for the design and implementation of teacher training, evaluation, and development policies and actions at the national level. It fulfills the third strategic objective of the PEN: “Well prepared teachers professionally exercise the teaching profession,” and it’s at the service of the three policies prioritized by the Ministry of Education for 2016: quality education and closing of gaps, teacher development based on agreed good teaching criteria, and modernization and decentralization of education management. The Framework is an agreement on the competencies that the country’s teachers are expected to master at successive stages of their professional career with the purpose of achieving all students’ learning. It consists of four domains, nine skills that in turn contain forty performances. The four 45 domains are: 1) preparation for student learning, 2) teaching about students learning, 3) participation in school management in community involvement, and 4) development of professionalism and teacher identity. Teacher Coaching Best Practices and Training of Trainers SUMA developed programs related to: teacher training, teacher coaching in PIPs, initial literacy diploma courses, coaching of coaches, and the Strategic Learning Achievements Program (PELA)’s trainer specialization program, which it implemented in the regions of intervention. Based on the identification of the usefulness and results of the active schools model, SUMA designed a training program in that pedagogical model. In the training proposal, it included subjects such as literacy, mathematics, teacher coaching, school management, and multi-grade schools teaching strategies. In this context, MINEDU requested the SUMA Program’s support to design a specialization course for teacher trainers, which served as the basis for establishing the teacher training system on teacher coaching under MINEDU’s purview, which is currently in execution. In the framework of the implementation of the regional PIPs in education and PELA, the SUMA Program developed a specialized technical assistance in favor of teacher coaches. It contributed to strengthening their capacity and offered inputs related to the coaching protocol, a key tool in the training of these teacher coaches, as well as a proposal for a basic coaches’ and facilitators’ skills profile. These inputs were used by MINEDU to develop, and subsequently formalize, the technical documents related to teacher coaching. The evaluation team identified that the SUMA Program has significantly contributed to the technical definition of teacher coaching by proposing that both, methodological strategies as well as basic content training should be included in the coaches’ training. From its experience in San Martin and Ucayali, the SUMA Program was able to strengthen this training. Regarding Teacher Evaluation and Professional Development Another contribution by SUMA was the systematization of a pilot teacher evaluation that MINEDU developed in several regions around the country. With SUMA’s support, this pilot was revised and analyzed. A set of guidelines on teacher performance evaluation were prepared, and lessons learned from this experience were identified. This input represented a significant contribution toward including teacher evaluation among sectoral priorities. Unfortunately, the issue of teacher performance evaluation was postponed due to changes in political priorities. In its technical advocacy work related to teacher training, the SUMA Program provided specialized assistance to CONEACES on processes for the accreditation of Higher Education Pedagogical Institutes; however, this process did not progress due to the priority that the institutes were giving to complying with the new law on the formalization of higher education institutes and schools promoted by MINEDU. Given this situation, the SUMA Program had to redevelop its program bid. In conclusion, the evaluation team identified that the main contribution made by SUMA, with respect to teacher training and professional development, was the generation of knowledge from the study on teaching best practices which were an input to the teaching performance framework, the systematization of the teacher performance evaluation pilot, and pedagogical coaching and teacher training programs. 46 2.1.6: What challenges and lessons learned were identified in the in-service training on teaching reading best practices in IIEE multi-grade and bilinguals, and in the process of incorporating them into the education sector’s training system? SUMA focused specifically on the pedagogical treatment of monolingual areas, and did not work in training of best teaching practices in bilingual IIEE. In the region of Ucayali, where there are various Educational Institutions (IIEE) of Intercultural Bilingual Education (EIB) the SUMA Program reached a tacit agreement with the UNICEF so that it could be fully in charge of the pedagogical service to the EIB in that region. That included teacher training, supply of materials in native languages, teacher coaching, etc.). At the national level, SUMA supported MINEDU with a consultancy on the characteristics of Educational Institutions of Intercultural Bilingual Education. One of the consulting firms in this study consequently took over the management of Intercultural Bilingual Education in MINEDU and promoted the organization of an EIB Roundtable, which once again took the results of the consultancy for discussing EIB guidelines. This debate resulted in the production of criteria used to identify the IIEE of EIB. Through the DGEIBIR, MINEDU then issued Ministry Resolution Nº008-2012-MED, a “Directive that Establishes the Procedures to Identify, Recognize, and Register Educational Institutions of Intercultural Bilingual Education.” 2.1.7: What were the improvements in educational achievements (ECE – communication area) by students from educational institutions located in the areas of intervention of the SUMA Program? Since 2007, MINEDU conducts the Student Census Evaluation (ECE) through standardized tests that measure mathematics learning and reading comprehension of students in second grade, whose mother tongue is Spanish.31 This evaluation is conducted at the end of each school year and it produces individualized reports with the results for each student, classroom, school, province, and department. MINEDU’s Quality Measuring Unit (Unidad de Medición de la Calidad, UMC) presents the results from a control sample, which allows them to ensure the quality of the indicators for the purposes of comparison among sub-populations and over time. The results of the evaluation are presented in two ways: as an average score that measures the student's ability, and as level of achievement reached. In 2013, 33% of students, at the national level, achieved a satisfactory level in the area of reading comprehension, which is a breakthrough compared to 16% in 2007. In 2007, 29.8% of students were at an initial level of achievement, a rate that fell to 15.6% in 2013. National data also showed that a higher percentage of women than men reached the satisfactory reading comprehension level (35% versus 29%), that multi-grade schools have a lower level of achievement than multi-teacher schools (14% versus 37%), and that students from rural schools are at very low levels, compared with those from urban schools (10% versus 49%). In San Martín, a region that was prioritized by the SUMA Program, 27% of students reached a satisfactory level in reading comprehension, 10 percentage points more than in 2011. In addition, San 31 In Educational Institutions of Intercultural Bilingual Education, it is applied a reading comprehension evaluation in Spanish to the fourth grade students with indigenous mother tongue. 47 Martin experienced significant progress compared to other departments, and its data indicate that the gaps between urban and rural areas were reduced.32 Students that reach the satisfactory learning level are able to handle reading comprehension according to what is expected for their grade and are in condition to continue learning. In 2013, the UGELs of Mariscal Cáceres and Lamas, areas were the SUMA decentralization pilots were implemented, reached lower achievement rates than the regional average, 21% and 18% respectively. To estimate the indicators at the level of the districts of intervention, they used a database of the census evaluations conducted between 2009 and 2013, which is available at the educational institution level,. There were 7 educational institutions in Cuñumbuqui and 11 in Pajarillo. During this period, the number of students registered in second grade varied between 78 and 48 in Cuñumbuqui, and between 108 and 103 in Pajarillo. Taking the results of all the students of all the schools of each district, the percentage that achieved satisfactory levels of reading comprehension increased from 6% to 21% in Cuñumbuqui, and from 8% to 19% in Pajarillo – levels that are close to those of their respective UGELs, but still below the regional average. Taking MINEDU’s criteria that only IIEE that have at least 10 students in second grade would be considered as a control sample, only two IIEE of Cuñumbuqui and three of Pajarillo would be subject to evaluation. In these IIEE, average reading comprehension level achieved by the evaluated students increased from 452 to 532 in Cuñumbuqui, and from 457 to 508 in Pajarillo. In the entire department, the average improved from 488 to 528 (see figure). These and the previous results show that the improvements in the districts of intervention followed the trend of the region, and that it was the result of a national effort to improve learning 32 How much do our children learn in the competencies evaluated? San Martin, results of the student census evaluation, ECE 2013. MINEDU Booklet, Education Quality Measuring Unit. 48 Figure 3: Level of achievement in reading comprehension and number of students: San Martín & 2 Districts 3. Lessons Learned from the Experience 3.1. Which critical elements (both internal and external) of the SUMA Program favored the achievement of the expected outcomes, and which did not? Favorable external elements  The political will of authorities. At the central level, MINEDU and at the regional and local levels, Presidents of the regional governments of San Martin, Ucayali, and Amazonas, and district mayors of San Martin and Ucayali, placed education as a priority in their agendas.  The presence of former executives of SUMA in important positions within the Ministry of Education (Vice Minister of Institutional Management, Director of Elementary Education).  Spaces for state and civil society participation in agendas that prioritize educational policies, such as the CNE and the ANGR. For specific subjects, spaces such as the Roundtable for Good Inter￾Institutional Teacher Performance convened by the CNE, which includes participation of several stakeholders. Unfavorable external elements  The volatility of the educational policies because of changes in administration, and within each administration, because of changes in MINEDU’s management teams.  The conservative behavior of public education sectors at the central level and in the regions, which prevents the implementation of results-oriented management measures. Favorable internal elements  The SUMA Program had a competent professional team that made significant efforts to coordinate % of students with satisfactory achievement Achievement score in reading comprehension 49 priorities and tasks with high-level stakeholders in MINEDU, MEF, and the Office of the Presidency of the Council of Ministers (Presidencia del Consejo de Ministros, PCM). The team also carried out their own activities and adapted to the regulatory changes and normative contrasts that were raised during the intervention period. Unfavorable internal elements  In defining its objectives, SUMA did not conduct a robust diagnostic process that would have helped relate its objectives in a causal manner and based on evidence. It is thus that the support provided to learning improvement and the effort to improve decentralized education management were not subject to an argumentative effort that would interlink them with each other.  The SUMA team was not able to adapt to the conditions of a contract they considered rigid and which caused administrative conflicts that in turn reduced their capacity to react quickly to changes. 3.1.1: What were the political, programmatic, and strategic aspects that contributed, positively or negatively, to the implementation of the components evaluated and to the achievement of the expected outcomes? The SUMA Program had limited success anticipating political expirations (i.e., changes of governments or administrations, or changes of sectoral or regional authorities) and this situation had direct consequences during the implementation stages. During the second administration of Alan Garcia, MINEDU’s openness to international cooperation was minimal. That meant that the relationship established between that sector and the SUMA Program was markedly formal. For the evaluation team, the first two years of the SUMA Program represent a “low intensity” implementation period due to the scarce coordination that it had at the time with MINEDU’s political authorities and other important stakeholders. However, with the beginning of the administration of Humala, and the transition of some of their former consultant to executive positions in MINEDU, the SUMA Program increased its mission of higher visibility and capacity to support policies. It began working, in practice, as a technical assistance service for multiple decentralized education management initiatives. From that point forward, the evaluation team identified a “high intensity” implementation period. On the other hand, there was an important reception and favorable political will from the Regional Government of San Martin (GRSM) in several tasks —planning, budget and operational programming; implementation of the regional educational management model; preparation of an emergency plan to raise learning achievements in the region; implementation of a decentralization laboratory; development of local governance model pilots in the districts of Cuñumbuqui and Pajarillo. Likewise, in the mature stage, the Program achieved for partnerships with National Directorates or Offices of MINEDU and other sectoral institutions such as the CNE, to bore fruit, even if not precisely as originally planned. An example of this was the early emphasis placed on the activation of Participatory Councils for Regional Education (Consejos Participativos Regionales de Educación, COPARE) and Participatory Councils for Local Education (Consejos Participativos Locales de Educación, COPALE). SUMA’s final stage, which coincided with this evaluation, was characterized by a feeling of frustration by both the Program’s operators as well as by the counterparts of the San Martin region, because some of the processes that took place throughout the implementation could not be completed. Such is the case of the validation of the self-teaching notebooks, which concluded without a report and without having 50 printed the final materials. Even in those aspects where efforts have been made to ensure continuity (e.g., in the coordination with regional and local government agencies through PIPs), and a correct institutional closure can be spoken of, the stakeholders and some collaborators feel that the experience is incomplete. 3.1.2: Did the SUMA Program indicators have the level of technical implementation required for efficient monitoring? The official documentation of the SUMA Program differentiated between two types of indicators: those that gave an account of the initiative’s contribution with respect to USAID’s Education33 area, called standard indicators (indicadores estándar, IE), and those that measured the achievement of the requirements defined in the contract. The definition of the former is quantitative and general in nature; in other words, it is focused on counting the number of individuals involved in skill development processes, by it is broken down by functional categories, and it is disaggregated by gender and geographical area. These indicators do not allow us to know the content of the trainings or the Program’s involvement mechanisms, just the number and type of beneficiaries. Likewise, even though they are disaggregated by gender, they do not have a gender approach,34 because they are not used to analyze how the Program activities affect the relationships between men and women, or boys and girls, and they do not analyze how gender relations affect the achievement of the expected outcomes. Table 5: Standard indicators reported by the project Disaggregated by gender Disaggregated by region Ia Number of administrators and officials trained (includes trainers and coaches) x x Ib Number of administrators and officials trained in monitoring and evaluation x x Ic Number of teachers trained with project support x Id Number of students registered in elementary schools supported by the project x Ie Number of parent or teacher associations, or similar government structures supported by the project na x If Number of laws, policies, regulations, or guidelines drafted or modified to improve access to, or the quality of educational services na X The indicators used to measure the achievement of the contract requirements had a specific descriptive definition, in some cases confusing its inclusion when defining goals. Table 6 and table 7 show several of this type of indicators, such as: “At least four national policies and/or reforms regarding education decentralization drafted and approved.” The monitoring and evaluation component of the SUMA Program technically organized the indicators in a performance-measuring matrix, reducing their quantity and 33 The Education area is part of the Investing in People objective 34 Annex 5 explains the Program’s approach in this matter. According to a member of the team, they did not consider it necessary to apply a gender approach, since in a preliminary analysis they did not find differences between the achievements of boys and girls. 51 redefining them in terms of numbers or percentages. The analysis of the monitoring reports shows that there was no cumulative logic in the indicators. In other words, they were not defined to give an account of the continuing progress that would be achieved in each of the five outcomes of the Program. On the contrary, they had a partial approach, giving account of the activities and their direct effects. This technical detail, i.e., offering process-specific and short-term information, has implications in the management of the initiative, because it does not allow knowing to what extent the expected outcomes have been achieved. In most cases, the final values of the quantitative indicators contained in the performance matrix35 were high and even surpassed the expected 100%, mainly because the goals were adjusted as the scope of the project got smaller (fewer regions, fewer schools). This situation poses doubts as to the relevance of the goals established and as to the usefulness of the indicators as monitoring tools. Regarding the indicators of Requirement 1.1, which are not reported in a quantitative manner – diagnoses, consultancies/recommendations – they reflect processes promoted by the Program that are translated into documents, lists of stakeholders, summaries, and others. Even though SUMA has recorded in its quarterly and annual reports the specific contributions made under these indicators, and visited informants from Lima and the regions have given general references about them, the Program have not documented the richness of the processes and their contribution. Regarding the indicator of Requirement 1.2, the Program identified and documented five decentralized education management practices (international and national). Each of these studies proposes a series of conclusions, but without offering precise guidance for the Peruvian case. The evaluation team has not been able to identify any documents produced by the SUMA Program offering an assessment of all the studies and proposing their application and adaptation to the minimum conditions and requirements posed by the national context, or the rural contexts. Regarding indicator 1.3.1, no intermediate organization was restructured within the framework of SUMA’s lifecycle. However, the Program did prepare and transfer inputs that were part of the discussions on the restructuring of the education sector. When developing this indicator, the long and complex process of regional and local maturing, necessary for the restructuring of institutions, was not taken into account, whereupon SUMA identified the ways in which its technical contribution could be useful. Given the generic nature of the definition of indicator 1.3.2, the SUMA Program identified practical solutions (in the form of tools) related to education management at the regional and local levels. Once again, we can perceive its capacity to adapt in order to seek alternatives that will make outcome achievement feasible. Thus, SUMA proposed a planning strategy for addressing educational needs, as a tool for the modernization of the education management processes (PALMA), and a planning system that addressed learning improvement and which, starting with the priorities of each educational institution, extended up to the regional government level (PATMA). Regarding indicator 1.3.4, no reliable evidence was found related to the use of sources of statistical information in the planning processes by the education authorities. Given the lack of a user-friendly statistical information tool at the regional level, the SUMA Program developed a strategic information 35 The team reviewed Indicator tables from the annual reports for the second, third, and fourth year. 52 system for regional and local education management, which was never installed for the Program’s expected purposes. Regarding indicator 1.4.1, we believe that the annual increase in the education sector’s budget throughout the Program’s lifecycle cannot be directly attributed to its actions. However, inputs such as the study on costs per student could have been useful for the regional governments to negotiate budget increases with the MEF, which are processes that have not been adequately documented. Table 6: List of indicators under the contract, Requirement 1 Indicators according to Amendment 6 Indicators reported in M&E R1 Improved institutional & policy framework R1.1 Strengthen the education institutional & policy framework 1.1.1 Initial assessment of the institutional and policy context, monitored and updated annually 1.1.2 Advice provided on key policy issues to USAID, MINEDU and other relevant stakeholders 1.1.3 At least four national policies and/or reforms regarding education decentralization drafted and approved IE36: Number of laws, policies, regulations, or guidelines drafted or modified to improve access to, or the quality of educational services 1.1.4 At least three decision-makers and key officials from MINEDU participate in the institutional and policy assessment stages IE: Number of administrators and officials trained R1.2 Review and evaluate best practices for decentralized management systems and processes 1.2.1 At least three methodologies of best practices on decentralized management identified, including requirements and conditions for replication in various contexts, particularly in rural areas. 1.2.1 Government counterparts engaged in the review and analysis and demonstrate ownership of the review process, five of them from MINEDU IE: Number of administrators and officials trained R1.3 Strengthen decentralized management systems and processes 1.3.1 DREs and UGELs in priority regions restructured in accordance with the local needs and the LOPE framework Restructuring of DREs and UGELs in priority Level of progress in the regions 1.3.2 Priority regions implement plans to modernize education management processes (planning, budgeting, monitoring) in line with national policies (a) Level of progress in the development and implementation of a monitoring and evaluation system to improve management in priority regions 1.3.3 Annually, at least 250 officials or authorities from MINEDU and 5 priority regions receive technical assistance and training for improving management systems under the decentralization framework. (In 2013, the focus is San Martin and the goal is 80 officials) (IE) Number of administrators and officials trained (it includes trainers and coaches) 36 IE: Standard indicators 53 Indicators according to Amendment 6 Indicators reported in M&E 1.3.4 Education managers in priority regions use data for decision-making, in coordination with MINEDU’s Quality Measuring Unit (a) 1.3.5 At least 20 demonstration schools in San Martin maintain current student learning outcomes (b) Number of schools supported by the project through the implementation of the decentralization lab in San Martin Percentage of progress in the implementation of the decentralization lab in San Martin Percentage of students that participate in the implementation of the decentralization lab in San Martin and who achieved advanced levels in math and communication learning R1.4 Enhance budget allocation and quality of expenditures in education Education budget submitted by MINEDU to Congress incorporates policy recommendations from dialogues promoted by the project At least one territorial authority, in the priority regions, prepares Public Investment Projects (PIP) to increase investment in public education and improve efficiency in spending. Annual amount in Soles executed by public investment projects (PIP) in capacity-building activities in education, with the project’s technical support Priority regional governments (at least 1) receiving project support implement the Budgeting-for-Results initiative in education (a) Percentage of implementation of the Learning Achievements Program (PELA) budget Percentage of annual increase in the PELA budget in the project’s regions (a) 5 regions according to the original contract, the number was gradually reduced (b) 30 schools according to the original contract The qualitative indicators of requirement 2.1, merit similar comments to those of requirement 1.1, in the sense that they reflect consultation processes with various stakeholders stemming from studies and diagnoses carried out. In the case of the educational quality improvement, the SUMA Program was able to identify the relevance of including a proposal related to expected teacher performance in the CNE agenda. This input and the dialogue on policies, later translated into the framework policy on good teaching performance, explained above. Regarding indicator 2.1.2, the SUMA Program identified that the technical assistance related to quality education was establishing links with the National Program for Continuous Education and Training (PRONAFCAP), to which it provided technical assistance at the beginning, as well as to conduct a study on in-service teacher training, especially as related to intercultural bilingual education (EIB) teachers. Likewise, the recommendations mentioned were never recorded in a specific document, but rather involved an information gathering process and discussion of various teacher-training matters. Regarding indicator 2.2.1, the most important effort carried out by the SUMA Project on this respect is the study on the Teacher Performance Evaluation Pilot Plan; no other specific contributions on teaching quality best practices were found. Regarding indicator 2.3.1, the SUMA Program focused its technical assistance on pedagogical coaches, as part of the implementation of the PIPs and PELA in the priority regions. It worked with them to develop the coaching protocol, basic coach profile, diploma course in literacy (where it materialized all the 54 aspects related to the reading promotion). A more in-depth analysis is required to identify which SUMA Program inputs were integrated into the in-service teacher training systems. Regarding indicator 2.3.3, we were not able to identify information on the progress. Indicator 2.3.4 and the indicators of Requirement 2.4 – eliminated in 2012 – were no longer reported. Table 7: List of indicators under the contract, Requirement 2 R2 Teacher training and professional development strengthened R2.1 Strengthen the national policy and institutional framework for teacher training and professional development 2.1.1 Initial assessment of the institutional and policy framework related to teaching quality, monitored and updated annually 2.1.2 Advice provided on key policy issues related to teaching quality to USAID, MINEDU and other relevant stakeholders 2.1.3 At least four national or regional policies or reforms regarding teacher training, professional development, and education accreditation drafted and approved IE: Number of laws, policies, regulations, or guidelines drafted or modified to improve access to, or the quality of educational services 2.1.4 At least three key decision-makers or senior officials from MINEDU actively participate in various stages of the institutional and policy assessment process IE: Number of administrators and officials trained R2.2 Review and evaluate best practices in teacher training and professional development 2.2.1 At least three methodologies of best practices on teaching quality identified, including requirements and conditions for replication in diverse contexts, particularly the rural areas 2.2.2 Government counterparts engaged in the review and analysis and demonstrate ownership of the review process, five of them from MINEDU IE: Number of administrators and officials trained R2.3 Provide technical assistance to enhance in-service teacher training particularly related to reading in multi-grade and bilingual schools 2.3.1 MINEDU and priority regions incorporate innovative methodologies and best practices related to teaching and in-service training (a) Level of progress in the development of programs for in-service training in priority regions 2.3.2 Cadres of at least 10 teacher-trainers for new teachers trained in teaching reading in each priority region (IE) Number of administrators and officials trained (includes trainers and coaches) 2.3.3 At least one teachers’ network focused on reading established in each province where the in-service teacher training is implemented (b) 5 regions according to the original contract, the number was gradually reduced Even though the SUMA Program’s lifecycle was set to be four years, there was no formal mid-term evaluation that would serve as an analytical exercise on what was executed during the first stage of the 55 implementation, in order to make the corresponding program adjustments for the next stage.37 Such oversight represented a lost opportunity to carry out a critical review of the program’s performance and identify the decisions that should have been made to ensure a higher level of efficacy and efficiency of the initiative. 3.1.3: How did the knowledge management process driven by the SUMA Program help achieve systemic reforms and capacity-building in the education sector? The quality of the contributions contained in specialized studies, commissioned by SUMA are very diverse. In some cases (e.g., the recommendations from the studies on PELA’s limited impact), they dealt with non-realistic assumptions about causal relationships. Other studies contain a smaller dose of political and contextual analysis, and base their views in a substrate of dispersed knowledge about the desirability of decentralization, without pausing to consider what the forces that actively play against it are. According to the information available on the website of the SUMA program, they produced and published 16 documents on issues that were the subject of technical assistance during its lifecycle. These issues include the decentralization of education, teacher training and development, education management models, intercultural bilingual education, investment in education, etc. Through these publications, the SUMA Program sought to generate public debate and identify alternative solutions to the problems surrounding decentralized education management. MINEDU was not only involved in the technical discussions leading up to each publication, but also participated in the definition of the scope of studies and in the management of the production process. Likewise, other key stakeholders were involved: members of the CNE, representatives from civil society and academic institutions, members of the international cooperation staff, and others. Given the process carried out and the outcomes achieved around each study, the evaluation team believes that the knowledge generated by the Program sought to be relevant and fruitful in reforms that the education sector was experiencing. This knowledge management process was developed at three government agency levels: national, regional, and local, and should be considered at the educational institution level. In some cases, it is clear that the process of generating, sharing, and using knowledge (expertise) responds to individual and/or collective needs. An example of this is the study on the situation of educational institutions of intercultural bilingual education, which was later used as an input to generate a standard on this regard. In many cases, organically linking the knowledge generated with the instances involved – such as MINEDU, CNE, DRE, UGEL and the educational institution – so as to know how to “manage” these organizational knowledge and learnings and use them to improve their operation, is still pending. In other words, identifying best practices – one’s own or others’ – through research is not enough; they must feed back into the design and implementation of strategies for action, to ensure positive and relevant results. 37 According to the perception of the SUMA team, the systematization of the initial experience of the Local Governance Model is assumed as an evaluation landmark, since lessons learned and best practices were extracted from its main programmatic proposal. From a technical point of view, such systematization cannot be assumed as an evaluation since it was focused on reconstructing the design and implementation process of one of the decentralized education management tools promoted by the SUMA Program. 56 3.1.4: Which decisions made and implemented by the SUMA Program should not be replicated by other similar interventions, and why? From the beginning, the failure to lay out the link between learning improvement and decentralized management components constitutes a conceptual deficit that must not be repeated. Inclusive, the program even lacked a problem tree and a logical framework. On the other hand, prolonged interventions such as SUMA’s, which are implemented over the course of numerous institutional and administrative counterpart changes, must include a degree of flexibility in their reorienting mechanisms. It is highly likely that this condition – changes in context – will exacerbate in the future. It would be a mistake to think that the interventions can be defined and managed at the slow-paced mode, typical of one or two decades ago. 3.2 What practical knowledge can be extracted from the experience carried out by the Program? Regarding pedagogical coaching A contribution from SUMA’s experience to the teacher coaching programs is that providing sporadic training to coaches on methodological strategies for teacher support is not enough. It is necessary to train them in subjects related to communication, mathematics, science and environment, and civics, within the framework of a program of at least two years of duration with sequential workshops throughout the year, learning circles, internships, and providing monitoring tools. Regarding in-classroom learning and tool alignment In addition to the learning achievement of students during the school year – indicated in MINEDU’s Learning Paths – teachers must know what they can achieve during each two-month period and each month. SUMA’s contribution to fill this void was to design the Curriculum Paths and the Teacher Compass, which suggest various strategies to apply during learning sessions, thus facilitating the teacher’s work in the classroom. In order to avoid confusion given the variety of instruments that teachers have, and especially to develop their ability to understand and organize the processes, the next step is for each Region – DREs and their respective UGELs – to generate a single organizing document, aligning the various management tools to adapt them to the quality-improvement and results-achievement goals. Key parental involvement To achieve adequate cognitive development, the role of parents is extremely important in laying the foundation for later development and awakening the desire to read and continue learning. The role of teachers involves organized teaching knowledge, and the role of families includes life, ethics, and values guidance, but both require integrated work. SUMA opens the school and the teacher’s gaze towards the community, and parents participate, by developing their role as educators, both at home and at school. 3.2.2: What findings can be extracted from the Decentralization Laboratory’s experience in San Martin? The Decentralization Laboratory in San Martin has had a positive impact on very different scales. Even though its contributions to increasing learning achievements in communication and mathematics cannot be clearly demonstrated, it has undeniable qualitative achievements in school climate and the integrated management of public services at the local level. It should be noted that the efforts made in San Martin take place against a backdrop of economic 57 momentum exerted in technical and financial aspects by both the regional administration and cooperating sources alike. It is technically difficult to attribute causality or quantify each contribution. We should point out that the region had a Regional Education Management Model, approved through Regional Ordinance No. 011-2010 GRSM/CR, which constitutes the framework for the SUMA experience. The GR of San Martín and SUMA decided to intervene in two pilot districts or decentralization laboratories, Pajarilllo and Cuñumbuqui, as space for integrating the methodological proposal of active schools within a local governance model, oriented toward learning improvement. SUMA’s pedagogical proposal incorporates a school management support, not only at the educational institutions (via parental involvement in CONEIs) and other schools (educational networks), but also for UGEL specialists and officials and for the district municipality’s authority, who along with representatives of other sectors, such as health and agriculture, constitute the local management committee for social development (CGLDS). The leadership role rests with the local authority (district mayor) and/or the sector authority (the UGEL director). Participation is multi-sectoral (health, education, agriculture). They agree on learning goals for the school year expressed in the local work plan (PALMA) and the responsibilities of each stakeholder. For example, the health sector contributes to the student learning achievement by promoting favorable conditions: de-worming campaigns, nutrition, hygiene, and cleaning of home, school, and community. The leading entity exercises supervision over processes and outcomes. They have permanent assemblies and have timeframes for evaluating results. UGEL specialists provide technical assistance, network coordinators meet, coaches provide support to principals and teachers; parents collaborate in the classroom; and principals and teachers report about the strategies to be used to attain the agreed-upon learning goals. There is a social dynamic of encouragement and responsibilities where accountability shows the outcomes achieved. This local governance model proposal is undergoing validation. According to the systematization report by Fanny Muñoz and Sandra Carrillo (2012), the experience in the two decentralization laboratory districts shows that “processes and outcomes are heterogeneous due to the social and municipal dynamics, and consistent with the level of involvement and leadership skills of each local government.”38 The weakness observed in the implementation of the local governance model – according to the authors of the systematization – is not being able to develop a plan for skills development that will allow identifying the gaps in knowledge, abilities, and skills of each stakeholder involved. This model profile requires stakeholders with the ability to work cooperatively and in teams, and with specific responsibilities and tasks. It is not easy to have adequate staff and there is a need for a budget to be able to provide continuity. In addition, there are issues with making the intergovernmental and inter-sectoral coordination effective at the base of the model. They mentioned inter-sectoral learning is very slow, and that administrative staff is often too conservative to transition to a results-based management approach with accountability. 38 Muñoz, F. y Carrillo, S. “Systematization of the implementation experience of the Local Education Governance strategy in San Martin. Lessons learned and replicable best practices” (Sistematización de la experiencia de la implementación de la estrategia de Gobernanza Local de la Educación en San Martin. Lecciones aprendidas y buenas prácticas replicables). Educational Forum, Lima, November 2012. 58 A critical knot identified is the problem defining the roles of the institutional stakeholders involved. For example, the technical/regulatory dependency shown by the UGELs with respect to the DRE, even though it is a decentralized execution instance of the regional government. Local governments perceive a lack of communication channels with the regional government, and do not participate in the planning process of the PER. The regional government, which has a leading role in regional policies, and the DRE, which is a specialized entity of the regional government, do not have an explicit role in the organization and implementation of the local governance proposal. The UGEL is the only entity present in the governance strategy – representing the regional level – but it is not clear about its role in the coordination with all other regional government instances Analyzing the level of progress in the two districts, the key aspect to highlight when replicating the local governance model, is the need to consider the historical, cultural, social, and political conditions of each context to better define the processes and skills to develop in each case. San Martin is the first region with a Regional Education Management model that incorporates the local level with intergovernmental and inter-sectoral coordination, and which is under implementation and validation. Consistent with the regional model approved by Resolution No. 3025-2011-GRSM/DRE, the Regional Government of San Martin refers to the local governance model as a Local Management model. San Martin is replicating the Local Management model for education in 11 additional districts, in addition to Pajarillo and Cuñumbuqui. Six (in bold) of these districts were listed in the resolution that formalized the implementation of the Regional Education Management model in 26 districts: Rioja, Awajún, Yuracyacu, Yantaló, Soritor, Calzada, Habana, Cuñumbuqui, Zapatero, Chazuta, Pucacaca, Cacatachi, Sauce, San Antonio, San José de Sisa, Buenos Aires, Pilluana, Tres Unidos, San Rafael, San Pablo, Sacanche, Piscoyaque, Pajarillo, Huicungo, Nuevo Progreso, and Shunté. According to the information provided by the DRE planner, “The districts were selected by each UGEL; they started from the idea that each UGEL had to implement the management model in one district. They did it based on the relationship experience they built with the mayors over the years, for example, when attempting to build or improve classrooms, or obtain furnishings, teacher trainings, and teacher or para-educator pay, and depending on whom they had the best results with in this effort, it is with them that they built this relationship. This entire process was handled with complete autonomy by the DRE and its UGELs”. The 13 districts shown in Table 8 have proceeded to conform and install their CGLDS, whose purpose is to implement inter-sectoral management (health, education, social development). The 13 local governments are committed and had agreed upon a budget of 1.7 million for 2014. According to the information provided by the DRE, and the CGLDS, the education networks and CONEI have redesigned their structures and aligned their plans and budgets to attain the education outcome goals agreed upon for 2014. 59 Table 8: Progress in the implementation of local education management Province District PATMA of IIEE Annual Network Plan Coordinated Plans CGLDS Rioja Nueva Cajamarca 77 4 1 Moyobamba Yantaló 7 1 1 Lamas Barranquita Cuñumbuqui 37 16 1 2 1 1 El Dorado Agua Blanca 9 1 1 San Martín Cacatachi Alberto Leveau 4 4 1 1 1 1 Picota Buenos Aires 26 1 1 Bellavista Bajo Biavo 52 2 1 Huallaga Alto Saposoa 12 1 1 Mariscal Cáceres Pachiza Pajarillo 27 24 1 2 1 1 Tocache Nuevo Progreso 43 3 1 13 districts 338 21 13 Source: DRE-GRSM, Planning Office, May 2014. The pedagogical part of SUMA’s proposal had greater precision at the design and implementation levels —pedagogical coaching, training programs for teachers and coaches, new tools— compared to the school and inter-sectoral management part which needs further development to make governance truly effective —networking, accountability, inter-sectoral management, participation of stakeholders from different sectors, compliance with institutional agreements and commitments—. Given its experimental nature, since it is part of a “laboratory,” it did not have an incremental design; it was not thought out with a process logic mind where validation, evaluation, readjustment, and transfer are important for sustainability. Instead, a project logic prevailed, giving an account of the execution of activities and attainment of products by component, even though the aspects of the proposal involved educational processes of empowerment of the individual and institutional stakeholders. A lesson learned from the experience of the local governance model in education in the two districts is that the transition from a sectoral management model to an inter-sectoral model is not achieved just with training. It rather involves changes in the organizational culture of the institutions and a new form of public management to which the state administration is not accustomed. Therefore, it is necessary to have:  An incremental design by stages and processes, with implementation/validation/evaluation and transfer deadlines;  A monitoring system for the various institutional stakeholder levels;  A real articulation and coordination among the various levels of government, decentralized education management instances, and civil society organizations that assume responsibility and 60 establish relationships with each other. 3.3 How to capitalize on what has been learned in the SUMA Program within the design of new interventions aimed at improving the management and quality of public services in the Amazon? 3.3.1: What type of political, programmatic, strategic, or operational considerations must be taken into account when developing and implementing similar interventions? Very interesting precautions have been taken in order to ensure the political continuity of the strategies implemented, but it is essential to hone the ability to intervene in a more orderly fashion before changing policy, due to differences in approach between successive administrations at the highest levels of the Executive branch. A similar program should increase its political analysis skills and its ability to identify the interests of the main stakeholders involved. For example, to understand why the MEF or MINEDU lacked the political will to release control over some crucial tasks that they should have legally dropped, or to adopt certain critical rules. The design of any new initiative must be based on a comprehensive and updated assessment of the sector(s) they seek to make a difference in. Finally, the importance of assuming each of the stages of an intervention cycle must not be ignored. Omitting or skipping parts of the closing process leads to a weakening of the product. 3.3.2: What development scenarios could be reconstructed from the Program’s analysis? As with many interventions, the formulation of the SUMA Program was not based on a logical framework. Rather, it assumed a hypothesis of development as articulating foundation for its work and from it, defined the expected outcomes as well as the operation strategy. In this sense, the development hypothesis establishes a direct relationship between its components — decentralized management and teaching quality—, expected outcomes, and operation strategy. From the analysis of the original contract as well as of its subsequent amendments, we can state that the SUMA Program conjugated five development hypotheses:  The strengthening of decentralized and participatory management is possible if there is information about the current status of educational decentralization policies and institutional framework, as well as about decentralized management practices, and if the educational management systems and processes, budget allocation, and quality spending in education are improved.  Improvement in teaching quality is possible if teacher training and professional development have clear policies applicable both at the national and regional levels, if it integrates and takes ownership of lessons learned from international experiences (teacher coaching, curriculum support, and teaching standards), and if it prioritizes in-service training focusing on reading.  The institutional implementation of best practices is possible if new education management methodologies and inter-institutional alliances are established. 61  The learning of educational institutions can improve if their teachers are trained in the active schools methodology, and if parental and community involvement is permanent.  Political dialogue, communication, and civil society participation around basic education can improve if the existing education monitoring mechanisms and communication strategy encourage public debate. Each of these conditions gave rise to the definition of a requirement (intermediate result). The first two conditions served as basis for the first two requirements related to education decentralization policies and pedagogical models and strategies. The remaining three conditions gave rise to the formulation of the following three intermediate results, which were crosscutting in nature. The critical analysis of the development hypotheses reconstructed above allows us to determine that they are based on two assumptions: on one hand, the sufficiency, and on the other, the linearity of the components that make them up. These two assumptions are not always right. The first assumption implies that the achievement of the expected outcomes is possible only from the intervention on those variables listed, without considering that there may be different operating alternatives. The second assumption is that the intervention on the variables directly affects the achievement of the outcomes, without admitting that there may be other connections (unexpected results). The Program has been conducted under the assumption that there are elements linked to good educational management (e.g., in the areas of planning, organization, selection and training of staff, leadership, and supervision, etc.), especially at the school autonomy level. They have direct impact on the quality of teaching that can be practice in the classroom and, therefore, on the quality of the learning achieved by the students. Based on this, it has transferred this conviction to the need to support the education decentralization process that the country was already going through, in order to ensure what in essence was its ultimate purpose: improve learning. 62 CONCLUSIONS Regarding the reform processes prioritized and supported, the SUMA Program constituted a knowledge management mechanism, an advocacy platform, and in the local area of San Martin, a space for empirical verification in support of the educational reform process in Peru. In this sense, the SUMA Program did not promote any specific changes in the education sector in the framework of the educational decentralization policy and institutional framework of recent years. Its purpose was to support ongoing reform processes aimed at improving the quality of basic education, without installing new priorities in the sector’s agenda. Its nature was to be functional with respect to the demands of its main counterpart and strategic partners. The original development of the SUMA Program gathered, in a general manner, the sectoral reform trends that existed towards the end of the previous decade and defined, based on the proposal of specific accomplishments related to these trends, some axes of work to promote them. Even though with each of its amendments it sought to have better alignment with education modifications, the programmatic modifications established were not always relevant because the sectoral priorities around these policies varied. Beyond the formal fulfillment of what the SUMA Program had set out to do, the sectoral reforms it supported are yet to be crystallized. While there can be different explanations about the reason for such unfinished processes, a question emerges regarding the relevance of the SUMA Program design: were the critical factors (drivers) that define the decentralized management, training, and professional development policies taken into account during the initiative’s formulation process? This situation reveals a level of deficit in the design stage.39 From the pedagogical point of view, SUMA was able to position a proposal with strategies for improving education quality based on self-teaching, on the starring role of students, and on parental involvement in education management, in the center of the educational environment. The proposal contains a set of pedagogical and management tools aimed at improving student-learning outcomes. This proposal is pertinent in the context in which it is applied. At the district level, the proposal was the center of the Local Governance Model. In the specific case of the post-SUMA scenario, it will be interesting to know how the institutional framework promoted by the program in the 11 San Martin districts that are in the process of adopting the model develops. The SUMA project has paid the consequences of an extended learning curve with regard to the political context in which it was forced to operate, an environment that included non-institutional or non-explicit variables that have affected national and regional decentralization processes. This evaluation estimates that some of this variability could have been anticipated, had there been a better operational base to act accordingly. On the other hand, the team recognizes that the link between the Program components has not been worked as an input to develop an internal logic. It is possible that the SUMA Program would have had 39 No evidence could be found in the documentary analysis and information gathered from key informants as to whether the SUMA Program’s design process included, at least, a stakeholders’ analysis. Through such an exercise, it would have been possible to identify the interests, legitimacies, and relative powers of multiple stakeholders linked to decentralized management and teacher training and professional development policies. 63 better possibilities in its two main missions, had it been able to dedicate less time to debating about the approach applied to reading, and more time and effort to the analysis of which decentralization policies would be feasible and which ones would not. We reaffirm the importance of closing the processes that are open with cooperation, as well as of helping define them in a virtuous network. In one word, sustainability has nothing to do with whether the beneficiaries know how to apply recipes, nor whether they actually apply the recipes; has to do with ensuring that they have the skills to generate their own solutions when changing circumstances demand it. 64 RECOMMENDATIONS 1. For future interventions, we recommend that –within the framework of a scenario and stakeholders analysis– what is going to be done, what is not going to be done, and how compliance with the assumptions and their effects on the program’s implementation will be monitored, be better anticipated and defined. In risk management logic, we must seek to minimize the negative impact of adverse events present in projects with a strong policy component. 2. In order to improve the monitoring of future programs, we recommend specifying the expected outcomes in a hierarchical and coordinated model and establishing indicators that will respond to final and intermediate results. Indicators that measure activities (such as the number of workshop participants) are important but not enough to measure expected changes. 3. In programs that seek changes that do not take place in the short term (such as learning improvement), the core of the intervention should not be concentrated so close to the program closing. In future interventions, it will be important to carefully ensure closure of micro-cycles in each phase of the processes. 4. We recommend including a gender approach in the design of educational interventions such as SUMA, since the student census evaluation continues to show differences in learning between boys and girls. 5. In educational programs such as SUMA, which are carried out in rural environments, we recommend designing strategies for improving learning in the group of children under level 1 (at the beginning), based on an analysis of the causes for poor performance. 6. We recommend linking the educational interventions directed at the elementary level with the initial level, because it has been proven that if early stimulation and the social and emotional development of children in early childhood are improved, reading and writing conditions also improve. 7. We recommend “managing” the knowledge generated from studies and research to lay down or consolidate the foundations of educational policies and measures. Installing sustainability mechanisms involves ensuring a correct and effective knowledge management. 8. We recommend supporting MINEDU in the consolidation of a national observatory of best pedagogical practices, including the SUMA experience, that will provide teachers and the community in general access to such practices. 9. We recommend a systematization of SUMA’s pedagogical experience –processes, strengths, and obstacles– from the facilitators themselves, to draw lessons from learning improvement experiences and from coaches or facilitators actions, in order to generate knowledge about the motivation processes for change, which is a defining factor to promote educational innovation. 10. We recommend specifically on site evaluation of (a) the use and handling of the Curriculum Paths and Teacher Compass, according to the testimony of the teachers themselves, and the observation of daily classroom work; (b) the quality of these tool in learning achievement; and (c) whether the use and handling of these tools contributed to generate the ability to create or recreate new materials or strategies for new groups of students in teachers. 11. We recommend monitoring the Local Education Management model validation process in the 13 implementation districts of San Martin, in order to draw conclusions on territorial management, methodological tools, and student learning improvement and to verify the validity of the 65 comprehensive pedagogical proposal, the quality and use of the educational materials, and their adaptation in public, multi-grade, and multi-teacher educational institutions, in various territorial environments. 12. We recommend that the region give continuity to the SUMA proposal in those schools that had the support, keeping the coaching teams, materials, learning resource centers, principals and teachers, so as to be able to validate the aspects of the pedagogical proposal that still require more time to be consolidated. 13. We recommend that the region include the Local Management model in the organizational chart of the Regional Education Management model of San Martin, for greater connection and to contribute to the definition of roles and tasks of stakeholders at every level. 14. We recommend enhancing the Local Education Management model in the districts where it is to be implemented, creating a link with the most dynamic economic stakeholders of the productive sectors (e.g., those from the dairy and coffee basins). 66 BIBLIOGRAPHY Bernbaum, M., Rivero Herrea, J., & Schiefelbein, E. (2010). Evaluation of USAID/PERU´s Education Program: AprenDes and CETT- Andino. USAID PERU, Washington,D.C. Bernbaum, M., Rivero Herrera, J., & Schiefelbein, E. (2010). Evaluación de los Programas en Educación de USAID PERÚ: AprenDes y Centro Andino. USAID, Washintong,D.C. Chirinos, L. S. (2011). Estado de las politicas de descentralización educativa 2010-2011. USAID|PERU|SUMA, Lima. Consejo Nacional de Educación. (2009). Boletín CNE Opina. Lima, Perú. Consejo Nacional de Educación. (2012). Balance de descentralización Educativa. Lima. Consejo Nacional de Educación Nacional. (2013). Proyecto Educativo Nacional - Balance y Recomendaciones. Lima. Defensoría del Pueblo. (2013). Serie Informes de Adjuntía- Informe N° 017-2013-DP/AAE " Una mirada a la escuela rural. Supervisión a Instituciones Educativas Públicas de Nivel Primaria". Lima. Dirección Regional de Educación de San Martín. (2007). Proyecto Educativo Regional Región San Martín 2005 -2021. Moyobamba, San Martín , Perú. Dirección Regional de Educación de San Martín. (2010). Reglamento de organización y funciones de las instancias de gestión educativa descentralizada de la Dirección Regional de Educación San Martín. Moyobamba, San Martín , Perú. Dirección Regional de Educación San Martín. (2012). PER San Martín Modelo de Gestión Educativa Regional. Moyobamba. DRE San Martín. (2012). Reporte SSII-PER 2012. Moyobamba. Gobierno Regional Amazonas. (11 de febrero de 2008). Ficha de registro del PIP " Mejoramiento de la calidad edcuativa en la zona rural de 12 distritos de la Región Amazonas". Amazonas, Perú. Gobierno Regional Ayacucho. (08 de abril de 2010). Ficha de Registro del PIP "Fortalecimiento de capacidades pedagógicas y de gestión en el sistema educativo de la region Ayacucho". Ayacucho, Perú. Gobierno Regional de Lima. (27 de julio de 2010). Ficha de registro del PIP " Desarrollo de las capacidades de los actores educativos de la EBR de 13 distritos de la región Lima. Lima, Perú. Gobierno Regional de San Martín . (23 de setiembre de 2013). Resolución Directoral Regional N° 2960- 2013-GRSM/DRESM. Moyobamba, San Martín, Perú. Gobierno Regional de San Martín. (03 de Marzo de 2013). Ordenanza Regional N° 002-2012-GRSM/CR. Moyobamba, San Martín, Perú. Gobierno Regional de San Martín. (2014). Plan Opertaivo Institucional . Moyobamba. Gobierno Regional de San Martín. (27 de enero de 2014). Resolución Directoral Regional N° 118-2014- GRSM/DRESM. Moyobamba, San Martín , Perú. 67 Gobierno Regional de Ucayali. (01 de Agosto de 2008). Ficha de registro del PIP " Fortalecimiento de capacidades de los procesos de enseñanza y aprendizaje en las instituciones educativas de la zona rural y bilingüe de las provicnias de Coronel Portillo, Padre Abad y Atalaya de la región Ucayali". Ucayali, Perú. Gobierno Regional Junin. (18 de octubre de 2010). Ficha de registro del PIP "Implementación a nivel piloto de la propuesta de escuelas activas y emprendedoras para mejorar la calidad de los servicios educativos en las Instituciones Edcuativas del nivel primaria de la EBR". Junin, Perú. Gobierno Regional San Martin. (04 de julio de 2007). Ficha de registro del PIP" Fortalecimiento Institucional y Docente para el Desarrollo de Capacidades en Instituciones Educativas Focalizadas en la Región San Martín". Moyobamba, San Martín, Perú. Gobierno Regional San Martín. (28 de diciembre de 2010). Ficha de registro del PIP " Implementación piloto del nuevo diseño curricular y del modelos de gestión educativa regional para mejorar la calidad de los servicios educativos en en nivel inicial y primaria de EBR de las II.EE de 6 distritos". San Martín, Perú. Gobierno Regional San Martín. (22 de octubre de 2010). Ficha de registro del PIP "Creación del sistema regional de desarrollo y formación docente continua para el perfeccionamiento de las competencias profesionales del profesorado de la región San Martín". San Martín, Perú. Gobierno Regional San Martín. (31 de mayo de 2010). Ordenanza Regional N° 011-2010-GRSM/CR. Moyobamba, San Martín, Perú. Gobierno Regional San Martín. (04 de Marzo de 2013). Ordenanza regional N° 003-2013-GRSM/CR. Moyobamba, San Martín, Perú. Instituto Nacional de Estadística e Informática - INEI. (2010). Resultados de la Encuesta a Instituciones Educativa de Nivel Inicial y Primaria - ENEDU . Lima. Linares, A. M. (2011). Sistematización de modelos de gestión educativa de los gobiernos regionales de San Martín, Arequipa y La Libertad. Consejo Nacional de Educación & USAID|PERU|SUMA, Lima. Ministerio de Educación. (12 de marzo de 2013). Ficha de registro del PIP " Mejoramiento de la gestión educativa descentralizada de II.EE en ámbito rural de 24 regiones del Perú. Perú. Ministerio de Educación. (2014). Marco Curricular Nacional Propuesta para el Diálogo - segunda versión. Lima. Ministerio de Educación, USAID PERU, AprenDes. (2009). Sistematización de la experiencia del Proyecto AprenDes. Lima. Municipalidad de Huanca Sancos. (19 de junio de 2013). Ficha de registro del PIP " Mejoramiento de los logros de aprendizaje en los estudiantes de la EBR a través del enfoque de las escuelas activas en la provincia de Huanca Sancos - Ayacucho. Ayacucho, Perú . Oficina de Coordinación Regional. (2013). Estudio Definitivo Parte I del Proyecto de Inversión Pública: " Mejoramiento de la Gestión Educativa de II.EE. en ámbitos rurales de 24 Regiones del Perú:GED RURAL. Ministerio de Educación, Lima. Torres, K. O. (05 de Diciembre de 2012). Modelo de Gestión Educativa Regional. Perú. 68 Ugarte, M., Arguedas, C., & Ángeles, N. (2012). Balance de la Descentralización de la Educación 2009 - 2012. USAID|PERU|SUMA, Lima. USAID . (April de 2012). USAID Education Strategy Technical Notes. USAID. (2005). Education Strategy - Improving lives through learning. USAID. (February de 2012). USAID Education Strategy Implementation Guidance. USAID PERU. (April de 2012). USAID Education Strategy Reference Materials. USAID|PERÚ|SUMA. (s.f.). Caja de Herramientas. Marco conceptual del modelo de gestión educativa local: La Gobernanza Local. San Martín, Perú. 69 Annex 1: Statement of Work Program to be evaluated The intervention to be evaluated is the Quality Basic Education Reform Support Program (named SUMA Program), implemented by Family Health International (FHI) as contractor. This initiative represented the main instrument for implementing the USAID/Peru Education Program Area for the fiscal year 2009 – 2013, which objective was to improve basic education quality in disadvantaged areas. The SUMA Program purpose was to provide expert technical services to the Government of Peru (GOP) at the national and sub-national levels to support initiatives that improve basic education quality in disadvantage areas of the country (Ayacucho, Junin, Lima Region, San Martin and Ucayali). The specific contribution of this Program was to support systemic reforms and capacity building processes in basic education (primary education) for improving student learning outcomes. The logic of the intervention can be conceptualized as follows: Output Outcome Impact Expert technical services provided. Systemic reforms and capacity building processes supported. Student learning outcomes improved. The SUMA Program concreted the approach shift that USAID/Peru assumed the last years: from a direct delivery of services to supporting systems reforms for the Government of Peru (GOP) for improving the quality education services under a decentralized framework. Background The Suma Program assisted Peru at national and sub-national levels. At the national level, the efforts were focused in strengthening the country´s overall education policy, institutional framework, and capacity. At this level of work, the main counterpart was the Ministry of Education. At the regional level, the efforts were focused in at least five regions, selected using the incidence of poverty (highest in the highlands and the Amazon jungle) and the operation of USAID education programs as criteria: San Martín, Ucayali, Amazonas, Lima Region, Junín and Ayacucho. Although the ultimate beneficiaries of the SUMA Program were the students of Peru´s public basic education system in disadvantage area, with emphasis on the most excluded, such as girls, persons with disability, and students from rural areas, the intermediate beneficiaries were officials, technocrats, teachers, parents, community leaders, academics, think tanks, and civil society at national and sub￾national levels. The relation between the two specific SUMA Program beneficiaries is conceptualized as follows: 70 The results´ structure of SUMA Program is as follows: Program Result 01: Participatory and Decentralized Education Management Strengthened. Outcome 01: Decentralization Policy and Institutional Framework Improved. Program Result 02: Teaching Quality Improved. Outcome 02: Teacher Training & Professional Development Strengthened. Cross-Cutting Outcomes: Outcome 03: Implementation of Best Practices Supported. Outcome 04: Active-School Methodology in Alternative Development Schools Implemented. Outcome 05: Policy Dialogue, Communications, and Civil Society Participation Enhanced. The development hypothesis of this Program consisted in five conditions: • If the decentralization policy and institutional framework of the education sector were improved in terms of: a) having a real diagnosis of its current situation, b) applying decentralized management systems and processes identified from international experiences and c) enhancing budget allocation and quality of expenditures for education, the participatory and decentralized education management would be strengthened. • If the teacher training and professional development was strengthened in terms of: a) establishing basic policies or regulation at national or sub-national levels, b) putting in action key lessons learned from international and local experiences (in-classroom teacher support, curriculum support and teaching standards) and c) enhancing in-service teacher training related to reading (multi-grade and bilingual school settings), the teaching quality would be improved. • If innovative methodologies and best practices on educational management were applied and if a variety of inter institutional alliances and public investment procedures were established, the implementation of best practices would be supported institutionally. • If teachers that work in schools located in alternative development areas were trained in active￾school methodology and the involvement of communities and parents was permanent, the alternative development schools would increase achievement scores. • If educational oversight mechanisms (CNE, COPARE and COPALE, for example) were supported for analyzing and advocating for the quality of education, and a communication strategy was technical assistance and training of... intermediate beneficiaries for... improving education services in favour of... ultimate beneficiaries (students) in disadvantaged areas 71 develop for discussing teaching quality, reading skills, educational policy and finance and decentralization, policy dialogue, communications and civil society participation on quality of basic education would be enhanced. All these five conditions shall contribute to support systemic reforms and capacity building in basic￾education, specifically primary education, to improve learning outcomes in disadvantages areas. Approach and Implementation The SUMA Program was focused on supporting educational systems reforms for the Government of Peru. It meant that invested resources assisting specific initiatives. During its five years of implementation, the Program was redefined and adjusted several times. The last one corresponds to the Amendment/Modification Nº 11 (submitted on May, 2013). The implementation of the Program was in charge of the key personnel that the contractor assumed essential to the work being performed (Chief of Party, Deputy Chief of party / Regional Coordinator, Director of Educational Finance and Decentralization, and Director for Teacher Professional Development). A team of external consultants was hired for implementing the actions. The SUMA Program focused on accomplishing two Program Results: 1) Participatory and Decentralized Education Management Strengthened and 2) Teaching Quality Improved. The first Program Result implied the following outcomes: a) a strengthening process of the institutional and policy environment (laws, regulations, management systems, standards, etc.) for enhancing education management within the context of decentralization and b) a capacity building process on management and technical issues of regional and local governments for carrying quality education services. The second Program Result involved a specific outcome: a set of efforts for improving the policy and institutional framework for delivering quality teaching, including the improvement of pre – and in-service teacher training applying proven strategies and tools from national and international experiences. The SUMA Program had some cross-cutting outcomes related to enhancing police dialogue, communication and outreach, creation of public-private partnerships, support for implementing best practices in decentralization and teacher training, and providing school-level assistance to selected schools in Alternative Development communities. Through all these outcomes the civil society and citizen will be involved directly. The Program defined for each of the five Outcomes some Requirements. Each requirement meant a sub-process defined for getting the Outcomes. Purpose and Used of the Evaluation The evaluation of SUMA Program will be a performance evaluation. It means that some descriptive and normative question will lead the revision of its level of relevance, effectiveness and efficiency. This performance evaluation has a double purpose: to analyze what extent the expected results were achieved and identify key factors, lessons learnt and recommendations derived from the programmatic initiative. This type of evaluation represents an explorative exercise on objective evidences about what the SUMA Program has achieved after its five years of implementation. The attention will not only be focused on the examination of how the values of the indicators have changed throughout the years. It will include an assessment on how it was being implemented, how it was perceived and valued by the key counterparts, 72 etc. Selected outcomes and their respective requirements will be assessed for measuring their contribution for getting the expected Program Results. The analysis of the accumulated experience of the SUMA Program will be a useful input for improving the USAID/Peru knowledge and accountability on one specific intervention. The evaluation results will increase the level of relevance, effectiveness and efficiency of new initiatives that will be designed and implemented and will realize on what works and doesn´t work. Specific objectives of the evaluation are:  Critically analyze the level of achievement of the expected results related to policy and institutional framework of educational decentralization and quality.  Identify success factors, lessons learned and key recommendations out from the evaluated components of the Program that may be applicable in the development of new initiatives or in general knowledge management, within the new cooperation strategy of USAID (2012 - 2016). The USAID/Peru staff will be the main user of the evaluation results. They will utilize the main findings in the design, management and measurement of new initiatives in the framework of its new cooperation policy (2012 – 2016). As all evaluation process, this will be an opportunity for institutional leaning. Other users will include policy and technical staff from the Education sector (central and sub-national levels), who will assess the add value and the potential scaling up of the education management strategies promoted by the Program in the decentralization process. Staff from national and sub-national academic institutions, think tanks, donors, national and international nongovernmental organizations, etc. will be included in the audience, and they will use the evaluation findings as key references for their institutional work. Evaluation Questions The performance evaluation is based in four evaluation questions, linked to some specific outcomes of the Program. The first two looks for an assessment on what extent the Program Results were reached. The last two focus the analysis on the factors that influenced (positive or negatively) their achievement, the lessons learnt derived from the experience and the applicability of them. The main evaluation questions are the following: 1. To what extent has the SUMA Program promoted specific changes in the education decentralization policy and institutional framework (O 01)? 2. To want extent has the SUMA Program promoted strategies and models in education quality that addressed the improvement of students’ achievement (O 02)? 3. What were the factors (internal and external) that affected (positively and negatively) the achievement of the expected results? 4. What practical knowledge can be derived from the SUMA Program and applied in the design process of new initiatives in the framework of the new USAID/Peru Cooperation Strategy? Around each evaluation question a set of more specific questions (sub evaluation questions) has been defined. Those interrogations try to analyze specific topics related with SUMA Program relevance, 73 effectiveness and efficiency. See Annex A (Evaluation Question Matrix). Each of the questions can be answered using the existing source of information mentioned in the matrix. Evaluation Methods Overall Design The present performance evaluation requires collecting, organizing, analyzing and triangulating quantitative and qualitative information related to selected program results and their respective outcomes and requirements. It means that this is a mixed evaluation exercise. As usual, this performance evaluation will ask about the program´s implementation process, its results and early evidence of the likelihood it will be sustained. The evaluation team will translate into operational tools the content of the Evaluation Questions Matrix. Some instruments will imply a documental analysis for rebuilding the logic of the intervention, considering the changes occurred during its implementation. Other instruments will focus the attention in collecting information from counterparts for showing evidences about the pertinence, effectiveness and efficiency of the Program Results and Outcomes. A complementary analysis is that of the ECE results from 2013 that will be available in March. This analysis will focus the measurements in the schools that were benefit from the Program intervention. The performance evaluation will imply a fieldwork in three regions: San Martin, Ucayali and Ayacucho. Data Collection and Analysis Plan The proposed data collection method is as follows: Evaluation Questions Type of Answer Needed Data Collection Methods Data Source Sampling or Selection Criteria Data Analysis Methods To what extent has the SUMA Program promoted specific changes in the education decentralization policy and institutional framework (O 01)? Descriptive and analytic (specification of identified changes). Formal request of documents. Management reports. M&E system. Comprehensive. Disaggregation by region and gender (as possible). To want extent has the SUMA Program promoted strategies and models in education quality that addressed the improvement of students’ achievement (ECE￾communication) (O 02)? Descriptive and analytic (analysis of ECE results) Formal request of information. Management reports. M&E system. Field work. Sampling (by region). Triangulation. Disaggregation by region and gender (as possible). What were the factors (internal and external) that affected (positively and negatively) the Descriptive and analytic Coordinatio n SUMA Program staff and Selection (by region). Triangulation. Disaggregation by region and 74 achievement of the expected results? stakeholders gender (as possible). What practical knowledge can be derived from the SUMA Program and applied in the design process of new initiatives in the framework of the new USAID/Peru Cooperation Strategy? Descriptive and analytic Coordinatio n SUMA Program staff and stakeholders Selection (by region). Triangulation. Disaggregation by region and gender (as possible). Methodological Strengths and Limitations As SUMA Program has produced different conceptual documents on education issues and has strengthened a social movement around quality education, at national and sub-national levels, these conditions represent a programmatic strength that the evaluation team will have to use in benefit of the performance evaluation process. One crucial limitation that SUMA Program faced is its programmatic redefinition. It meant that from one year to another, there have been some significant changes in the activities. This situation represents a limitation that the evaluation team will have to assume. Existing Performance Information During its implementation, the SUMA Program prepared, submitted and discussed with USAID/Peru all the Quarterly Reports. They are available on print or electronic version. Its M&E system contains all the information related to the Program indicators (defined at Outcome level starting with the baseline). The different amendments/modifications have to be requested to USAID/Peru. The SUMA Program has a website (http://www.sumaeducacion.pe/) where there are different documents on the implementation process and specific products achieved (institutional documents, education management tools, press notes, etc.). All these material are process references of the implementation. Deliverables and Timeline In the framework of the present performance evaluation, the main expected product is the evaluation report (draft and final version) with the specification of practical recommendations derived from the conclusions. But some process products are key deliverables: work plan and calendar, kit of tools (collection, analysis and organization of information), etc. 75 Table 1: Contents and schedule of deliverables Deliverables Contents Due date 1 Tools and protocols for data collection 2th week 2 Documentary report 4th week 3 Presentation of preliminary findings 8th week 4 Final report including USAID/Peru revision 10th week The evaluation process will last 10 weeks. It should start the last week of January and should end by the end of March. It means that the preliminary version of the report should be submitted on March 21 and the final version should be submitted after receiving USAID/Peru comments, no later than March 30. A timeline for review is presented below: Table2: Timeline for Evaluation Implementation Tasks Weeks 1 2 3 4 5 6 7 8 9 10 1 Evaluation design and work plan (tools included). 2 Desk review and initial documentary findings. Validation of tools for the field work. Field work (national and sub-national level). Analysis and triangulation of information. Statistical analysis of ECE results 3 Presentation of evaluation report (preliminary). Improvement of evaluation report. 4 Presentation of evaluation report (final). Evaluation Team Project staff will work with a team of evaluators to complete the design, which includes the data collection tools, the list of key informants. The evaluation team should be composed as follows: One team leader: evaluation expert, with specialization in education issues, with a record of evaluation processes in the Peruvian context. Two education specialists (policy and institutional framework and educational results) with broad experience at sub-national level. One statistical analyst with experience in educational achievements analysis. 76 The evaluation team will be accompanied and monitored by USAID/Peru and Evaluations project staff. All the contacts with national or local institutions and actors in the framework of this performance evaluation will be established by USAID/Peru. Each product (preliminary or final version) will be reviewed and commented by USAID/Peru and Evaluations Project staff. All the logistics arrangements will be held by the subcontractor (travels, airfare, accommodation, and daily stipends. Reporting and Dissemination It is expected that this report will be written to address a decision-making audience, which includes USAID/Peru staff, the implementing partners, and regional and local authorities. The report should be a thoughtful, well-researched, well-organized, and objectively evaluate what worked, what did not, and why. Main parts are the:  Executive Summary: A brief overview of the evaluation purpose, project background, evaluation questions, methods, findings, and conclusions.  Evaluation Questions: Address all evaluation questions in the statement of work.  Methods: Explain evaluation methodology in detail. Disclose evaluation limitations, especially those associated with the evaluation methodology.  Findings: Present findings as analyzed facts, evidence and data supported by strong quantitative or qualitative evidence.  Recommendations: Support recommendations with specific findings. Provide recommendations that are action-oriented, practical, specific, and define who is responsible for the action. Annexes: Include the following as annexes, at minimum:  Statement of Work.  Full description of evaluation methods.  All evaluation tools (questionnaires, checklists, discussion guides, surveys, etc.).  A list of sources of information (key informants, documents reviewed, other data sources). 77 Anexo A: Matriz de preguntas de evaluación Objetivos Preguntas Principales Sub-Preguntas Fuentes Analizar críticamente el nivel de logro de los resultados esperados del Programa SUMA en relación con sus componentes de política y marco institucional de descentralización educativa y de mejoramiento de la calidad de la enseñanza. ¿En qué medida el Programa SUMA promovió cambios significativos tanto en la política de descentralización educativa como en el marco institucional del sector educación a nivel nacional y regional? ¿Cuáles fueron los problemas priorizados en términos de política de descentralización educativa y marco institucional del sector educación que buscaron ser resueltos y/o minimizados por medio del Programa SUMA a nivel nacional y regional? ¿Cuán pertinente y articulada fue la propuesta inicial de intervención del Programa SUMA respecto a los problemas priorizados en relación con descentralización educativa y marco institucional y sus posteriores modificaciones programáticas? ¿Qué razones estuvieron detrás del conjunto de modificaciones programáticas realizadas a lo largo de la implementación y qué ganó y qué perdió el Programa SUMA a raíz de ellas en relación con los resultados esperados en política de descentralización educativa y marco institucional? ¿En qué consistió el proceso de fortalecimiento de la política y del marco institucional para la descentralización educativa a nivel nacional y regional implementado por el Programa SUMA y qué evidencias objetivas lograron ser identificadas como resultados (Requisito 1.1)? ¿Cuáles fueron las buenas prácticas en gestión educativa descentralizada identificadas y discutidas por el Programa SUMA con las contrapartes a nivel nacional y regional y cuál fue el nivel de apropiación de las mismas por el sector educación (Requisito 1.2)? ¿Qué contenidos y procesos claves fueron implementados por el Programa SUMA en el marco del fortalecimiento de los sistemas y procesos de gestión educativa promovidos a nivel nacional y regional y qué evidencias objetivas lograron Sistema de monitoreo del Programa SUMA. Análisis documental interno: balance sobre política y marco institucional para la descentralización educativa, informe de lecciones aprendidas sobre experiencias exitosas de gestión descentralizada, reportes de asistencia técnica a nivel nacional y regional y documentos técnicos de presupuesto público. Análisis documental externo (contexto): diagnóstico del sistema educativo e identificación de sus prioridades (nacionales y regionales), iniciativas similares en implementación, tendencias en la región de América Latina, etc. Entrevistas a personal clave a nivel nacional y regional: funcionarios del MINEDU, investigadores, académicos, periodistas, líderes de opinión, representantes de organizaciones, etc. 78 ser identificadas como resultados (Requisito 1.3)? ¿Cuáles fueron los procesos de diálogo, desarrollo de capacidades y toma de decisión promovidos por el Programa SUMA relacionados con el incremento de la inversión pública en educación y qué evidencias objetivas lograron ser identificadas como resultados (Requisito 1.4)? ¿Cuán funcional fue la estructura organizacional del Programa SUMA (recursos humanos) para el logro de resultados relacionados con la política de descentralización educativa y el marco institucional del sector educación a nivel nacional y regional? ¿Cuáles fueron los factores internos y externos que contribuyeron / limitaron el logro de resultados del Programa SUMA relacionados con la política de descentralización educativa y el marco institucional del sector educación a nivel nacional y regional? ¿Qué decisiones no previstas tuvo que asumir el Programa SUMA en relación con el mejoramiento de la política de descentralización educativa y marco institucional del sector educación y qué implicancias prácticas tuvieron? ¿En qué medida el Programa SUMA promovió modelos y estrategias de mejoramiento de la calidad de la enseñanza conducentes al logro educativo de los estudiantes y que fueran institucionalizados en el sector educación nivel nacional y regional? ¿Cuáles fueron los principales problemas detectados en relación con la calidad de la enseñanza y sus efectos que el Programa SUMA asumió a nivel nacional y regional y respecto a los cuales ofreció una alternativa de respuesta? ¿Cuán coherente y funcional fue la propuesta de solución seleccionada e implementada por el Programa SUMA a la problemática de calidad de la enseñanza y en qué aspectos se distinguió de otras iniciativas en curso? ¿En qué aspectos dicha propuesta de solución a la calidad de la enseñanza tuvo cambios a raíz de las modificaciones programáticas aplicadas al Programa SUMA? ¿Qué hitos formaron parte del proceso de fortalecimiento Sistema de monitoreo del Programa SUMA. Análisis documental interno: balance sobre política y marco institucional de capacitación docente, informe de lecciones aprendidas sobre experiencias exitosas de capacitación docente y reportes de asistencia técnica. Análisis documental externo (contexto): política de capacitación docente del MINEDU, 79 de la política nacional y regional y marco institucional para la capacitación docente y desarrollo profesional implementado por el Programa SUMA y qué evidencias objetivas lograron ser identificadas como resultados (Requisito 2.1)? ¿Cuáles fueron las buenas prácticas en capacitación docente y desarrollo profesional identificadas y promovidas por el Programa SUMA con las contrapartes a nivel nacional y regional y cuál fue el nivel de institucionalización de las mismas por el sector educación (Requisito 2.2)? ¿Qué desafíos y qué lecciones pudieron ser identificados en la capacitación en servicio de las buenas prácticas de enseñanza de lectura en IIEE multigrado y bilingües promovidas por el Programa SUMA y en su proceso de incorporación dentro del sistema de capacitación del sector educación (Requisito 2.3)? ¿Cuáles fueron las mejoras en el logro educativo de los estudiantes (ECE - área de comunicación) de las instituciones educativas en las zonas de intervención promovidas por el Programa SUMA? intervenciones similares que integran capacitación docente, tendencias en la región de América Latina, etc. Entrevistas a personal clave a nivel nacional y regional: funcionarios del MINEDU, investigadores, académicos, periodistas, líderes de opinión, representantes de organizaciones, etc. Regiones de análisis: San Martín, Ucayali y Ayacucho Resultados de ECE 2013 (disponibles marzo 2014) Identificar los factores de éxito, lecciones aprendidas y recomendaciones clave extraíbles de los componentes evaluados del Programa SUMA que pueden ser aplicables en la formulación de nuevas iniciativas, en especial, o en la gestión del conocimiento, en general, dentro de la nueva estrategia de ¿Cuáles fueron los elementos críticos (internos y externos) del Programa SUMA que favorecieron el logro de los resultados esperados y cuáles no? ¿Cuáles fueron los aspectos de carácter político, programático y estratégico que contribuyeron positiva o negativamente en la implementación de los componentes evaluados y en el logro de los resultados esperados? ¿Qué nivel de articulación y complementariedad hubo entre los resultados esperados y los requisitos definidos por el programa? ¿Contaron los indicadores del Programa SUMA con el nivel de operacionalización técnica requerida para su monitoreo eficiente? ¿Cómo la gestión de conocimiento (estudios especializados) impulsada con cierto énfasis por el Análisis documentos (fuentes diversas). Entrevistas a personal clave. 80 cooperación (2012 – 2016). Programa SUMA sirvió para lograr reformas sistémicas y desarrollo de capacidades en el sector educación? ¿Cuáles decisiones asumidas por el Programa SUMA fueron útiles en la relación con los PIP (diseño y/o implementación) y el programas presupuestal vinculado por el sector educación y cuáles no? ¿Qué decisiones asumidas e implementadas por el Programa SUMA no deberían ser replicadas por ninguna otra intervención similar y por qué? ¿En qué medida y cómo el Programa SUMA recogió e integró los objetivos de desarrollo definidos en la nueva estrategia de desarrollo de USAID? ¿Qué conocimiento práctico puede ser derivado de la experiencia desarrollada por el Programa? ¿Qué aprendizajes específicos pueden ser extraídos de los contenidos (qué) y las modalidades (cómo) de asistencia técnica especializada implementados por el Programa? ¿Cuáles son las constataciones que pueden ser extraídas de la experiencia del Laboratorio de Descentralización en San Martín? Análisis documentos (fuentes diversas). Entrevistas a personal clave. Distritos seleccionados en San Martín ¿Cómo capitalizar lo aprendido en el Programa SUMA dentro del diseño de nuevas intervenciones orientadas a mejorar la gestión y calidad de los servicios públicos en la Amazonía? ¿Qué tipo de consideraciones de carácter político, programático, estratégico u operativo deben ser tomadas en cuenta en la formulación e implementación de intervenciones similares? ¿Qué hipótesis de desarrollo podría reconstruirse a partir del análisis del Programa? Análisis documentos (fuentes diversas). Entrevistas a personal clave. 81 Anexo B: Modelo de guía de entrevista para informantes regionales Guía de entrevista a entidades regionales y locales Nota: La presente guía de entrevista presenta una operacionalización de las preguntas de evaluación y están focalizadas en los productos del Programa SUMA. Está orientada a ser aplicada al conjunto de entidades regionales y locales con los cuales el Programa SUMA trabajó directamente. En la aplicación de este instrumento, el/la entrevistador/a puede / debe formular preguntas adicionales que ayuden a abordar temas más específicos relacionados con la evaluación de desempeño. 1. Identificación de informante Estimado/a funcionario/a:  Esta entrevista forma parte de la evaluación de desempeño del Programa SUMA. Sus respuestas son importantes porque ayudarán a identificar lecciones aprendidas y buenas prácticas que pueden ser aplicadas en futuras iniciativas.  Toda la información será tratada con confidencialidad. Le solicitamos su autorización para registrar la entrevista ya que la grabación nos servirá para revisar con mayor detenimiento la información.  Agradecemos su colaboración. 1.1. Institución / entidad 1.2. Nombre de informante: 1.3. Cargo / función: 1.4. Tiempo en el cargo: 1.5. Lugar de aplicación: 1.6. Fecha de aplicación: 1.7. Hora de aplicación: 1.8. Responsable de aplicación: 82 2. Criterios de evaluación: pertinencia y eficacia Preguntas Respuestas Códigos de preguntas de evaluación 2.1. ¿Qué problemas del sector educación, en general, intentó resolver / aminorar el Programa SUMA (planificación, organización, capacidad de ejecución, participación / información, calidad de la enseñanza y formación docente)? 1.1 2.1 2.1.1. ¿Cuáles fueron las respuestas del Programa SUMA a los problemas identifica -dos? 1.2 2.1.2. ¿Cuán articuladas fue -ron las respuestas a los problemas identifica -dos? 1.2 2.2. ¿Qué resultados logró el Programa SUMA (a nivel nacional y regional) en relación con los problemas detectados? 1.4 1.6 1.7 2.2.1. ¿Habían otros actores proponiendo soluciones a dichos problemas?, ¿qué proponían? 1.4 1.6 1.7 2.2.2. ¿Trabajó el Programa SUMA con el GR / GL / DRE / UGEL / IIEE para responder a los problemas del sector educación detectados?, ¿en qué consistió dicho trabajo? 1.4 1.6 1.7 2.2.3. ¿Qué hitos y productos pueden identificarse en relación con ese trabajo conjunto? 1.4 1.6 1.7 2.2.4. ¿Fue apropiado por el GR / GL / DRE / UGEL / IIEE algún producto del Programa SUMA?, cuál/es? 1.5 83 1. Sobre productos del Programa SUMA Dimensiones Productos del Programa SUMA ¿En qué consistía el instrumento denominado…? ¿Existía en el sector previamente algún instrumento similar a este? ¿Cuál fue la novedad o valor agregado de este instrumento? ¿Qué preparación tuvieron para usar este instrumento? ¿Cómo utilizaron este instrumento? ¿Qué resultados tuvieron con este instrumento? ¿Qué desafíos aún existen en relación con este instrumento? Planificación Guía del PMP POI DRE / UGEL San Martín Guía del PALMA (marco estratégico y operativo) Guía de PAT￾RED Guía de PATMA 84 Dimensiones Productos del Programa SUMA ¿En qué consistía el instrumento denominado…? ¿Existía en el sector previamente algún instrumento similar a este? ¿Cuál fue la novedad o valor agregado de este instrumento? ¿Qué preparación tuvieron para usar este instrumento? ¿Cómo utilizaron este instrumento? ¿Qué resultados tuvieron con este instrumento? ¿Qué desafíos aún existen en relación con este instrumento? Otro: ¿Plan de Emergencia para la mejora de aprendizajes…? Presupuesto Guía de gestión del PELA Instrumento del Cálculo del Costo por Alumno (INCCA) Guía del PMP (sección costeo) Guía del PALMA (sección costeo) 85 Dimensiones Productos del Programa SUMA ¿En qué consistía el instrumento denominado…? ¿Existía en el sector previamente algún instrumento similar a este? ¿Cuál fue la novedad o valor agregado de este instrumento? ¿Qué preparación tuvieron para usar este instrumento? ¿Cómo utilizaron este instrumento? ¿Qué resultados tuvieron con este instrumento? ¿Qué desafíos aún existen en relación con este instrumento? Metodología SNIP Costeo de PELA a través de SIGMA Otro: Control: sistemas de información Sistema de información estratégica para la toma de decisiones (regional / local). Sistema Integrado de la Gestión del Monitoreo y Acompañamiento del PELA (SIGMA) 86 Dimensiones Productos del Programa SUMA ¿En qué consistía el instrumento denominado…? ¿Existía en el sector previamente algún instrumento similar a este? ¿Cuál fue la novedad o valor agregado de este instrumento? ¿Qué preparación tuvieron para usar este instrumento? ¿Cómo utilizaron este instrumento? ¿Qué resultados tuvieron con este instrumento? ¿Qué desafíos aún existen en relación con este instrumento? Sistema de Información de la Implementación del PER (a través del CNE). Instrumentos de monitoreo y evaluación para los planes. Mecanismos de Vigilancia y Control Participativo (CGLDS, CONEI, COPRED) Otro: Organización Modelo de Gestión Educativa Regional 87 Dimensiones Productos del Programa SUMA ¿En qué consistía el instrumento denominado…? ¿Existía en el sector previamente algún instrumento similar a este? ¿Cuál fue la novedad o valor agregado de este instrumento? ¿Qué preparación tuvieron para usar este instrumento? ¿Cómo utilizaron este instrumento? ¿Qué resultados tuvieron con este instrumento? ¿Qué desafíos aún existen en relación con este instrumento? Modelo de Gobernanza Local de la Educación ROF DRE / UGEL / GL de Pajarillo y Cuñumbuqui Guía para CGLDS Plan de desarrollo de capacidades en gestión local de la educación Otro: 88 Dimensiones Productos del Programa SUMA ¿En qué consistía el instrumento denominado…? ¿Existía en el sector previamente algún instrumento similar a este? ¿Cuál fue la novedad o valor agregado de este instrumento? ¿Qué preparación tuvieron para usar este instrumento? ¿Cómo utilizaron este instrumento? ¿Qué resultados tuvieron con este instrumento? ¿Qué desafíos aún existen en relación con este instrumento? Calidad de la enseñanza Cuadernos de auto aprendizaje Centros de recursos Municipio escolar Brújula Maestra Programa de Formación de líderes en Alfabetización inicial Formación docente 89 1. Lecciones aprendidas y buenas prácticas Preguntas Respuestas Códigos de preguntas de evaluación 1.1. ¿Qué aprendizajes pueden ser extraídos de las modalidades de asistencia técnica brindadas por el Programa SUMA al sector educación ? 3.8 1.1.1. ¿Cuál fue el valor agregado ofrecido por el Programa SUMA? 3.8 1.2. ¿Cuáles son las lecciones que se pueden derivar del Laboratorio de Descentra -lización promovido por el Programa SUMA ? 3.8 1.2.1. ¿Cuáles son las condiciones mínimas que deben asegurarse para implementar dicha experiencia? 3.8 1.2.2. ¿En qué consiste la estrategia de involucra -miento de los actores? 3.8 1.2.3. ¿Cuáles son los aspectos de la experiencia que aún requieren de mayor maduración? 3.8 1.3. ¿Qué de lo hecho por el Programa SUMA no debería replicarse en otra iniciativa relacionada con el sector educación ? 3.8 3.10 1.4. ¿Qué de lo hecho por el Programa SUMA sí debería replicarse en otra iniciativa relacionada con el sector educación ? 3.8 3.10 1.5. ¿Percibieron mejoras en los aprendizajes de los estudiantes de las IIEE beneficiadas por el Programa SUMA?, ¿cómo así? 3.8 3.10 90 Anexo C: Modelo de guía de entrevista para informantes nacionales Guía de Entrevista Nota: La presente guía de entrevista presenta una operacionalización de las preguntas de evaluación y las organiza, básicamente, según dos de los tres criterios definidos para esta evaluación de desempeño: pertinencia y eficacia. Además, integra preguntas relacionadas con las lecciones aprendidas y buenas prácticas derivadas de la implementación. En la aplicación de este instrumento, el/la entrevistador/a puede / debe formular preguntas adicionales que ayuden a abordar temas más específicos relacionados con la evaluación de desempeño del Programa SUMA. 1. Identificación de informante Estimado/a funcionario/a:  Esta entrevista forma parte de la evaluación de desempeño del Programa SUMA. Sus respuestas son importantes porque ayudarán a identificar lecciones aprendidas y buenas prácticas que pueden ser aplicadas en futuras iniciativas.  Toda la información será tratada con confidencialidad. Le solicitamos su autorización para registrar la entrevista ya que la grabación nos servirá para revisar con mayor detenimiento la información.  Agradecemos su colaboración. 1.1. Institución / entidad 1.2. Nombre de informante: 1.3. Cargo / función: 1.4. Tiempo en el cargo: 1.5. Lugar de aplicación: 1.6. Fecha de aplicación: 1.7. Hora de aplicación: 1.8. Responsable de aplicación: 91 2. Criterio de evaluación: pertinencia Preguntas Respuestas Códigos de preguntas de evaluación 2.1. ¿Qué problemas del sector educación, en general, intentó resolver / aminorar el Programa SUMA?  1.1 2.1 2.2. ¿Qué problemas de planificación del sector educación trató de resolver el Programa SUMA?  1.1 2.2.1. ¿Cuáles fueron las respuestas del Programa SUMA a los problemas de planificación identifica -dos?  1.2 2.2.2. ¿Cuán articuladas fue -ron las respuestas a los problemas identifica -dos?  1.2 2.3. ¿Qué problemas de organización del sector educación trató de resolver el programa SUMA?  1.1 2.3.1. ¿Cuáles fueron las respuestas del Programa SUMA a los pro - blemas de organización identificados?  1.2 2.3.2. ¿Cuán articuladas fue -ron las respuestas a los problemas identifica -dos?  1.2 92 2.4. ¿Qué problemas de capacidad de ejecución del sector educación trató de resolver el Programa SUMA?  1.1 2.4.1. ¿Cuáles fueron las respuestas del Programa SUMA a los problemas de capacidad de ejecución identificados?  1.2 2.4.2. ¿Cuán articuladas fue -ron las respuestas a los problemas identifica -dos?  1.2 2.5. ¿Qué problemas de participación / información del sector educación trató de resolver el Programa SUMA?  1.1 2.5.1. ¿Cuáles fueron las respuestas del Pro -grama SUMA a los problemas de participación / información identificados?  1.2 2.5.2. ¿Cuán articuladas fue -ron las respuestas a los problemas identifica -dos?  1.2 2.6. ¿Qué problemas de calidad de la enseñanza trató de resolver el Programa SUMA?  2.1 2.6.1. ¿Cuáles fueron las respuestas del Programa SUMA a los  1.2 93 problemas de calidad de la enseñanza identificados?  2.6.2. ¿Cuán articuladas fue -ron las respuestas a los problemas identifica -dos?  1.2 2.7. Qué problemas de formación docente trató de resolver el Programa SUMA?  2.1 2.7.1. ¿Cuáles fueron las respuestas propuestas por el Programa SUMA a los problemas de formación docente iden - tificados?  1.2 2.7.2. ¿Cuán articuladas fue -ron las respuestas a los problemas identifica -dos?  1.2 3. Criterio de evaluación: eficacia Preguntas Respuestas Códigos de preguntas de evaluación 3.1. ¿Qué resultados logró el Programa SUMA (a nivel nacional y regional) en relación con la problemática de planificación detectada?  1.4 1.6 1.7 3.1.1. ¿Habían otros actores proponiendo soluciones a la problemática de planificación?, ¿qué proponían?  1.4 1.6 1.7 94 3.1.2. ¿Trabajo el Programa SUMA con el CNE para responder a los problemas de planificación del sector educación?, ¿en qué consistió dicho trabajo?  1.4 1.6 1.7 3.1.3. ¿Qué hitos y productos pueden identificarse en relación con ese trabajo conjunto?  1.4 1.6 1.7 3.1.4. ¿Fue apropiado por el sector educación algún producto del Programa SUMA – CNE sobre planificación ?  1.5 3.2. ¿Qué resultados logró el Programa SUMA (a nivel nacional y regional) en relación con la problemática de organización detectada?  1.4 1.6 1.7 3.2.1. ¿Habían otros actores proponiendo soluciones a la problemática de organización?, ¿qué proponían?  1.4 1.6 1.7 3.2.2. ¿Trabajo el Programa SUMA con el CNE para responder a los problemas de organización del sector educación?, ¿en qué consistió dicho trabajo?  1.4 1.6 1.7 3.2.3. ¿Qué hitos y productos pueden identificarse en relación con ese trabajo conjunto?  1.4 1.6 1.7 3.2.4. ¿Fue apropiado por el sector educación algún producto del Programa SUMA – CNE sobre organización ?  1.5 95  3.3. ¿Qué resultados logró el Programa SUMA (a nivel nacional y regional) en relación con la problemática de capacidad de ejecución detectada?  1.4 1.6 1.7 3.3.1. ¿Habían otros actores proponiendo soluciones a la problemática de capacidad de ejecución?, ¿qué proponían?  1.4 1.6 1.7 3.3.2. ¿Trabajo el Programa SUMA con el CNE para responder a los problemas de capacidad de ejecución del sector educación?, ¿en qué consistió dicho trabajo?  1.4 1.6 1.7 3.3.3. ¿Qué hitos y productos pueden identificarse en relación con ese trabajo conjunto?  1.4 1.6 1.7 3.3.4. ¿Fue apropiado por el sector educación algún producto del Programa SUMA – CNE sobre capacidad de ejecución ?  1.5 3.4. ¿Qué resultados logró el Programa SUMA (a nivel nacional y regional) en relación con la problemática de participación / información detectada?  1.4 1.6 1.7 3.4.1. ¿Habían otros actores proponiendo soluciones a la problemática de participación / información?, ¿qué proponían?  1.4 1.6 1.7 96 3.4.2. ¿Trabajo el Programa SUMA con el CNE para responder a los problemas de participación / información del sector educación?, ¿en qué consistió dicho trabajo?  1.4 1.6 1.7 3.4.3. ¿Qué hitos y productos pueden identificarse en relación con ese trabajo conjunto?  1.4 1.6 1.7 3.4.4. ¿Fue apropiado por el sector educación algún producto del Programa SUMA – CNE sobre participación / información ?  1.5 3.5. ¿Qué resultados logró el Programa SUMA (a nivel nacional y regional) en relación con la problemática de calidad de la enseñanza detectada?  2.2 3.5.1. ¿Habían otros actores proponiendo soluciones a la problemática de calidad de la enseñanza?, ¿qué proponían?  2.2 2.4 3.5.2. ¿Trabajo el Programa SUMA con el CNE para responder a los problemas de calidad de la enseñanza del sector educación?, ¿en qué consistió dicho trabajo?  2.2 2.4 3.5.3. ¿Qué hitos y productos pueden identificarse en relación con ese trabajo conjunto?  2.4 3.5.4. ¿Fue apropiado por el sector educación algún producto del  2.5 97 Programa SUMA – CNE sobre calidad de la enseñanza ?  3.6. ¿Qué resultados logró el Programa SUMA (a nivel nacional y regional) en relación con la problemática de formación docente detectada?  2.2 3.6.1. ¿Habían otros actores proponiendo soluciones a la problemática de formación docente?, ¿qué proponían?  2.2 2.4 3.6.2. ¿Trabajo el Programa SUMA con el CNE para responder a los problemas de formación docente del sector educación?, ¿en qué consistió dicho trabajo?  2.2 2.4 3.6.3. ¿Qué hitos y productos pueden identificarse en relación con ese trabajo conjunto?  2.4 3.6.4. ¿Fue apropiado por el sector educación algún producto del Programa SUMA – CNE sobre formación docente ?  2.5 3.7. ¿Qué factores (internos / externos o de carácter político / programático / estratégico), en general, ayudaron y/o limitaron al Programa SUMA para alcanzar los resultados esperados?  1.9 3.8. ¿Qué tipo de procesos de reforma pudo apalancar el Programa SUMA, en general, dentro del sector educación?  3.4 3.5 98 3.8.1. ¿Qué pudo apalancar el Programa SUMA con el conocimiento generado (estudios)?, ¿qué dificultades aparecieron?, ¿cuáles fueron los resultados?  3.4 3.8.2. ¿Qué pudo apalancar el Programa SUMA con el diseño e implementación de PIPs?, ¿qué dificultades apare - cieron?, ¿cuáles fueron los resultados?  3.5 3.8.3. ¿Qué pudo apalancar el programa SUMA con el modelo de gobernanza local?, ¿qué dificultades aparecieron?, ¿cuáles fueron los resultados?  3.5 3.8.4. ¿Qué pudo apalancar el Programa SUMA con la asistencia técnica al PELA?, ¿qué dificultades aparecieron?, ¿cuáles fueron los resultados?  3.5 4. Lecciones aprendidas y buenas prácticas Preguntas Respuestas Códigos de preguntas de evaluación 4.1. ¿Qué aprendizajes pueden ser extraídos de las modalidades de asistencia técnica brindadas por el Programa SUMA al sector educación ?  3.8 4.1.1. ¿Cuál fue el valor agregado ofrecido por el Programa SUMA?  3.8 4.2. ¿Cuáles son las lecciones que se pueden derivar del Laboratorio de  3.8 99 Descentralización promovido por el Programa SUMA ?  4.2.1. ¿Cuáles son las condiciones mínimas que deben asegurarse para implementar dicha experiencia?  3.8 4.2.2. ¿En qué consiste la estrategia de involucra-miento de los actores?  3.8 4.2.3. ¿Cuáles son los aspectos de la experiencia que aún requieren de mayor maduración?  3.8 4.3. ¿Qué de lo hecho por el Programa SUMA no debería replicarse en otra iniciativa relacionada con el sector educación ?  3.8 3.10 4.4. ¿Qué de lo hecho por el Programa SUMA sí debería replicarse en otra iniciativa relacionada con el sector educación ?  3.8 3.10 100 Annex 2: List of interviewees Nro. Nombre de informante Lugar Institución / organización Función 1 Idel Vexler Lima CNE Consejero 2 Mariano Aliaga Lima CNE Equipo técnico 3 Yina Rivera Lima CNE Equipo técnico 4 Carolyna Neyra Lima CNE Asesora 5 Nancy Torrejón Lima CNE Directora Ejecutiva 6 María Diez Lima Consultora Consultora 7 Sibory Paz Dolores Lima DRE Lima Especialista 8 Carlos Arámbulo Lima MCLCP Especialista 9 Elena Burga Lima MINEDU Directora DIGEIBIR 10 Daniel Jesús Lima Programa SUMA Responsable de M&E 11 Cecilia Ramírez Lima Programa SUMA Directora 12 Marita Palacios Lima Programa SUMA Ex Directora 13 Equipo implementador del Programa SUMA Lima Programa SUMA Equipo técnico 14 Marita Palacios Lima Programa SUMA (ex) Ex Directora 15 Lilian Hidalgo Lima Tarea Directora 16 Gisele Cuglievan Lima UNICEF Oficial de Educación 17 Walter Twanama Lima USAID/Peru Especialista de Educación 18 Angela Asunción Arévalo San Martín CGLDS - Cuñumbuque Miembro 19 Alodia Lara San Martín CGLDS - Cuñumbuque Directora de Red Educativa 20 Sary Trigoso Paredes San Martín CRA - UGEL - Cuñumbuque Responsable de CRA 21 Mercedes Ortiz Torres San Martín DRE Directora 22 Wilson Guerra Villacorta San Martín DRE Especialista 23 Tito Portocarrero / Milagros Mendoza San Martín DRE / Programa SUMA Especialista / Facilitadora 24 María Irrazabal / Patzi Tuesta / Pedro Isla / Víctor Panduro San Martín IIEE - UGEL Cuñumbuque Docentes / directores / especialista 25 Marren Rubí Ríos San Martín IIEE - UGEL Pajarillo Director 26 Robinson Viena San Martín IIEE - UGEL Pajarillo Director / Profesor 27 Amalia Vasquez San Martín IIEE - UGEL Pajarillo Profesora 28 Danilo Panduro San Martín IIEE - UGEL Pajarillo Profesor 29 Lenin Malaver San Martín Municipalidad de Pajarillo Profesor 30 Teócrito Pineado San Martín Programa SUMA Coordinador Regional 101 31 Sofía Paredes Flores San Martín UGEL - Cuñumbuque Directora 32 Josefa Hurtado Dávila San Martín UGEL - Cuñumbuque Especialista 33 Silvia Vargas San Martín UGEL Cuñumbuque Profesora 34 Marco A. Saldaña San Martín UGEL Cuñumbuque Director / Profesor 35 Gisela Godier Ucayali DRCyT Directora 36 Angel Rolando Wong Guerra Ucayali DREU Especialista 37 Isabel Arce Ucayali DREU Especiliasta 38 Teresa Revilla Ucayali DREU Consultora 39 Víctor García Ucayali DREU Planificador 40 José Díaz Ucayali DREU Director de la DRE 41 Jorge Zagaceta Ucayali DREU Planificador 42 Wagner Sánchez Ucayali DREU Especialista 43 Pedro Rivero Paredes Ucayali GRU Sub Gerente de Desarrollo Social 44 Ludgardo Gutierrez Ucayali GRU Gerente de Desarrollo Social 45 Richard Dávila / Jorge Guevara Ucayali MCLCP Coordinador Regional / Secretario Técnico 46 Enrique Vega / Clara Amelia Quispe Ucayali PELA Acompañantes pedagógicos 102 Annex 3: Status of public investment projects reported by SUMA Tabla 9: Relación de Proyectos de Inversión Pública (PIP) según código y fecha de aprobación SNIP, código presupuestal y estado actual CODIGO SNIP Fecha Perfil CODIGO DNPP Fecha último registro Estado del registro Región Nombre del Proyecto Presupuesto Perfil 34904 01/06/2006 2078233 04/07/2007 Ejecutado San Martín Fortalecimiento institucional y docente para el desarrollo de capacidades en instituciones educativas focalizadas de la región San Martín S/. 1,955,696 69189 27/11/2007 2078196 07/08/2008 Ejecutado Ucayali Fortalecimiento de capacidades de los procesos de enseñanza y aprendizaje en las instituciones educativos de la zona rural y bilingüe de las provincias de Coronel Portillo, Padre Abad y Atalaya de la región Ucayali S/. 5,701,555 76010 11/02/2008 2060631 25/02/2008 Ejecutado Amazonas Mejoramiento de la calidad educativa en la zona rural de 12 distritos de la región Amazonas S/. 5,870,647 123350 06/07/2009 2115836 09/04/2010 En ejecución Ayacucho Fortalecimiento de capacidades pedagógicas y de gestión en el sistema educativo de la región Ayacucho S/. 5,987,467 131794 21/07/2010 2132069 27/07/2010 En ejecución Lima Provincias Desarrollo de capacidades de los actores educativos de la EBR de 12 distritos de la región Lima Provincias S/. 4,997,419 151844 14/04/2010 09/11/2010 Viable Junín Mejoramiento de la calidad educativa en las instituciones educativas del nivel primaria de la EBR ubicados en 17 distritos del primer quintil de la región S/. 5,876,422 156154 09/06/2010 27/10/2010 Fase de inversión San Martín Creación del sistema regional de desarrollo y formación docente continua para el perfeccionamiento de las competencias profesionales del profesorado de la región S/. 5,210,571 169746 27/12/2010 19/01/2011 Observado San Martín Implementación piloto del nuevo diseño curricular regional y del modelo de gestión educativo regional para mejorar la calidad de los servicios educativos en el nivel inicial y primario de EBR de las IIEE de 6 distritos S/. 5,946,569 185286 03/08/2011 2150557 03/05/2013 En ejecución Ayacucho (Huanca Sancos) Mejoramiento de los logros de aprendizaje en los estudiantes de la EBR a través del enfoque de escuelas activas en la provincia de Huanca Sancos S/. 3,679,777 208856 19/03/2012 2171369 12/03/2013 Fase de inversión Nacional Mejoramiento de la gestión educativa descentralizada de IIEE en ámbito rural de 24 regiones S/. 103,987,794 TOTAL S/. 149,213,917 Fuente: MEF, Banco de Proyectos SNIP http://www.mef.gob.pe/contenidos/inv_publica/new-bp/operaciones-bp.php 103 Tabla 10: Monto asignado según Presupuesto Institucional Modificado (PIM) y Monto Ejecutado según Devengados, por PIP y por año CODIGO SNIP Asignación Presupuestal (miles) Monto ejecutado (miles) % Asig nado % Ejecu 2009 2010 2011 2012 2013 2014 2009 2010 2011 2012 2013 2014 tado 34904 S/.689 S/.1,005 S/.689 S/.1,005 87% 100% 69189 S/.850 S/.1,100 S/.2,223 S/.1,510 S/.731 S/.913 S/.922 S/.1,288 100 68% 76010 S/.1,306 S/.2,208 S/.1,731 S/.1,584 S/.465 S/.1,114 S/.1,785 S/.1,726 S/.1,584 S/. 465 124 91% 123350 S/. 494 S/.1,948 S/.907 S/.2,245 S/.433 S/.1,946 S/.869 S/. 17 93% 58% 131794 S/. 71 S/. 16 S/. 54 S/. 28 S/. 67 S/.16 S/. 54 S/. - 3% 81% 151844 0 0 156154 0 0 169746 0 0 185286 S/.849 S/. 29 S/.848 S/. - 24% 97% 208856 0 0 TOTAL S.2,846 S/.4,314 S/.4,520 S/.5,060 S/.2,276 S/.2,303 S/.2,535 S/.3,704 S/.3,150 S/.4,836 S/.2,237 S/. 17 14% 77% Fuente: MEF, Seguimiento de la ejecución presupuestal. http://apps5.mineco.gob.pe/transparencia/mensual/default.aspx?y=2014&ap=Proyecto 104 Annex 4: Table of indicators per contract and indicators reported Indicadores según Modificación 6 Indicadores reportados R1 Marco de políticas e institucional mejorado R1.1 Fortalecer el marco de políticas e institucional en educación Diagnóstico inicial del contexto institucional y de políticas, monitoreado y actualizado anualmente Asesoría en temas de política clave proporcionada a USAID, MINEDU y otros actores relevantes Por lo menos cuatro políticas nacionales y/o reformas vinculadas a descentralización de la educación elaboradas y aprobadas IE40: Número de leyes, políticas, normas o lineamientos desarrollados o modificados para mejorar el acceso o la calidad de los servicios educativos Por lo menos tres tomadores de decisión y funcionarios clave del MINEDU participan en las etapas del diagnóstico institucional y de políticas IE: Número de administradores y funcionarios capacitados R1.2 Revisar y evaluar buenas prácticas sobre sistemas y procesos de gestión descentralizada Al menos tres metodologías de buenas prácticas sobre gestión descentralizada identificadas, incluyendo los requerimientos y las condiciones para replicarlas en diferentes contextos, particularmente en áreas rurales Contrapartes del gobierno se involucran y apropian de la revisión y análisis realizado, cinco del MINEDU IE: Número de administradores y funcionarios capacitados R1.3 Fortalecer sistemas y procesos de la gestión descentralizada DRE y UGEL en regiones priorizadas se reestructuran de acuerdo a las necesidades locales en el marco de la LOPE Nivel de avance en la reestructuración de DREs y UGELs en regiones priorizadas Regiones priorizadas implementan plan para modernizar los procesos de gestión educativa (planificación, presupuesto, monitoreo) de acuerdo con las políticas nacionales (a) Nivel de avance en el desarrollo e implementación de un sistema de monitoreo y evaluación para mejorar la gestión en regiones priorizadas Anualmente, por lo menos 250 funcionarios o autoridades del MINEDU y de 5 regiones priorizadas reciben asistencia técnica y capacitación para mejorar los sistemas de gestión en el marco de la descentralización (En el 2013 el foco es San Martín y la meta 80 funcionarios) (IE) Número de administradores y funcionarios capacitados (incluye capacitadores y acompañantes) Funcionarios de educación de regiones priorizadas usan datos para la toma de decisiones en coordinación con la Unidad de Medición de la Calidad de MINEDU (a) (IE) Número de administradores y funcionarios capacitados en monitoreo y evaluación Por lo menos 20 escuelas demostrativas en San Martín mantienen los niveles de aprendizaje de los alumnos (b) Número de escuelas apoyadas por el proyecto a través de la implementación del laboratorio de descentralización en San Martín Porcentaje de avance en la implementación del laboratorio de descentralización en San Martín 40 IE: indicadores estándar 105 Porcentaje de estudiantes que participan en la implementación del laboratorio de descentralización en San Martín y alcanzan niveles avanzados en el aprendizaje de matemáticas y comunicación R1.4 Mejorar la asignación del presupuesto y la calidad del gasto en educación Presupuesto de educación sometido por MINEDU en el Congreso incorpora las recomendaciones de política de los diálogos promovidos por el proyecto Al menos una autoridad territorial, en las regiones priorizadas, prepara proyectos de inversión pública (PIP) para aumentar la inversión en educación y mejorar la eficiencia en el gasto. Monto anual en Soles ejecutado por proyectos de inversión pública (PIP) en actividades de desarrollo de capacidades en educación, con el apoyo técnico del proyecto Gobiernos regionales priorizados (al menos 1) que reciben apoyo del proyecto implementan la iniciativa de presupuesto por resultados en educación (a) Porcentaje de ejecución del presupuesto del Programa para el Logro de Aprendizajes (PELA) Porcentaje de aumento anual en el presupuesto del PELA en las regiones del proyecto R2 Capacitación y desarrollo profesional docente fortalecidos R2.1 Fortalecimiento de la política nacional y el marco institucional para la capacitación y el desarrollo profesional docente Diagnóstico inicial del contexto institucional y de políticas relacionado con la calidad de la enseñanza, monitoreado y actualizado anualmente Asesoría en temas clave de política vinculados a la calidad de la enseñanza, proporcionadas a USAID, MINEDU y otros actores relevantes Por lo menos cuatro políticas nacionales o regionales, o reformas vinculadas a la formación, desarrollo profesional y acreditación de la educación elaboradas y aprobadas IE: Número de leyes, políticas, normas o lineamientos desarrollados o modificados para mejorar el acceso o la calidad de los servicios educativos Por lo menos tres tomadores de decisión o funcionarios clave del MINEDU participan en las diferentes etapas del diagnóstico institucional y de políticas IE: Número de administradores y funcionarios capacitados R2.2 Revisar y evaluar buenas prácticas en capacitación y desarrollo profesional docente Al menos tres metodologías de buenas prácticas sobre calidad de la enseñanza identificadas, incluyendo los requerimientos y las condiciones para su replicabilidad en diversos contextos, particularmente en áreas rurales Contrapartes del gobierno se involucran en la revisión y análisis y muestran apropiación del proceso de revisión, cinco del MINEDU IE: Número de administradores y funcionarios capacitados R2.3 Proporcionar asistencia técnica para mejorar la capacitación docente en servicio, particularmente relacionada con la lectura en escuelas multi-grado y bilingües El MINEDU y las regiones priorizadas incorporan metodologías innovadoras y buenas prácticas relacionadas con la enseñanza y la capacitación en servicio (a) Nivel de avance en el desarrollo de programas para la capacitación en servicio en regiones priorizadas Cuadros de al menos 10 profesores-capacitadores de nuevos maestros son entrenados en enseñanza de lectura en cada región priorizada (IE) Número de administradores y funcionarios capacitados (incluye capacitadores y acompañantes) 106 Al menos una red de profesores enfocados en enseñanza de lectura en cada provincia donde se implemente la capacitación en servicio Tecnologías de información y comunicación se aplican en la capacitación de maestros en servicio con estrategias para mejora de las habilidades de lectura (d) R2.4 Proporcionar asistencia técnica para mejorar el desempeño docente Apoyo al gobierno en el desarrollo, validación e implementación de sistemas de evaluación docente (estándares, indicadores, metodologías e instrumentos) en regiones priorizadas y/o en el MINEDU (e) Asistencia técnica y capacitación en regiones priorizadas para apoyar la implementación del sistema de evaluación docente, con apoyo para la implementación local ( e) R3 Implementación de buenas prácticas R3.1 Evaluar y apoyar la réplica de programas innovadores que contribuyen con los logros de aprendizaje Al menos 400 autoridades/funcionarios de educación capacitados para evaluar e implementar buenas prácticas en gestión educativa (IE) Número de administradores y funcionarios capacitados (incluye capacitadores y acompañantes) Al menos un proyecto de demostración (grupo de escuelas o proyecto de capacitación) en cada región para la demostración de buenas prácticas (al primer año) Al menos 10% de profesores de escuelas multi-grado en 4 regiones priorizadas so capacitados en estas metodologías (al segundo año del proyecto) con recursos regionales (IE) Número de maestros capacitados con apoyo del proyecto Al menos 6% de mejora en comunicación y 4% en matemáticas en los estudiantes de las zonas asistidas (al final del contrato) Porcentaje de estudiantes en escuelas beneficiarias de PIP que alcanzan niveles de excelencia en comunicaciones y matemáticas R3.2 Apoyar la creación de alianzas públicas y privadas Número de alianzas con organizaciones públicas y privadas formadas para apoyar metodologías basadas en evidencia (AprenDes o CETT) para fortalecer la calidad de la educación en comunidades de menores recursos (primeros dos años del proyecto) Número de escuelas con programas para mejorar la calidad de la educación apoyadas por el Partnership Challenge Fund Monto de recursos (al menos $8 millones) generado de proyectos públicos o privados para mejorar la calidad educativa (no infraestructura ni equipamiento). ($1 millón de contribuciones privadas; 60% en los primeros dos años del proyecto) Monto de recurso privados o públicos gestionados a través del PCF Número de personas (al menos 15) incluyendo actores regionales capacitados para desarrollar o implementar PIPs y apoyar las actividades de PELA (IE) Número de administradores y funcionarios capacitados (incluye capacitadores y acompañantes) Número de programas de mejora educativa expandidos o implementados a través del “Partnership Challenge Fund” (PFC) (f) 107 Número de organizaciones educativas locales, incluyendo beneficiarios del PCF, reciben asistencia técnica en gestión y administración del proyecto (o número de personas capacitadas en gestión y administración de la educación) Implementar programa de donaciones que contribuyen a las prioridades temáticas del PCF R4 Metodología de Escuelas Activas se implementa en escuelas en zonas de desarrollo alternativo (DA) Al menos 135 escuelas rurales multi-grado apoyadas y al menos 240 profesores capacitados en comunidades DA con la metodología de Escuelas Activas Número de escuelas con programas de mejora de la calidad educativa implementados directamente por el proyecto Involucramiento de la comunidad y participación de los padres al 100% en 135 escuelas Porcentaje de consejos estudiantiles que diseñan, implementan y monitorean planes de acción en apoyo de la escuela y la comunidad Al menos de 2% de mejora en los puntajes de comunicación y matemáticas (al 2010), basados en la metodología existente Porcentaje de estudiantes que sobresalen en las áreas de comunicación integrada y lógica matemática R5 Mejora de los diálogos de política, comunicaciones y participación de la sociedad civil R5.1 Apoyo para mejora el dialogo de políticas en aspectos críticos de la calidad educativa COPAREs y COPALEs (u otras formas de sociedad civil organizada) promueven la participación en diálogos de política y hacen recomendaciones de política educativa (g) Número de COPAREs y COPALEs que participan en diálogos de política y hacen recomendaciones, en las regiones del proyecto Regiones prioritarias comunican políticas e iniciativas educativas y resultados a la población (a) Reuniones organizadas y desarrollados con metas claras, conclusiones y acciones en apoyo a los objetivos del proyecto Por lo menos 50 organizaciones de sociedad civil participan en diálogos de política y toma de decisiones en el nivel regional y local Número de organizaciones de sociedad civil que participan en diálogos de política y toma de decisiones en el nivel nacional, regional y local R5.2 Desarrollo e implementación de una estrategia de comunicación Una estrategia de comunicación desarrollada y actualizada anualmente con enfoque costo efectivo para mejorar la comunicación del proyecto Numero de autoridades y líderes de sociedad civil, nacional y regional, que reciben periódicamente información del proyecto, publicaciones y comunicaciones o asisten a los eventos del proyecto o de instituciones alineadas al proyecto Información sobre calidad de la enseñanza de lectura, habilidades de aprendizaje en lectura, políticas educativas, financiamiento y descentralización compartida con funcionarios y autoridades Regiones prioritarias han desarrollado formas para comunicar políticas e iniciativas educativas y resultados a la población (c) 5 regiones según contrato original, el número se fue reduciendo (b) 30 escuelas según contrato original (d) Estrategia e indicador se eliminan (e) El requisito se elimina al final y no se reportan indicadores (f) 2 según contrato original (g) 5 COPARE y 15 COPALE 108 Annex 5: Gender approach in the SUMA Program El Programa SUMA incluye en su contrato el enfoque de equidad de género, el cual precisa que deben plantearse dos cuestiones relevantes: la relación de género con el logro de resultados sostenibles y la relación de las actividades del Programa con el status de niñas y niños. La Política de Género de USAID enfatiza la necesidad de dar coherencia a las iniciativas de cambio que USAID pudiera emprender o apoyar. Fija la atención en los varones y mujeres, sujetos o actores del cambio, las personas con discapacidad, de distintas religiones o ámbitos geográficos y niveles socio económico. Reconoce la existencia de brechas de género y señala la necesidad de desarrollar con creatividad la tecnología para colocarla al servicio de varones y mujeres y disminuir así las brechas existentes. Género en el diseño del programa SUMA El Marco de Políticas de Género de USAID es del año 2010 y el contrato del Programa SUMA es del 2009, por lo tanto no podríamos evaluar en este programa la aplicación de la política de género. Lo que se analizará es de qué manera SUMA consideró el enfoque de género en el diseño del Programa y en su implementación, y algunas observaciones sobre pistas o potencialidades que pudieron ser aprovechadas en ambos casos. En la página 10 del contrato de SUMA, y en los acápites sobre los beneficiarios y los indicadores propuestos, hay referencias específicas al enfoque de equidad de género “Para aumentar la equidad, el contratista debería fomentar eficazmente la equidad de género en todas las actividades del programa. Si bien el último informe de Educación para Todos señala que los asuntos de género en la educación en el Perú han sido tratados ampliamente, el contratista debe continuar atento a cuestiones de género al implementar el programa, sobre todo en lo que concierne a la asistencia técnica y a la capacitación, y al recomendar enfoques de capacitación docente que tomen en cuenta factores de género y culturales. Todavía es posible que las niñas de zonas rurales y de otras zonas desfavorecidas se beneficien de modo desproporcionadamente. El contratista debe hacerse como mínimo dos preguntas:  cómo afectan las relaciones de género el logro de resultados sostenibles,  cómo afectarán las actividades propuestas el status relativo de las niñas/mujeres y de los niños/varones. El contratista debe identificar claramente qué estrategias y enfoques seguirá para asegurarse que las cuestiones de género sean centrales en todas las actividades realizadas bajo el contrato. Todos los resultados e indicadores cuando resulte apropiado, deben estar desagregados por sexo.” Respecto a los beneficiarios del Programa, en la página 5 del contrato se indica que “son los estudiantes del sistema de educación pública de zonas desfavorecidas. Se dará especial énfasis a los más excluidos como son las niñas, las personas con alguna discapacidad y los estudiantes de zonas rurales donde muchas escuelas son, unitaria o multigrado”. Los indicadores de interés para USAID son los del Marco de Asistencia, o “indicadores estándares”, que en este caso se aplican para monitorear el programa de educación. El contratista es responsable de recoger los datos que muestren los resultados bajo cada uno de estos indicadores:  Número de administradores y funcionarios capacitados (desagregados por género).  Número de estudiantes matriculados en escuelas primarias o en escenarios equivalentes, apoyados por el gobierno estadounidense (disgregados por género) 109  Número de docentes/educadores capacitados con apoyo del gobierno estadounidense (desagregados por género). Pistas que pudieron ser aprovechadas para la consideración del enfoque de género en el diseño del Programa En la página 2 del contrato se señala que el objetivo del programa de educación es “ayudar al Perú a mejorar la calidad educativa, por ejemplo las tasas de conclusión de educación primaria en las zonas rurales y el porcentaje de estudiantes que logra los estándares para su grado, en comunicación integral y lógico matemática, a la vez que alienta al gobierno Peruano y al sector privado a aumentar la inversión en el sector social. El contratista intentará movilizar los considerables recursos humanos y financieros del Perú, sobre todo a la luz del fuerte y reciente crecimiento económico, para hacer que el sector educativo sea más equitativo, eficaz, eficiente y que pueda acoger un adecuado respaldo financiero de países extranjeros.” En este párrafo existen pistas que daban pie para incluir el enfoque de igualdad de género en las estrategias del Programa:  Aludiendo a las tasas de conclusión de educación primaria en las zonas rurales, los datos refieren que existen diferencias entre las niñas y los niños de las zonas rurales.  La noción de equidad a la que se aspira por parte del sector educativo, así como la eficacia y eficiencia, tendrían que haber considerado la problemática particular de las niñas y adolescentes de las zonas rurales, a partir de lo cual se pudieron haber generado actividades específicas para atender sus demandas, necesidades e intereses.  El diseño pudo haber considerado la formación del equipo técnico de SUMA en la elaboración de estrategias específicas con enfoque de género y su inclusión en los contenidos curriculares, en los materiales, y en la capacitación.  En los indicadores, la mención de desagregar por género alude a que se considere a mujeres y varones. Más allá de la cuestión numérica, podría haberse considerado la particular situación de las niñas, adolescentes y mujeres jóvenes rurales, sus necesidades, intereses y expectativas. Género en la implementación del programa SUMA Este acápite presenta el análisis de género en la implementación del programa SUMA, revisado a partir de lo reportado en los documentos de monitoreo. Lo único que se ha encontrado en la documentación son los resultados e indicadores, desagregados por sexo en las actividades principales del Programa. Los reportes trimestrales de SUMA no consideran información sobre las preguntas centrales del enfoque de género, porque no fueron consideradas en la implementación del programa. Según el informe de un integrante del equipo, al inicio del programa aplicaron una prueba a niños y niñas en la región San Martin y los resultados fueron similares en ambos grupos, con lo cual dedujeron que no era necesario aplicar un enfoque de Género. De tal manera que no incluyeron estrategias para considerar con mayor énfasis el enfoque de género en las actividades. 110 Indicadores de Género reportados en el Monitoreo Indicadores julio 2011 año 2 trimestre 4 setiembre 2013 varones mujeres varones mujeres varones mujeres Numero de administradores y funcionarios capacitados 165 93 84 36 Numero de formadores/acompañantes/ facilitadores capacitados 68 95 121 143 Numero de administradores y funcionarios capacitados en temas de monitoreo y evaluación 58 83 20 26 Número de participantes en actividades del programa Nacional 5 2 Número de participantes en actividades del programa Regional 330 295 Número de facilitadores 110 140 Número de docentes capacitados 32 31 Número de estudiantes de las IIEE de EBR 683 626 Conclusión Existe un documento oficial de USAID sobre el marco de políticas de género, donde claramente se sustenta la razón por la cual es necesario considerar tanto a las mujeres y los varones como agentes de cambio, y por lo tanto es necesario trabajar con estrategias, recursos y materiales adecuados y pertinentes para ello. El programa SUMA que se inicia en el 2009, solo tuvo en cuenta el número de varones y número de mujeres participantes en las diferentes actividades del proyecto. Por decisión institucional no aplicaron enfoque de género y estrategias específicas. Sin embargo, la evaluación censal de los estudiantes de segundo grado (umc.minedu.gob.pe) muestra que más niñas que niños alcanzan una capacidad lectora satisfactoria (35% de niñas y 31% de niños en el 2013), y que la mejora entre el 2012 y el 2013 fue mayor entre las niñas. Lo contrario ocurre cuando se mide nivel de logro en matemáticas (15% de niñas y 19% de niños logran nivel satisfactorio), lo que sugiere que habría una diferencia en el modo de aprender (o enseñar a niños y niñas) que debería ser tomado en cuenta para futuros programas. Recomendaciones USAID podría propiciar al inicio de sus programas y proyectos, una discusión interna sobre la necesidad de considerar a las mujeres como agentes de cambio. Para que ello ocurra no es suficiente promover que el número de mujeres participantes sea similar al de los varones, sino además impulsar su posicionamiento con estrategias específicas de participación y creación de espacios de inter-aprendizaje. Las brechas de género aún existen, los objetivos de desarrollo y cambio social no pueden ser alcanzados sino se toman en cuenta las diferencias existentes. Los programas que USAID implemente requiere que sus integrantes reciban formación sobre género, y conozcan la incorporación de indicadores con enfoque de género, y estrategias de género en la planificación, implementación, monitoreo y evaluación; sobre lo cual ya existe bibliografía calificada y experiencias concretas con resultados U.S. Agency for International Development Av. La Encalada s/n, Santiago de Surco Lima, Peru