DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU EVALUATION REPORT Reading and Access Evaluation Contract READING AND ACCESS EVALUATION REPORT FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU Submitted July 1, 2019, revised May 22, 2020 Prepared under Contract No.: AID-OAA-M-13-00010 Submitted to: Ben Sylla, USAID E3/ED Marisol E. Pérez, USAID/Peru Martín Castro, USAID/Peru Prepared by: Betty Alvarado de Lozán Ingrid Rojas Arellano Katharine Mark Carla Martínez Submitted by: NORC at the University of Chicago DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. I | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV TABLE OF CONTENTS Tables .............................................................................................................................................................. ii Figures............................................................................................................................................................ iii Acronyms...................................................................................................................................................... iv ABSTRACT .................................................................................................................................................. vi EXECUTIVE SUMMARY ..........................................................................................................................vii 1. BACKGROUND ..................................................................................................................................... 1 2. EVALUATION PURPOSE AND EVALUATION QUESTIONS .................................................. 6 3. EVALUATION METHODOLOGY AND LIMITATIONS ............................................................ 8 4. FINDINGS AND CONCLUSIONS .................................................................................................. 13 5. RECOMMENDATIONS ...................................................................................................................... 50 ANNEX I. STATEMENT OF WORK OF THE FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE........................................................................................................................................... 55 ANNEX II. DATA COLLECTION INSTRUMENTS ........................................................................ 68 ANNEX III SCHEDULE OF KEY INFORMANT INTERVIEWS, FOCUS GROUPS AND CLASSROOM VISITS ................................................................................................................................ 96 ANNEX IV. DOCUMENTS CONSULTED ........................................................................................ 98 ANNEX V. KEY TEAM MEMBERS ..................................................................................................... 101 ANNEX VI. DISCLOSURE OF ANY CONFLICTS OF INTEREST ........................................... 103 USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | II TABLES Table 1. Summary of Evaluation Question 1.1 ......................................................................................... 16 Table 2. Number of schools by type and region...................................................................................... 17 Table 3. Population characteristics, by region and main cities ............................................................. 18 Table 4. Number and percentage of polidocente completo and polidocente multigrado schools in rural settings, by treatment and region ...................................................................................................... 19 Table 5. Number of students per teacher in polidocente completo, polidocente multigrado, and unidocente multigrado schools, per treatment and region ...................................................................... 19 Table 6. Percentage of polidocente completo, polidocente multigrado, and unidocente multigrado schools in communities where Spanish is not the main language, by treatment and region ........ 20 Table 7. Summary of Evaluation Question 1.2 ......................................................................................... 21 Table 8. Summary of Evaluation Question 1.3 ......................................................................................... 24 Table 9. Summary of Evaluation Question 1.4 ......................................................................................... 27 Table 10. Summary of Evaluation Question 2.1 ....................................................................................... 29 Table 11. Three-year budget estimate of the Capacity Building Component in Amazonas (sample of polidocente completo and polidocente multigrado schools) ................................................... 31 Table 12. Three-year budget estimate of the Capacity Building Component in Loreto (sample of polidocente completo schools) ........................................................................................................................ 32 Table 13. Three-year budget estimate of the Capacity Building Component in Loreto (sample of polidocente multigrado schools) ...................................................................................................................... 33 Table 14. Three-year budget estimate of the Capacity Building Component in Madre de Dios (sample of polidocente completo and polidocente multigrado schools) ................................................... 34 Table 15. Summary of Evaluation Question 2.2 ....................................................................................... 35 Table 16. Summary of Evaluation Question 2.3 ....................................................................................... 37 Table 17. Summary of Evaluation Question 2.4....................................................................................... 39 Table 18. Summary of Evaluation Question 3.1 ....................................................................................... 41 Table 19. Cost-effectiveness analysis comparing Amazonía Lee and Soporte Pedagógico in polidocente completo Schools .......................................................................................................................... 43 Table 20. Cost effectiveness results comparing Amazonía Lee in polidocente completo and polidocente multigrado schools ....................................................................................................................... 44 Table 21. Summary of Evaluation Question 3.2 ....................................................................................... 45 Table 22. Summary of Evaluation Question 3.3 ....................................................................................... 46 Table 23. Summary of Evaluation Question 3.4 ....................................................................................... 47 Table 24. Summary of Evaluation Question 3.5 ....................................................................................... 48 Table 25. Summary of Evaluation Question 3.6 ....................................................................................... 49 III | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV FIGURES Figure 1. Logic Model of the Amazonía Lee Project .................................................................................... 3 Figure 2. Number of workshops attended by teachers (as a percentage of respondents) ........... 26 Figure 3. Average Score on ECE Reading Comprehension Test and Net Gain in Scores on ECE 2018 in polidocente completo Schools by Region and Program ............................................................. 44 USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | IV ACRONYMS AL Amazonía Lee, Amazon Reads AprenDes Innovaciones en Descentralización y Escuelas Activas, Innovations in Decentralization and Active Schools CAS Contrato por Administración de Servicio, Service Administration Contract CDE Capacity Development and Engagement CE Cost effectiveness CIAM Comunidades Interamazonicas, Inter-Amazonian Communities CNE Consejo Nacional de Educación, National Council of Education DDL Development Data Library DIGEIBIR Dirección General de Educación Intercultural, Bilingüe y Rural, General Directorate of Intercultural Education DRE Dirección Regional de Educación, Regional Education Directorate ECE Evaluación Censal de Estudiantes, Students’ Census Evaluation ESCALE Estadística de la Calidad Educativa, Statistics of Quality of Education EQ Evaluation Question FGD Focus Group Discussion FONDEP Fondo Nacional de Desarrollo de la Educación Peruana, Peruvian National Education Development Fund GIA Grupos de Inter Aprendizaje, Study Groups GORESAM Gobierno Regional de San Martín, Regional Government of San Martín GOREU Gobierno Regional de Ucayali, Regional Government of Ucayali G2G Government to Government KII Key Informant Interview LAC Latin America and the Caribbean LANN Liderando los Aprendizajes de Niñas y Niños, Leading Children’s Learning MEF Ministry of Economy and Finance MINEDU Ministry of Education V | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV NGO Non-Governmental Organization PEEL Programa Enseñar Es Liderar, Teaching is Leading Program PELA Programa Educativo de Logros de Aprendizaje, Learning Achievements Education Program SIAF Sistema Integrado de Administración Financiera, Integrated System of Financial Management SIAGIE Sistema de Información de Apoyo a la Gestión de la Institución Educativa, Information System for the Support of the Management of Schools SP Soporte Pedagógico, Pedagogical Support SUMA Apoyo a la Reforma de la Educación Básica de Calidad, Support for the Reform of Quality Basic Education UGEL Unidad de Gestión Educativa Local, Local Education Management Unit UPCH Universidad Peruana Cayetano Heredia UNICEF United Nations Children's Fund USAID United States Agency for International Development USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | VI ABSTRACT The purpose of this performance evaluation is to assess the applicability, replicability, and sustainability of key components of the Amazonía Lee project in Peru. Amazonía Lee was designed to improve early grade reading performance and address the learning gaps of two Amazonian regions in Peru. The project was implemented by the Regional Education Directorates (Direcciones Regionales de Educacion, DREs) of San Martín and Ucayali starting in 2015 with technical support from the Universidad Peruana Cayetano Heredia (UPCH) and with funding from the United States Agency for International Development (USAID). The performance evaluation sought to answer the following evaluation questions, each with several sub-questions: 1) How applicable is Amazonía Lee to other Amazonian regions of Peru? 2) How can applicable components of Amazonía Lee be replicated in other Amazonian regions of Peru? 3) To what extent are the applicable and/or key components of Amazonía Lee sustainable? The Evaluation Team reviewed documents, analyzed quantitative data, and conducted Key Informant Interviews, Focus Group Discussions, and school visits, including classroom observation and rapid teacher surveys. Amazonía Lee’s outcomes and costs are compared with those of a similar project, Soporte Pedagógico, implemented by the Ministry of Education from 2014 to 2018. Soporte Pedagógico is more cost-effective than Amazonía Lee in schools in Ucayali, and both projects are similarly cost-effective in San Martín, based on the costs per student and student performance on the Students’ Census Evaluation (Evaluación Censal de Estudiantes, ECE). However, some of the elements and approaches of Amazonía Lee are applicable to other Amazonian regions. In particular, the findings indicate the value of incorporating phonetic awareness when teaching children to read, especially in remote or poor areas, and the importance of having sufficient, qualified coaches and contextualized educational materials when replicating the project. VII | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV EXECUTIVE SUMMARY PURPOSE The purpose of the Amazonía Lee final Performance Evaluation is to assess the applicability, replicability, and sustainability of key components of Amazonía Lee to inform future education programming in Peru and other countries with similar education contexts. This final performance evaluation complements the Amazonía Lee’s Impact Evaluation, which examined effects on teaching practices and literacy skills of participating students. BACKGROUND Amazonía Lee was designed to improve early grade reading performance and reduce the learning gaps of two Amazonian regions in Peru: San Martín and Ucayali. The project was implemented by the Regional Education Directorates (Direcciones Regionales de Educacion, DREs) of San Martín and Ucayali starting in 2015, with funding from USAID. Universidad Peruana Cayetano Heredia (UPCH) provided technical assistance to the regional governments of San Martín and Ucayali under an activity called “Desarrollo de Capacidades y Compromiso” (Capacity Development and Engagement, CDE). In San Martín, Amazonía Lee is known as “Teaching is Leading” (Programa Enseñar es Liderar, PEEL); in Ucayali, it is entitled “Leading Children’s Learning” (Liderando los Aprendizajes de Niñas y Niños, LANN). Amazonía Lee was structured in four components: Capacity Building, Community Engagement, Regional Assessment System, and Regional Teacher Incentive System. The first component has the following sub-components: workshops for teachers, coaching of teachers, study groups (Grupos de Inter Aprendizaje, GIAs), appropriate educational material, and training for principals and regional staff. Amazonía Lee used what was termed, a “balanced approach”, which introduces “phonetic awareness” during the early years of primary education, and combines it with the Ministry of Education (MINEDU)’s “whole language approach”. The phonetic approach focuses on the words and decoding parts of the words, using sounds and the combination of them to construct words. This differs from the whole language approach, under which texts are perceived as a whole and not as isolated words. This approach does not focus on word decoding because the text – and not the word – is the minimum element. MINEDU implemented a project called Pedagogical Support (Soporte Pedagógico) between 2014 and 2017. The main components are workshops and in-classroom coaching supporting pedagogical practices from 1st to 6th grades, and study groups for teachers and students. Additionally, Soporte Pedagógico distributed teaching guides to teachers and other printed materials to parents, and included activities to strengthen parents and community involvement. Its similarity to Amazonía Lee permits it to serve as a comparison in some aspects of the evaluation. Amazonía Lee was implemented in two types of schools: schools with at least six teachers and a minimum of one teacher per classroom, known as polidocente completo, and schools with between two and five teachers, with at least two grades are grouped in one classroom, or polidocente multigrado. On the other hand, Soporte Pedagógico prioritized polidocente completo schools only. Neither project was introduced in the third type of primary school, unidocente multigrado, where just one teacher teaches all grades in one classroom. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | VIII EVALUATION QUESTIONS This performance evaluation focuses on three key Evaluation Questions (EQs), each with several sub-questions. They are as follows: Evaluation Question 1 (EQ1): How applicable is Amazonía Lee to other Amazonian regions of Peru? 1.1 Which elements of Amazonía Lee are applicable to other Amazonian regions of Peru? Why? 1.2 How similar or different are the needs of teachers and learners in other Amazonian regions to the Amazonía Lee regions? 1.3 To what extent could a single Amazonía Lee strategy be developed for all Amazonian regions while remaining applicable to the needs of each region? Do certain Amazonía Lee elements lend themselves to a single strategy better than others? 1.4 How could Amazonía Lee, or key components of it, be packaged to present to the DREs? Evaluation Question 2 (EQ2): How can applicable components of Amazonía Lee be replicated in other Amazonian regions of Peru? 2.1 Are there certain key components that are more replicable than others? Why and why not? 2.2 Based on Peru’s decentralization policy, do the costs of the approach enable replicability? Are there certain elements of the model that should be prioritized? 2.3 What are the human resource needs? How can they be met given monetary and time constraints? 2.4 What steps are needed to operationalize the applicable components in other Amazonian regions? Evaluation Question 3 (EQ3): To what extent are the applicable and/or key components of Amazonía Lee sustainable? 3.1 What factors contribute to their sustainability? 3.2 What are the cost considerations? 3.3 If they are not sustainable, what steps would the MINEDU and DREs need to take to make them sustainable? 3.4 How can the DREs continue to update and develop relevant materials similar to those developed under the project? 3.5 What further steps could the MINEDU, DRE, and schools take in the future to ensure the approach stays relevant and responds to the needs of the region, teachers, and students? 3.6 What steps can the MINEDU take to ensure the DREs feel supported and/or empowered to adapt the general curriculum to incorporate applicable key components of Amazonía Lee? IX | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV METHODS The Evaluation Team used a mixed methods design, and relied on primary and secondary sources. The Evaluation Team conducted a thorough review of MINEDU’s relevant documents, and analyzed quantitative data from the Schools Census, as well as students’ performance indicators gathered through the Students’ Census Evaluation. In addition, the Evaluation Team collected cost data from financial statements and the Ministry of Economy and Finance’s online budgetary platform. The Evaluation Team also conducted extensive qualitative data collection, including Key Informant Interviews (KIIs), Focus Group Discussions (FGDs), and school visits for a total of: ● 19 KIIs in Lima, San Martín, and Ucayali (with representatives of USAID/Peru, UPCH, MINEDU, DREs, and Local Education Management Units (Unidades de Gestión Educativa Local, UGELs), as well as school principals and Amazonía Lee’s regional coordinators), ● 2 FGDs with coaches (one in San Martín, and one in Ucayali), and ● 4 school visits in San Martín and Ucayali (with 2 classroom observations, and rapid teacher surveys in each), which included one Amazonía Lee and one Soporte Pedagógico school in each region, to shed light on Amazonía Lee relative to alternative pedagogical projects. FINDINGS AND CONCLUSIONS APPLICABILITY ● The Amazonía Lee model has a comprehensive structure and a holistic philosophy that was conceptualized and implemented following a single strategy in the Amazonian context of San Martín and Ucayali regions (mainly urban and semi-urban locations with polidocente completo and polidocente multigrado schools). The strategy, besides introducing the phonetic approach, included activities promoting teachers’ capacity to teach children to read, such as workshops, school coaching visits, community participation, and pedagogical training for school principals and regional education staff. Some of the elements of the single strategy could be applicable to other Amazonian regions (namely, Amazonas, Loreto, and Madre de Dios), as follows:  The whole language approach combined with the phonetic approach is applicable in remote or poor areas that lack visible written messages and/or support from family.  Contextualized materials are applicable throughout the Amazonian region, although they were only tested in the Spanish language. It is important to consider this because Loreto and Ucayali have a higher percentage of polidocente completo schools in communities where Spanish is not the main language (more than 25%), while all regions but Amazonas have more than 30% of polidocente multigrado schools in similar communities. ● The implementation of the project and its comparison to the implementation of Soporte Pedagógico revealed that it is possible to insert flexibility in some elements to enable the use of the standard set of inputs while maintaining the effectiveness of the strategy and teaching approach of Amazonía Lee. Flexible strategies include: coaches with diverse contract relationship with the UGEL, variable number of teachers per coach (depending on the type USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | X and distance of schools), and adjusting the use of phonetics and whole language according to students’ needs. REPLICABILITY ● Teacher coaching is the key element of Amazonía Lee replicability as the salary of coaches represents more than 70 percent of the Capacity Building budget. It would be challenging to replicate Amazonía Lee without sufficient qualified coaches available and willing to conduct training in dispersed territories. ● Educational materials have been contextualized for all Amazonian regions, so they are ready for printing. ● There is no problem foreseen for the replication of a regional non-monetary incentive system for teachers because it does not require additional funding. However, time is needed for its roll-out. ● According to three-year budget estimates of the replication of the Capacity Building component:  Amazonas will require 47 coaches and 2.25% of additional primary education budget.  Loreto will require 25 coaches for the polidocente completo schools, and 37 coaches to support polidocente multigrado schools. An additional 0.86% and 1.03% of the recurrent budget for primary education is required to cover polidocente completo and polidocente multigrado schools, respectively.  Madre de Dios will require 6 coaches and an additional 1.81% of the recurrent budget for primary education. ● To implement the applicable components of Amazonía Lee in other Amazonian regions, it is necessary to conduct an analysis of the stakeholders and their ownership of the project, engagement with national policies, selection of a specialized operator to work with UGELs, prioritization of sub-components and activities of the Capacity Building component, budget analysis and increase, and a functionining administrative process in place. SUSTAINABILITY ● The Amazonía Lee model contributes to the strengthening of students’ competencies as identified in the National Curriculum, and the role of coach is already considered in the teachers’ merit system. Both of these factors contribute to the institutional sustainability of the model. ● Using costs per student and student performance data from the Students’ Census Evaluation to measure effectiveness, Soporte Pedagógico is more cost-effective than Amazonía Lee in polidocente completo schools in Ucayali, and both are similarly cost-effective in San Martín. In the same regions, polidocente multigrado schools are more costly per student than polidocente completo schools due to lower number of students in the former. ● MINEDU and the regional governments can advocate to include the role of coaches in recurrent education regional budget, while the regional governments can request budget for XI | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV full time civil servants in the role of pedagogical specialists in order to make the model sustainable. ● The DREs can print more contextualized workbooks and teacher guides that were developed under Amazonía Lee. Since not all UGELs have the technical capacity to develop new materials, the schools can look for local resources to complement MINEDU books. ● Schools can continue with teaching approaches that respond to the students’ needs and other strategies that complement Amazonía Lee. Additionally, the DRE directors can promote the relevancy of the approach in forums wherein other DREs from the Amazonian region participate. ● DREs and MINEDU can establish a dialogue on how Amazonía Lee’s novel practices can help to develop National Curriculum’s competencies, and MINEDU can establish the areas where Amazonía Lee elements can be inserted as strategies. RECOMMENDATIONS APPLICABILITY ● Consider the type of school (polidocente completo and polidocente multigrado), as well as the location and dispersion of the schools (urban or rural) to determine applicability. ● Maintain the main structure of the model and approach, such as the contents and sequence of activities, but develop different operational arrangements to implement the components. REPLICABILITY ● Prioritize the number and type of schools that the regions are able to manage and finance with the regional budget. ● Select schools based on cost effectiveness and efficiency. ● Look for technical support for the DREs and UGELs in the new regions. ● Enrich the Capacity Building Component with strategies that can improve affordability of the project. ● Depending on the size and location of the schools, explore the option for UGELs to work directly with school principals. ● Prioritize the availability of the workbooks and the teacher’s guides online. SUSTAINABILITY ● Consider incorporating a teaching approach that promotes phonetic awareness in children in the current MINEDU policy to improve teaching processes of the National Curriculum. ● Advocate for the financing of coaches as part of the teachers’ tenure track (Carrera Pública Magisterial). USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | XII ● Continue the technical discussion at the national and regional levels on the methods of teaching children to read and write. ● Highlight how the participation of regional education staff in the design and adaptation of the workbooks created ownership. 1 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV 1. BACKGROUND The Amazonía Lee project is part of the USAID/LAC Reads effort to strengthen reading capabilities in Latin America and the Caribbean by generating and using information on successful approaches to improving early grade reading. In-line with this goal, Amazonía Lee was designed to improve early grade reading performance and address the learning gaps of two Amazonian regions in Peru: San Martín and Ucayali. The Amazonía Lee education project consisted of two Government to Government (G2G) agreements and a cooperative agreement with the UPCH to support the work of the two G2Gs. The G2Gs were signed with the regional governments of San Martín (Gobierno Regional de San Martín, GORESAM) and Ucayali (Gobierno Regional de Ucayali, GOREU). In San Martín, the project is known as “Teaching is Leading” (Programa Enseñar es Liderar, PEEL); in Ucayali, it is called “Leading Children’s Learning” (Liderando los Aprendizajes de Niñas y Niños, LANN). Amazonía Lee was targeted to children in primary school in 1st to 3rd grades. The project was implemented by the Regional Education Directorates (Direcciones Regionales de Educación, DREs) from 2015 to 2017 with funding from USAID. Both regions shared the same technical assistance provider, UPCH. USAID also commissioned an Impact Evaluation of the two G2Gs. Amazonía Lee was implemented in two types of schools, schools with at least six teachers and a minimum of one teacher per classroom, known as polidocente completo, and schools with between 2 and 5 teachers, with at least two grades are grouped in one classroom, or polidocente multigrado. It was not introduced in the third type of primary school unidocente multigrado, where just one teacher teaches all grades in one classroom. Amazonía Lee was structured in 4 components: ● Capacity Building, with the following sub-components:  Workshops for teachers  Coaching of teachers  Study groups (Grupos de Inter Aprendizaje, GIAs)  Appropriate educational material, and  Training for principals and regional staff ● Community Engagement  Book fairs and promotional kiosks in regional trade fairs  Family workshops at school  Family storytelling ● Regional Assessment System  Data collection and processing of student performance at the beginning and end of school year ● Regional Teacher Incentive System USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 2  Non-monetary incentives like performance acknowledgment with certificates and ceremonies. It was mentioned that teachers received laptops but it was not a common practice. PROJECT LOGIC AND IMPACT Figure 1 presents a logic model for the Amazonía Lee project, summarizing the main outputs of the intervention and the expected intermediate and final outcomes. The first column depicts the project’s components/sub-components. UPCH provided a training project for Education Specialists (“formadores” or coaches), who then conducted in-classroom coaching for teachers through individual demonstration and observation sessions. Training workshops and study groups were also carried out to strengthen participant teachers’ pedagogical skills and instructional capacity. Appropriate curriculum and varied instructional materials were developed to bolster effective learning. School principals and regional governments received specialized technical assistance to improve institutional planning, budgeting and administrative leadership. Strategies such as community workshops, reading assessments, and an incentive system for teachers were designed to motivate all stakeholders of the education sector to engage in the efforts of improving early grade students’ reading skills. The components and sub-components were designed to lead to a sequence of intermediate outcomes, including better teaching skills, improved educational materials, and enhanced school leadership, as well as increased external awareness and support. Jointly these improvements in teaching and the learning environment contribute to the final outcome of improved foundational reading skills (reading comprehension, phonological awareness, phonics fluency, vocabulary and oral/written expression). Amazonía Lee’s Impact Evaluation1 revealed positive impacts on teachers and students. For example, it found that the intervention enhanced teacher instructional practices. Moreover, Amazonía Lee significantly improved students’ reading outcomes, compared to students in schools that had not received Amazonía Lee or Soporte Pedagógico en Ucayali. However, in San Martín, the reading outcomes of students in Amazonía Lee schools was not statistically different from those of Soporte Pedagógico schools. Given that most control schools benefitted from Soporte Pedagógico in San Martín, the impact evaluation was unable to estimate a cleaner treatment-control difference in this region. 1 USAID (2018). Evaluation of Amazonía Lee Reading Intervention in Peru: Final Report. Mathematica Policy Research. 3 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Figure 1. Logic Model of the Amazonía Lee Project AMAZONÍA LEE’S TEACHING APPROACH Amazonía Lee used the “balanced approach”, which employs a mixture of two approaches to teach children to read and write. The balanced approach introduces the “phonetic awareness” during the early years of primary education, and combines it with MINEDU’s “whole language approach,” also known as "enfoque comunicativo textual”). The phonetic approach focuses on the words and decoding parts of the words, which allows students to link the letters with sounds instead of memorizing letters of the alphabet. In sum, this approach looks for the construction of words using sounds and the combination of them (Taylor 1973, cited by Ferrer 2007: 100)2 . On the other hand, the whole language approach perceives texts as a whole and not as isolated words. Therefore, this approach emphasizes the reading of stories and other text instruments, and do not focus in word decoding. Accordingly, the text and not the word is the minimum element. MINEDU also points out the need to consider the culture and previous experiences of 2 Ferrer (2007) Enseñanza de la lectura y escritura. Edited by UPCH and Centro Andino, USAID Lima. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 4 children, and to use different types of texts (such as comics, short stories and letters) for the purpose of communicating a message3 . The balanced approach used by USAID considers that there are advantages and disadvantages in both approaches. In the past, both the Andean Center for Excellence in Teacher Training (CETT - Andino, or Centro Andino, implemented in urban and big rural polidocente completo schools) and Innovations in Decentralization and Active Schools (Innovaciones en Descentralización y Escuelas Activas, AprenDes, implemented in rural polidocente multigrado and unidocente multigrado schools) USAID projects were able to use the phonetic awareness and reconcile it with the whole language approach.4 Amazonía Lee also employed the balanced approach, making it different from other in-service teacher training support programs in Peru that are not supported by USAID. EDUCATIONAL CONTEXT Amazonía Lee was implemented in two Amazonian regions: San Martín and Ucayali. In 2014, students in San Martín and Ucayali earned lower scores than the national average on the Students’ Census Evaluation (Evaluación Censal de Estudiantes, ECE) Reading Comprehension Exam, which annually evaluates reading comprehension levels of second and fourth grade students in public and private schools across Peru. The average scores of second grade students on the reading comprehension exam in San Martín and Ucayali were 5505 and 5166 , respectively, in comparison to the national average of 568. Across all of Peru, 43.5% of second graders earned a “Satisfactory” score on the 2014 ECE reading comprehension exam. In San Martín and Ucayali, 35.7% and 21.8% of second grade students earned a satisfactory score, respectively. In both San Martín and Ucayali, students in urban areas received higher scores than students in rural areas. In San Martín, 17.1% of rural students earned a satisfactory score, while 46.6% of urban students did. Only 10.0% of students in rural areas of Ucayali received a satisfactory score, versus 24.0% of their urban counterparts: A year before Amazonía Lee was implemented, in 2014, MINEDU launched a similar project, Soporte Pedagógico (Pedagogical Support) in urban polidocente completo schools in 10 regions7 . By 2016, it was operating in 18 regions, including San Martín and Ucayali8 . The project was implemented until 2017. 3 MINEDU. “Qué y cómo aprenden nuestros estudiantes. Área Curricular, Comunicación III ciclo, 1ro y 2do grado de Educación Primaria.” In Rutas del Aprendizaje, edition 2015. 4 USAID. “Evaluación de los Programas en Educación de la USAID/Peru: AprenDes y Centro Andino. Summary Evaluation Report.” April 2010. 5 Unidad de Medición de La Calidad Educativa del MINEDU. “¿Cuánto Aprenden Nuestros Niños En Las Competencias Evaluadas? San Martín: Resultados De La Evaluación Censal De Estudiantes 2014 – ECE 2014, 2do. Grado De Primaria.” 2015. 6 Unidad de Medición de La Calidad Educativa del MINEDU. “¿Cuánto Aprenden Nuestros Niños En Las Competencias Evaluadas? Ucayali: Resultados De La Evaluación Censal De Estudiantes 2014 – ECE 2014, 2do. Grado De Primaria.” 2015. 7 Chinen, M., & Bonilla, J. (2017). “Evaluación de impacto del programa Soporte Pedagógico del Ministerio del Educación del Perú.” Grupo FORGE, GRADE, Global Affairs Canada. According to the authors, polidocente completo schools in urban areas represented 94.5% of educational institutions targeted by Soporte Pedagógico in 2015. Some polidocente multigrado schools were included by the MINEDU at the request of the regions (p. 16). 8 MINEDU Web Portal: www.minedu.gob.pe/soporte-pedagogico 5 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Soporte Pedagógico had four objectives: ● Promote the adequate pedagogical performance of the teacher so that he or she can favorably affect the development of the students' competences, ● Strengthen the pedagogical leadership practices of school management, ● Involve the parents so they support the learning process of their children, and ● Organize workshops and meetings with parents about the learning achieved by their children. The main components were workshops and in-classroom coaching supporting pedagogical practices from 1st to 6th grades, study groups for teachers and students, and online support. Additionally, Soporte Pedagógico provided tutoring in Mathematics and Communications targeted to students from 1st to 3rd grades led by a “Docente Fortaleza.” The project also included activities to strengthen the involvement of parents and the community, and paid special attention to the distribution of educational materials, teaching guides for teachers and other printed materials for parents. It is worth mentioning that MINEDU also supports coaching and group studies for teachers in bilingual and non-bilingual schools. These and other strategies are financed by the Learning Achievements Educational Program (Programa Educativo de Logros de Aprendizaje, PELA). USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 6 2. EVALUATION PURPOSE AND EVALUATION QUESTIONS According to the Statement of Work (SOW), the purpose of this final evaluation is to summarize and validate key findings to provide an evidence base for decision-makers and stakeholders to help inform future education programming, particularly in regions with similar teaching and learning challenges in Peru. The main objective of this Performance Evaluation is to assess the applicability, replicability, and sustainability of key components of Amazonía Lee. This final Performance Evaluation complements the Impact Evaluation, which examined effects on teaching practices and literacy skills of participating students. EVALUATION QUESTIONS The SOW for the Performance Evaluation focused on three key Evaluation Questions (EQs), each with several sub-questions, around applicability, replicability and sustainability of the Amazonía Lee project. They are as follows: Evaluation Question 1 (EQ1): How applicable is Amazonía Lee to other Amazonian regions of Peru? 1.1 Which elements of Amazonía Lee are applicable to other Amazonian regions of Peru? Why? 1.2 How similar or different are the needs of teachers and learners in other Amazonian regions to the Amazonía Lee regions? 1.3 To what extent could a single Amazonía Lee strategy be developed for all Amazonian regions while remaining applicable to the needs of each region? Do certain Amazonía Lee elements lend themselves to a single strategy better than others? 1.4 How could Amazonía Lee, or key components of it, be packaged to present to the DREs? Evaluation Question 2 (EQ2): How can applicable components of Amazonía Lee be replicated in other Amazonian regions of Peru? 2.1 Are there certain key components that are more replicable than others? Why and why not? 2.2 Based on Peru’s decentralization policy, do the costs of the approach enable replicability? Are there certain elements of the model that should be prioritized? 2.3 What are the human resource needs? How can they be met given monetary and time constraints? 2.4 What steps are needed to operationalize the applicable components in other Amazonian regions? Evaluation Question 3 (EQ3): To what extent are the applicable and/or key components of Amazonía Lee sustainable? 3.1 What factors contribute to their sustainability? 3.2 What are the cost considerations? 7 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV 3.3 If they are not sustainable, what steps would the MINEDU and DREs need to take to make them sustainable? 3.4 How can the DREs continue to update and develop relevant materials similar to those developed under the project? 3.5 What further steps could the MINEDU, DRE, and schools take in the future to ensure the approach stays relevant and responds to the needs of the region, teachers, and students? 3.6 What steps can the MINEDU take to ensure the DREs feel supported and/or empowered to adapt the general curriculum to incorporate applicable key components of Amazonía Lee? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 8 3. EVALUATION METHODOLOGY AND LIMITATIONS For the final Performance Evaluation of Amazonía Lee, the Evaluation Team used a mixed methods design, both to accommodate the nature and the variety of the EQs and to ensure that a sound evidence base is produced. The evaluation was based on a combination of qualitative data drawn from a series of Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) with the diverse stakeholders of the project; quantitative data sources including test scores and data from performance indicators; cost information; and a thorough review of reports, agreements, and other documents. To shed light on Amazonía Lee relative to alternative pedagogical project, the Evaluation Team visited schools supported by Amazonía Lee and by Soporte Pedagógico. The reason for including Soporte Pedagógico schools was to be able to ascertain how Amazonía Lee is implemented in comparison with similar projects that support the improvement of teaching and learning, using similar strategies. The variety of sources allowed the Evaluation Team to triangulate and substantiate findings. The design ensures the information generated is useful not only to USAID, but also to the government of Peru (specifically MINEDU) and DREs. SAMPLING STRATEGY AND SELECTION OF RESPONDENTS SELECTION OF INTERVIEWEES The sampling plan was divided into two parts, namely, high-level KIIs conducted in Lima, and KIIs and other activities conducted in San Martín and Ucayali. LIMA For the high-level KIIs in Lima, the Evaluation Team selected individuals who represent different stakeholder groups, including USAID staff, UPCH staff, and MINEDU staff. The preliminary list was based on USAID’s initial suggestions, and supplemented by the Evaluation Team to add interviewees that can provide specific information about topics like unitary costs. These KIIs were conducted during week 1 of field work, with additional interviews scheduled as needed in week 4 to reinforce the findings from the regions. The schedule of KIIs can be observed in Annex III. SAN MARTÍN AND UCAYALI The Evaluation Team employed a stratified sampling approach to ensure equal representation of the two regions, San Martín and Ucayali. This helped the Evaluation Team to understand the institutional dynamics given the different G2G agreements, as well as the way in which Amazonía Lee operated in different regional (geographical, cultural, political, institutional, economic) settings. The Evaluation Team interviewed representatives of the DREs and Local Education Management Units (Unidades de Gestión Educativa Local, UGELs), as well as regional coordinators, coaches, school principals and teachers. The Evaluation Team used KIIs, Rapid Surveys, classroom visits and FGDs, as can be observed in Annex III. SELECTION OF UGELS AND SCHOOLS To efficiently collect the wide range of information necessary to address the EQs and sub-EQs in the second and third week of the field period, the Evaluation Team followed a purposive sampling approach to select the UGELs and schools to visit during their second and third weeks 9 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV in the field. To this end, the Evaluation Team chose one UGEL in one urban center for each region: Moyobamba (where the offices of the Regional Government and DRE are located) in San Martín; and Pucallpa (Coronel Portillo) in Ucayali. The Evaluation Team conducted KIIs with the Directors of the DREs, former regional coordinators of Amazonía Lee, and pedagogical specialists at the UGELs. In each region, the Evaluation Team also visited two schools: one where Amazonía Lee had been implemented and another where MINEDU provided support through Soporte Pedagógico. In each of the four schools, the Evaluation Team carried out KIIs with the principals, observations of two classrooms, and rapid surveys of at least six teachers (of 1st, 2nd and 3rd grades). Finally, with the support of the Regional Project Coordinators, the Evaluation Team organized one FGD in each region with four coaches who provided information about their experiences. DATA SOURCES AND COLLECTION METHODS The Evaluation Team reviewed secondary data sources, including project documents and MINEDU’s relevant documents, and analyzed Amazonía Lee monitoring and evaluation data and performance indicators. In addition, the Evaluation Team conducted high level KIIs in Lima with central level education officials, USAID and Implementing Partners from UPCH; regional level KIIs and FGDs; and non-invasive classroom visits in selected sites in San Martín and Ucayali. In summary, the Evaluation Team conducted: ● 19 KIIs with representatives of USAID/Peru, UPCH, MINEDU, DREs, and UGELs, as well as school principals and Amazonía Lee’s regional coordinators, ● 2 FGDs with coaches, and ● 4 school visits (with 2 classroom observations, and rapid teacher surveys in each). DOCUMENT REVIEW The purpose of the systematic document review was to develop a good understanding of Amazonía Lee and context, and inform the design of the instruments and tools for field work. To this end, the Evaluation Team reviewed background documents on Amazonía Lee and other USAID documents, as well as other reports on Peru’s education sector development. Annex IV includes the full list of documents reviewed. QUANTITATIVE DATA The Evaluation Team reviewed general information of the education sector, such as the location and size of schools served by Amazonía Lee, and compared this information to the rest of the Amazonian regions that could potentially implement a similar project. This information came from MINEDU’s Statistics of Quality of Education (Estadística de la Calidad Educativa, ESCALE), which manages information coming from the Schools Census (Censo Escolar), DREs and UGELs. Additionally, the Evaluation Team compared information on reading comprehension exam results. The data was gathered by MINEDU’s Office of Measurement of Learning Quality (Oficina de Medición de la Calidad de los Aprendizajes) through the ECE. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 10 Moreover, the Evaluation Team collected cost data through a number of sources. This information came from financial statements, school visits and in-depth interviews. The Evaluation Team also downloaded data from the Ministry of Economy and Finance (MEF)’s Information System for the Support of the Management of Schools (Sistema de Información de Apoyo a la Gestión de la Institución Educativa, SIAF), a platform that presents budgetary information of the public sector. In addition, the Evaluation Team visited UPCH premises in Lima to review the printing quality and number of pages of Amazonía Lee materials for cost calculation purposes. All MINEDU’s materials for students were available in all classrooms and in PDF format online, which facilitated the construction of the comparative package. Finally, the Evaluation Team utilized Amazonía Lee’s monitoring data. Besides UPCH reports, the Evaluation Team gathered information from the regional coordinators to have a full understanding of how the project was implemented and outcomes achieved. HIGH-LEVEL KIIS IN LIMA KIIs were conducted in Lima with USAID, MINEDU, and UPCH. The Evaluation Team designed a protocol to guide the KIIs. High level officials’ input was crucial to understanding the extent of the possibility of applicability and replicability of Amazonía Lee to other regions as well as the extent to which sustainability can be achieved. The KIIs with USAID and UPCH focused on understanding Amazonía Lee’s distinctive characteristics, how it was implemented, enabling factors, successes achieved, and recommendations. In the case of MINEDU, the Evaluation Team explored the actual flexibility regional governments have to change or adjust curricula, especially if changes require new books and materials to be developed, in order to assess the replicability of Amazonía Lee. The instrument used for these KIIs can be found in Annex II. REGIONAL-LEVEL KIIS/FGDS KIIs in San Martín and Ucayali were also important to understand the perspective of local actors in the success of the Amazonía Lee activity and the inputs required for potential expansion. The Evaluation Team conducted interviews with officials of the Regional Governments, DREs and UGELs, as well as project coordinators. The KIIs were used to gather information about challenges and limitations faced by each interviewee during the implementation of Amazonía Lee, elements they find feasible to replicate considering their current capacity, and disaggregated cost data of inputs. In addition to the KIIs, the Evaluation Team conducted FGDs with the coaches. Their testimonies shed light over the whole process of hiring, training and coaching. The FGDs focused on how the coaches were trained, how they transferred this knowledge to the teachers, the extent to which this can be transferred to others, and how the methods and materials can be sustained. To complement the findings, the Evaluation Team requested operational information during the KIIs at UGELs. SCHOOL VISITS The Evaluation Team visited pre-selected schools during one school day.9 They conducted one KII with the Principal, visited two classrooms and also administered a rapid survey to all of 1st , 9 The development of any activity in schools has to be approved by the UGEL to avoid missing teaching hours. The Evaluation Team coordinated the visit with the pedagogical staff in San Martin DRE and the UGEL; in the case of Ucayali, the coordination was done with the Pedagogical Director in DREU. In both cases, they recommended which schools could be visited (one that had received Amazonía Lee support, and one that had received Soporte Pedagogico from MINEDU). 11 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV 2 nd, and 3rd grade teachers. Principals have been a key target population for Amazonía Lee, so their opinion on the applicability and replicability of the strategies is fundamental, especially in the application of the strategies of coaching, and the process to obtain books and other materials. The classroom visits and observations of the available materials were a crucial part of data collection and they complemented the information provided ahead by UPCH, MINEDU, DREs, UGELs and school principals. First, they clarified whether teachers who received support from Amazonía Lee were using the phonetic approach10 and Amazonía Lee materials. In that case, the Evaluation Team verified if the teachers were using the Amazonía Lee approach either alone or integrated into MINEDU’s “whole language” approach. Second, the proper identification of materials was necessary to compute the actual cost of the approach, independently of who sends or produces the materials. Finally, the identification of materials was useful to determine which materials could be financially sustained and potentially institutionalized in the future. The Evaluation Team communicated to the principal and the teachers that the visits did not have the goal of evaluating teachers’ performance, but rather of obtaining information from teachers about the utilization of the materials and who the users are (student, classroom, and teacher) for costing purposes. Through rapid surveys, the Evaluation Team also asked teachers a few questions about the project - for example, whether the teacher had participated in another project besides Amazonía Lee, the number of hours of training received, the number of hours of coaching and demonstration classes, and their relationship with the coaches. LIMITATIONS LIMITED KNOWLEDGE OF NEW STAFF IN KEY POSITIONS KIIs at MINEDU were affected by the change of Minister and Directors during the first week of fieldwork, but the Evaluation Team managed to conduct interviews that provided information about MINEDU’s perception of regional innovations. The Evaluation Team also had access to publicly available information, such as online videos. Regional Governments elections brought new authorities and staff that lacked the knowledge of the Amazonía Lee project and kindly declined to be interviewed in both regions.11 Nevertheless, interviews with the DRE’s Directors, Pedagogical Directors and UGEL Specialists in San Martín and Ucayali provided valid information about the project implementation. In terms of budget data of the Amazonía Lee project, the information was provided by the DRE in the case of San Martín and by USAID provided the information of Ucayali. LIMITED AVAILABILITY OF COACHES IN THE REGIONS Since the Amazonía Lee project ended coaching activities in 2017, some of the professionals moved out of the city or returned to their previous or new teaching posts. Thus, only four coaches in each region (of nine coaches registered in Moyobamba city [San Martín region] and 10 USAID, G2G in Education Case Study: Peru, May 2018. Box: Good Practice – “Getting to yes” with new approach for reading. Page 14 11 The interview in Ucayali region with the Social Manager was canceled because the staff was called to have a meeting with the Governor to analyze the regional government’s performance during his first 100 days in office. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 12 fourteen in Pucallpa city and surroundings [Ucayali region] participated in the FGDs. The small number, however, allowed more participatory sessions. LAST-MINUTE CHANGES TO THE RAPID SURVEYS’ TIMEFRAME AND SAMPLE During the visits to Amazonía Lee schools in San Martín and Ucayali, the Evaluation Team had to instruct the teachers that the questions referred to 2017 because Amazonía Lee distributed books but did not implement an in-service training activity in 2018. The rapid survey also had to be quickly adjusted during a visit to a Soporte Pedagógico school in Ucayali, since the school principal indicated that the project ended in 2016 and not in 2017 (as in San Martín). The Evaluation Team planned to administer just three surveys (one in 1st, one in 2nd, and one in 3 rd grade) in each Amazonía Lee and Soporte Pedagógico school, but the principals of all visited schools asked the Evaluation Team to survey all teachers in those grades. This explains why the Evaluation Team obtained a larger sample. The Soporte Pedagógico school visited in Ucayali served only 1st and 2nd grade in the morning and 3 rd grade in the afternoon due to the lack of infrastructure. Consequently, the survey was not administered to the 3rd grade teachers because the Evaluation Team could not assure that they would not receive information about the questions ahead of time since the principal received an example of the questionnaire during the morning visit. THE VISITS AFFECTED THE DEVELOPMENT OF CLASSES IN ONE AMAZONÍA LEE SCHOOL The principals and the teachers knew about the Evaluation Team’s visit, and they prepared classes showing the phonetic approach in one Amazonía Lee school. Even though the purpose of the visit was to check availability and use of materials without advance warning, the class presentation in that instance helped the Evaluation Team better understand the differences between the Amazonía Lee and MINEDU approaches. 13 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV 4. FINDINGS AND CONCLUSIONS EQ1: HOW APPLICABLE IS AMAZONÍA LEE TO OTHER AMAZONIAN REGIONS OF PERU? The review of the applicability of Amazonía Lee project implemented in San Martín and Ucayali regions was done with regard to three other Amazonian regions12: Amazonas13, Loreto, and Madre de Dios. The evaluation takes into account the type and location of schools in said regions. 1.1 WHICH ELEMENTS OF AMAZONÍA LEE ARE APPLICABLE TO OTHER AMAZONIAN REGIONS OF PERU? WHY? The implementation of Amazonía Lee in the regions of San Martín and Ucayali makes it possible to test and validate the elements of the components and subcomponents in a context of the Amazonian geography in polidocente completo and polidocente multigrado schools. The applicable elements are: ● The “balanced approach” used to teach children in 1st, 2nd, and 3rd grades to read and write. ● Characteristics, content and methodology of training modules for different stakeholders designed for coaches, teachers, school principals, and regional education specialists. ● Contextualized educational workbooks for the students of 1st, 2nd and 3rd grade and educational material that supports the content of the training practices. ● Strategies such as community engagement, incentive systems for teachers, and regional reading assessment. TEACHING APPROACH Informants commented that it is a challenge to promote only the whole language approach in the Amazonian regions and that the phonetic approach is more adequate. Family backgrounds and the lack of written messages in contexts like advertising, newspapers, and road signs cause children to discover the relation between letters and sounds for the first time at school. This absence of written messages outside of the school represents a limitation of the whole language approach in the Amazonian regions. Early phonetic awareness could solve this situation at least for first grade readers, or even at preschool. “But if you look at the Coast, the whole language approach works, because you find signs everywhere and ads everywhere. Everywhere… In other words, the child from a young age sees STOP, for example, and already from there he sees it and his mother tells him STOP and he already knows… But in the Jungle that does not happen, well… In the Jungle you only find signage of… that type of signs and notices in the capitals of the provinces. But in the districts, in the population centers you do not find anything. But the issues is that … they are different realities. That is, the phonetic helps a lot in the Jungle because of its reality.” Former DRE Director 12 This report uses the terms departments and regions interchangeably, referring to the geographic and political divisions of Peru. Regional governments refer to the administrative and political bodies. 13 According to the National Institute of Statistics, the Amazonas department has approximately 75% of its territory in the Amazonian region, and the rest is located in the Andes (Sierra) region. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 14 CAPACITY BUILDING COMPONENT: IMPLEMENTATION METHODOLOGY AND PRACTICES Amazonía Lee capitalized on the experience of the academic and research institution UPCH14,15during component implementation. As several informants commented, implementation itself was a continuous learning process, as elements were adjusted and improved, such as the selection of coaches with better credentials during year two and validation of the students' workbooks; all done in the context of the Amazonian environment. “… We have a style of training, in which we do not only train… That is, because we are a university and because of everything we have learned, we are also rigorous and systematic, because we always are… Since we set foot, starting the experience, we have thought that it is replicable and that others can do it and that we know how to do it….” University Specialist The training process for teachers was organized in three phases and focused on four practices (identified as estrategias formativas): Workshops, coaching visits (observation and teaching advice), demonstration classes, and study groups. The same training structure was designed and implemented for the school principals and other regional education staff. 16 Teacher training workshops: The training process considered a sequence of three workshops supported by educational material tailored to teach the learning process (elements of the phonetic approach, reading comprehension, phonetic awareness, and fluency), classroom environment and organization, differentiated instruction17, among other curricula contents. The total number of hours of training in the workshop per academic year was 120 hours.18 Coaching visits and in-class demonstration class model: The coaching visits were designed to help put into practice the methodologies and pedagogical strategies reviewed during the workshops. The visits varied from purely observational visits with technical advice and feedback after the observation to demonstrations in the classroom followed by discussions of what strategies and content were administered. The success of this activity was based on the coaches’ capabilities and the training they received from UPCH. Study Groups (GIAs): This was a practice in which a coordinating teacher or the school principal organized a meeting once a month for four hours. GIAs were held primarily on the weekends to avoid missing classes. The content and structure of meetings varied. They covered classroom planning, reviewed educational materials, and addressed any questions or concerns that teachers had. At the beginning of the GIAs, if a coach was attending the session, she or he 14 UPCH is a reputable academic institution with experience supporting pedagogical processes and teaching methodologies in primary schools. UPCH was selected in 2002 to lead entro Andino, a USAID-supported program implemented in three countries (Bolivia, Ecuador, and Peru), which aimed to strengthen teachers capacities and improve reading and writing performance of primary students. The program was implemented from 2003 to 2010 and focused on the improvement of reading and writing in grades 1 through 3. In Peru, it was implemented in the Lima, Callao, Cusco, Piura, and Ucayali regions. The training structure was similar to the one tested by Amazonia Lee, with three workshops for teachers, study groups, and coaching activities for teachers during the academic year, including demonstration classes. The training activities were supported by educational material prepared by UPCH, such as guidelines and modules for teachers, and workbooks for students. 15 USAID. Evaluación de los Programas en Educación de la USAID/Peru: AprenDes y Centro Andino. Summary Evaluation Report. April 2010. 16 Amazonía Lee. Mejoramos los aprendizajes en Lectura y Escritura en la Educación Primaria. UPCH. s/f. 17 This refers to strategies used in class to support students with different levels of reading skills (KII UPCH). 18 USAID (2018). Evaluation of Amazonía Lee Reading Intervention in Peru: Final Report. Pp 28-31. 15 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV could cover areas observed during teachers’ classes. Staff from different schools met together on some occassions. (KII in Ucayali and USAID impact evaluation report) Training for School Principals and Education Regional Staff: Amazonía Lee also structured and tested a training project for school principals and education regional staff. These modules empowered principals and regional education specialists in pedagogical strategies and school management. During the interviews, principals and specialists were knowledgeable about the teaching approaches, school management, and how their work is related to the development of the National Curriculum. EDUCATIONAL MATERIALS SUB COMPONENT19 Amazonía Lee implemented its model in polidocente completo and polidocente multigrado schools in communities where Spanish is the main language so the instructional language of the educational materials was also Spanish. Educational Materials for Students: These materials were designed and tested by UPCH with close collaboration of teachers and regional staff from the San Martín and Ucayali regions. The latter created a sense of ownership in regional staff and authorities. ● Three Student Workbooks on Orality, Reading, and Writing (Oralidad, Lectura, y Escritura). UPCH and regional staff prepared, reviewed, and validated the workbooks for 1st, 2nd and 3rd grade. These contextualized educational materials for children contain stories and images tailored to the culture of Amazonian regions and are pedagogically structured to be used with the phonetic and balanced approach. ● A guide for teachers to use the above mentioned workbooks. Educational Materials for teachers: UPCH also designed and edited materials to support the teaching practices during implementation. ● Three Teacher Guides to teach reading and writing ● A Guide to Differentiated Instruction in the Classroom. This resource included strategies for teaching and organizing the classroom to form groups of children with different reading and writing skill levels. Educational Materials for school principals and education regional specialists: Two modules (books) on education management: “School Management and Pedagogical Leadership” and “School and Context Diagnosis Methodology” COMMUNITY PARTICIPATION COMPONENT Community participation is a key component to maintain the approach’s relevance. Amazonía Lee involved parents and community members in reading fairs, and displayed students’ work at regional fairs. REGIONAL ASSESSMENT SYSTEM COMPONENT This component made it possible to follow up on students’ improvements in reading and writing, with the result that teachers were able to plan classroom strategies and differentiated 19 The visit to the UPCH campus allowed the Evaluation Team to observe the books and guidelines edited during USAID’s Amazonía Lee project implementation and during USAID’s Centro Andino project implementation. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 16 instruction more effectively. Unfortunately, the system was only partially implemented in San Martín and was implemented only in the treated schools of Amazonía Lee in Ucayali and not the rest of the schools as planned. REGIONAL INCENTIVE SYSTEM FOR TEACHERS COMPONENT A teacher reward system (using non-monetary incentives) was implemented in Amazonía Lee schools to promote students’ reading accomplishments, similar to a model implemented before by MINEDU.20 Amazonía Lee’s use of non-monetary incentives to improve the performance of teachers is an interesting tool, but rolling out the process did involve the investment of time. Table 1 below summarizes this section. Table 1. Summary of Evaluation Question 1.1 PHASE ELEMENT APPLICABILITY Design Balanced approach: Whole language combined with Phonetic approach Especially in remote or poor areas where there are few visible written messages and limited support from family Workshop for teachers Implemented successfully during two years for teachers of polidocente completo and polidocente multigrado schools of Amazonian regions Use of Coaches Responded to the needs of the Amazonian region, UPCH set the standard number of coaches per school Capacity building for school principals For school principals of polidocente completo and polidocente multigrado schools in communities where Spanish is the main language Contextualized materials Applicable throughout the Amazonian region, but only tested in the Spanish language Citizen participation Engagement with parents and community at large, congregating families and private sector to the reading and book fairs Implementation Regions own the program Participation in the implementation process, hiring human resources and validating of books created ownership Continual Improvement UPCH experience made it possible to support regions in improving the hiring of coaches with better profiles. The educational material was validated with stakeholders. The teacher workshops sequence allowed to review the advances and solved questions from previous phase Regional Reading Assessment Limited applicability if not active in all the schools of the region. Test other monitoring and evaluation systems used by MINEDU, avoiding duplications. Regional Incentive Systems for teachers (non-monetary) Applicable if expanded to all schools in the region; does not require additional budget, but it does require time to roll out the process 20 During the impact evaluation study (2017), only Amazonía Lee schools had this system functioning. 17 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV 1.2 HOW SIMILAR OR DIFFERENT ARE THE NEEDS OF TEACHERS AND LEARNERS IN OTHER AMAZONIAN REGIONS TO THE AMAZONÍA LEE REGIONS? The Amazonian regions share similar and different characteristics in the education sector. The variation in needs between Amazonía Lee regions and the other regions is examined below with regard to the following variables: ● Number and type of schools ● Urbanization and density of the region ● Number of students per teacher ● Primary language in the schools’ community NUMBER AND TYPE OF SCHOOLS The types of schools and characteristics of the settlements in the Amazonian region provide information on the similarities and differences of teachers’ and learners’ needs in the five regions: San Martín and Ucayali (with previous Amazonía Lee experience), and Amazonas, Madre de Dios, and Loreto (potential implementers). The informants also found similarities among Amazonian schools. “The reality of the schools in the Peruvian jungle are very similar, in a few aspects: the aspects of geographic dispersion, the aspects of rurality, the aspects of extreme poverty in which they are also under, they have similar contexts and characteristics. And I would even dare to say, that it is not only in the Amazonian environment, but that the country has many of those characteristics.” – Amazonía Lee Regional Specialist First, in terms of number and type of schools, Madre de Dios and Loreto represent two different cases. The first one has a smaller number - less than one tenth - of schools in comparison to the second. All regions represent a challenge due to the dispersion of schools in their territory beyond the cities, increasing the costs of education in general or of any project like Amazonía Lee. In relation to type of schools, Ucayali, Loreto, and Madre de Dios schools are more than 50% unidocente multigrado, while Amazonas and San Martín, have 44% and 37% respectively. Between 20% and 44% of the regions’ schools are polidocente multigrado, and between 12% and 26% are polidocente completo. It is worth restating that Amazonía Lee was implemented only in polidocente completo and polidocente multigrado schools. As presented in the next section, the percentage structure of type of school is related to the population settlements and geography. See Table 2. Table 2. Number of schools by type and region REGION AMAZONAS MADRE DE DIOS LORETO SAN MARTIN UCAYALI TYPE NO. % NO. % NO. % NO. % NO. % Polidocente completo 121 12.0 44 25.7 331 13.8 311 23.7 123 14.9 Polidocente multigrado 445 44.2 35 20.5 800 33.4 514 39.1 248 30.1 Unidocente multigrado 441 43.8 92 53.8 1,261 52.7 489 37.2 453 55.0 USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 18 REGION AMAZONAS MADRE DE DIOS LORETO SAN MARTIN UCAYALI TYPE NO. % NO. % NO. % NO. % NO. % Total 1007 100.0 171 100.0 2,392 100.0 1314 100.0 824 100.0 SOURCE: MINEDU ELABORATION: NORC URBANIZATION AND DENSITY OF THE REGION POPULATION Peru is becoming increasingly urbanized with growing medium-sized cities, and the Amazonian region is no exception. However, there are also territories with low population density, which represent a challenge for providing services like education and implementing programs like Amazonía Lee. According to the 2017 National Census, Peru had 29.4 million inhabitants. Amazonas, Ucayali, San Martín, and Loreto regions represented 1.3, 1.7, 2.8, and 3.0 percent of the national population, respectively. Madre de Dios only represented 0.5 percent, the smallest region in terms of population. At the same time, it is one of the fastest growing regions (INEI, 2018). The provinces of Maynas with Iquitos city (Loreto) and Coronel Portillo with Pucallpa city (Ucayali) are in the group of the 20th most populated provinces in the country. On the other hand, the region of Madre de Dios, despite having an important capital city representing 60% of its regional population, has a density of only 1.7 people per km2 compared to San Martín which has 15.9 people per km2 . Moreover, according to INEI, aside from San Martín, these Amazonian regions have the lowest density in the country. See Table 3. Table 3. Population characteristics, by region and main cities REGION POPULATION (2017) DENSITY (INHAB/KM2) 2007-2017 INTERCENSAL POPULATION GROWTH MAIN CITIES POPULATION (2017) % OF REGION POPULATION San Martín 813,381 15.9 1.10 Tarapoto 143,300 17.6 Moyobamba 50073 6.2 Ucayali 496,459 4.9 1.40 Pucallpa 326,040 65.7 Amazonas 379,384 9.7 0.10 Chachapoyas 32,026 8.4 Bagua Grande 50,841 13.4 Loreto 883,510 2.4 -0.10 Iquitos 377,609 42.7 Yurimaguas 62903 7.1 Madre de Dios 141,070 1.7 2.60 Puerto Maldonado 85,024 60.3 SOURCE: INEI ELABORATION: NORC The geographical distribution of schools in the Amazonian regions provides information on how similar the schools are and where to possibly apply the project. Given the urban concentration in Ucayali, most of the polidocente completo schools are located in the main urban center, 19 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV including Amazonía Lee schools. This is different from Loreto, San Martín, Madre de Dios, and Amazonas, where the number of polidocente completo schools not supported by Soporte Pedagógico or Amazonía Lee located in rural areas represent 44, 51, 64, and 71 percent, respectively. Additionally, as was expected in all regions, polidocente multigrado schools are mainly located in settings defined as rural. See Table 4. Table 4. Number and percentage of polidocente completo and polidocente multigrado schools in rural settings, by treatment and region REGION AMAZONAS MADRE DE DIOS LORETO SAN MARTIN UCAYALI TREATMENT SP NONE SP NONE SP NONE AL SP NONE AL SP NONE Polidocente completo schools No. 73 48 30 14 180 151 125 107 79 24 44 55 % rural 16 71 0 64 2 44 11 6 51 8 0 24 Polidocente multigrado schools No. 0 445 0 35 0 800 22 0 492 10 0 238 % rural 93 -- 86 -- 95 64 -- 92 90 -- 94 SOURCE: MINEDU ELABORATION: NORC STUDENT/TEACHER RATIO The polidocente completo schools that received Amazonía Lee and Soporte Pedagógico in Ucayali had a higher number of students per teacher (26 and 27, respectively) in comparison to Amazonía Lee and Soporte Pedagógico schools in San Martín, where the student-teacher ratio in polidocente completo schools was 21 to 1. This is important because the number of students per teacher (as a proxy for class size) could affect Amazonía Lee’s strategy of “differentiated instruction” during class, used when having students with different levels of reading skills. If the project is further implemented in Amazonas, Madre de Dios and Loreto, it would have classes in polidocente completo schools of 16, 22, and 20 students per teacher respectively, lower or similar to San Martín. Considering the polidocente multigrado schools without previous intervention (Amazonía Lee or Soporte Pedagógico), the ratio in Madre de Dios (15 students per teacher) would be lower than Amazonas and Loreto (18 and 20 students per teacher respectively). It is worth mentioning that Ucayali also had the highest number of students per teacher (26 to 1) in polidocente multigrado schools. See Table 5. Table 5. Number of students per teacher in polidocente completo, polidocente multigrado, and unidocente multigrado schools, per treatment and region REGION AMAZONAS MADRE DE DIOS LORETO SAN MARTIN UCAYALI TREATMENT SP NONE SP NONE SP NONE AL SP NONE AL SP NONE Polidocente completo 16 18 25 22 23 20 21 21 20 26 27 24 USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 20 REGION AMAZONAS MADRE DE DIOS LORETO SAN MARTIN UCAYALI TREATMENT SP NONE SP NONE SP NONE AL SP NONE AL SP NONE Polidocente multigrado 18 15 20 20 18 26 21 Unidocente multigrado 15 11 17 15 18 Average 16 16 25 13 23 18 21 21 17 26 27 19 SOURCE: MINEDU ELABORATION: NORC PRIMARY LANGUAGE IN THE SCHOOLS’ COMMUNITIES Similar to San Martín and Ucayali, Amazonas, Madre de Dios, and Loreto also have schools in communities where Spanish is not the main language. According to informants, Amazonía Lee was implemented in a small number of those schools as using Spanish as the instructional language was less recommended in this context. One pedagogical DRE Director mentioned that this was a little bit “forced.” For cases where there are more than one language spoken in the community, MINEDU issued a policy guideline in 2018 to promote teaching children to read in their native language and adding Spanish as an instructional language at an appropriate time.21 These schools could eventually participate in the Amazonía Lee replication project, but would have to coordinate first with MINEDU’s Directorate for Intercultural, Bilingual and Rural Education (Dirección General de Educación Intercultural, Bilingüe y Rural, DIGEIBIR) and establish how and where to insert the phonetic approach tested in Spanish. With regard to non-intervened schools, with the exception of San Martín, the rest of the regions have more than 26% of polidocente completo schools in communities where Spanish is not the main language. Amazonas stands out with a high percentage of polidocente completo schools (54%) in this type of communities. The same pattern is observed in the case of non￾intervened polidocente multigrado schools: all of the regions have more than 30% polidocente multigrado schools in these communities, except for San Martín (with only 7%). Table 6. Percentage of polidocente completo, polidocente multigrado, and unidocente multigrado schools in communities where Spanish is not the main language, by treatment and region REGION AMAZONAS MADRE DE DIOS LORETO SAN MARTIN UCAYALI TREATMENT SP NONE SP NONE SP NONE AL SP NONE AL SP NONE Polidocente completo 1% 54% 0% 14% 1% 26% 2% 4% 11% 0% 0% 29% Polidocente multigrado 33% 31% 35% 5% 7% 0% 44% 21 MINEDU (2018). Modelo del Servicio Intercultural Bilingüe. In http://repositorio.minedu.gob.pe/handle/MINEDU/5971, June 23, 2019. 21 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV REGION AMAZONAS MADRE DE DIOS LORETO SAN MARTIN UCAYALI TREATMENT SP NONE SP NONE SP NONE AL SP NONE AL SP NONE Unidocente multigrado 12% 13% 18% 3% 21% Average 1% 24% 0% 18% 1% 25% 3% 4% 5% 0% 0% 29% SOURCE: MINEDU ELABORATION: NORC Table 7 below summarizes this section. Table 7. Summary of Evaluation Question 1.2 CHARACTERISTIC SIMILARITIES AND DIFFERENCES ACROSS REGIONS Type of schools Unidocente multigrado schools represent more than 50% of schools in the Amazonian regions, except for San Martín. However, Amazonía Lee was only implemented in polidocente completo and polidocente multigrado schools Number of schools and rurality Loreto has an important number of polidocente completo and polidocente multigrado schools in rural settings without previous intervention (44% of 151 and 95% of 800, respectively). While Madre de Dios also has a high percentage of schools with no intervention, the total number is much smaller than in the rest of the regions. All of the regions have similar challenges of rurality when dealing with polidocente multigrado schools Concentration of school population in main cities Madre de Dios, Loreto, and Ucayali regions have high levels of population concentration in their main cities, facilitating coverage of education services in urban centers. At the same time, they have a high level of population dispersion in the rest of the territory, which represents a challenge to reaching the rural student population. Amazonas and San Martín have smaller cities (in terms of the percentage of the regional population living there), but have greater population density in the rest of the territory than the other three regions. Class size The polidocente completo schools of Amazonía Lee and Soporte Pedagógico in Ucayali and Madre de Dios have a higher ratio of students per teacher in comparison to schools receiving these programs in Amazonas, San Martín, and Loreto. Concerning polidocente multigrado schools, Ucayali has the highest level of students per teacher in comparison to the rest of the Amazonian regions. Non-Spanish as primary language Amazonas has more than 50% of polidocente completo schools in communities where Spanish is not the main language. Loreto and Ucayali also have a high percentage of polidocente completo schools in communities where Spanish is not the main language (more than 25%), while all of the regions, except for San Martín, have more than 30% of polidocente multigrado schools in communities where Spanish is not the main language. 1.3 TO WHAT EXTENT COULD A SINGLE AMAZONÍA LEE STRATEGY BE DEVELOPED FOR ALL AMAZONIAN REGIONS WHILE REMAINING APPLICABLE TO THE NEEDS OF EACH REGION? DO CERTAIN AMAZONÍA LEE ELEMENTS LEND THEMSELVES TO A SINGLE STRATEGY BETTER THAN OTHERS? The Amazonía Lee model has a comprehensive structure and a holistic philosophy that was conceptualized and implemented following a single strategy in the Amazonian context, mainly in urban and peri-urban areas with polidocente completo and polidocente multigrado schools. At the same time, the implementation showed that the approach had flexibility in some elements to make it possible to use the standard set of inputs while maintaining the effectiveness of the approach. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 22 IDENTIFIED ELEMENTS THAT CAN BE IMPLEMENTED USING A SINGLE STRATEGY ● The overall teaching approach combining the phonetic approach with the whole language approach for polidocente completo and polidocente multigrado schools where Spanish is used as instructional language or in communities where Spanish is the main language ● Capacity Building sequence and contents of practices: workshops, coaching and demonstration classes ● Inclusion of different stakeholders in the education sector, who participate in training and management activities: coaches, teachers, school principals, and regional education staff ● Use of contextualized educational materials already edited by UPCH: workbook for students, modules and guidelines for teachers, school principals and regional educational staff ELEMENTS THAT CAN BE IMPLEMENTED WITH FLEXIBLE STRATEGIES The implementation of Amazonía Lee allowed identifying the list of ingredients of the model and the parameters (quantity) of them. This list serves as a reference to analyze the flexibility of strategies and develop financial estimates to evaluate the possibility of replication. The parameters could be modified depending on the school type and population dispersion.22 ● Four-day workshops for teachers each year (120 hours per year). School principals and regional educational staff could also attend the events ● One full-time coach supporting 30 teachers in polidocente completo ● One full-time coach supporting 15 teachers in polidocente multigrado ● Six visits from coaches to teachers per year23 ● Study groups once a month for at least 4 hours ● Educational materials per each student, teacher, and coach replaced every year ● Four educational specialists from UPCH for 30 coaches ● One organization that provides technical assistance, considering direct and overhead costs, located in Lima ● Three-year timeline The items above are described in additional detail below. Capacity Building Training Practices. Workshops with training modules of adjustable length, while maintaining the prioritized content. Online classes or blended classes (online combined with in-person training), that include guidelines for using the students’ workbooks. Coaching and demonstration classes. Based on the analysis of the Amazonía Lee model and field visits, the main strategies revolve around the need to provide the adequate number of 22 The list has been identified from primary (KIIs, UPCH visit), and secondary sources. 23 During the FGD with coaches, it was mentioned that there were cases when some teachers needed more coaching, and the coached visited the schools up to eight times 23 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV experienced coaches necessary to maintain the standards of inputs. As discussed before, there are similarities and differences among Amazonian regions with respect to the type of schools and their location. The model has established a standard of 30 teachers per coach in polidocente completo schools and 15 teachers per coach in polidocente multigrado schools (KII UPCH). The smaller groups of teacher for polidocente multigrado schools were necessary to maintain around six coaching visits per year. Working with polidocente multigrado schools require different strategies to ensure that teachers have enough coaching support, with attention to distance and the costs of transportation. (This topic is discussed further in question 2.3.) In terms of contracts, the coach could be a full-time teacher in a polidocente completo school and also carry out training activities for surrounding small schools. For example, the principal of a Soporte Pedagógico school was satisfied with the support of the “internal coach” that Soporte Pedagógico financed for the school. According to her, the role of internal coach is already considered in the national normative framework. Networking of polidocente multigrado schools. The concept of Resource Centers for polidocente multigrado schools applied by past USAID education projects (AprenDes, and Support for the Reform of Quality Basic Education (Apoyo a la Reforma de la Educación Básica de Calidad, SUMA)) could be another alternative. Likewise, a teacher who also fills the role of school principal in a big polidocente multigrado school could be trained to assume the role of coach for the network. During the interviews with school principals, they commented that they were prepared to act as teachers or as advisers after preparation in pedagogical strategies and approaches. Study Groups (GIAs). The teachers’ meetings with the pedagogical coordinator to plan the weekly classes was also utilized by other programs, like Soporte Pedagógico, to support teachers’ practices. The location of these meetings varied. For example, one interviewed teacher, who was the pedagogical coordinator of the school, offered her house for the meetings on Saturdays. Project Technical Coordination and Assistance. A Non-Governmental Organization (NGO) or academic institution like UPCH could provide support to the project and continue close coordination with regional educational agencies like the DRE and the UGEL. A second strategy could be a more active participation of institutions from the same region. Having different stakeholders in the sector, there are more than two options to organize the coordination and technical assistance. Community Participation Component. This practice will depend on the common practices of the Amazonian community in each settlement. Besides the reading fairs, or displaying the students’ work in commercial regional fairs, the project could also use the practice of the “Día del Logro” used in AprenDes and SUMA projects and institutionalized in MINEDU. With respect to strategies at the school and classroom level, KIIs and FGDs revealed that: ● The implementation of Amazonía Lee allowed for some flexibility regarding how much of each approach (phonetic and whole language) to use. During the KIIs in both visited regions, pedagogical regional staff, principals, and teachers argued that the application of the balanced approach does not mean using 50% of the phonetic approach and 50% of the whole language approach. ● Principals and teachers can adjust the strategies according to the needs and advancement of the children, and the teachers can decide when to use the phonetic and whole language approaches alone or combined. This is slightly different from what was gathered through the USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 24 initial interviews administered in Lima, which appeared to suggest a more rigid combination of the two approaches. ● A specialist who worked on Amazonía Lee commented that the phonetic approach is not recommended for children that already have reading skills when they start first grade. The differentiated instruction of Amazonía Lee considers the possibility to work different activities with students with different levels of advancement. “Now, related to the phonology, obviously, the child that already reads, sometimes, already does not need as much help from this phonetic approach, from the sound of each letter to put together. Because he already has the experience, already knows the sound, despite having come already knowing “ma-me￾mi-mo-mu,” decodes… but with the children who did not decode,… then, with those children the phonetic awareness helped us…” Amazonía Lee Regional Specialist Table 8 below summarizes this section. Table 8. Summary of Evaluation Question 1.3 STRATEGY ELEMENTS THAT CAN BE IMPLEMENTED Single Strategy Teaching approach for polidocente completo and polidocente multigrado schools where Spanish is used as instructional language Sequence and content of Capacity Building practices Consideration of stakeholder participation Use of contextualized books Flexible strategies Workshops adjusted time and modality (online, blended) Coaches with diverse contract relationship with the UGEL Number of teachers per coach depending on the type and distance of schools Possible school networking to administer coaching Teachers’ study groups Different option to have project technical assistance agency: national and decentralized agency (DRE, UGEL) with regional academic institution Varied parents and community engagement Classroom strategies: adjusting the use of phonetics and whole language according to students needs 1.4 HOW COULD AMAZONÍA LEE, OR KEY COMPONENTS OF IT, BE PACKAGED TO PRESENT TO THE DRES? According to the KIIs and FGDs conducted, Amazonía Lee could be packaged around the insertion of phonetic awareness and the Capacity Building component. Informants identified the following as key subcomponents: training for educational specialists (coaches) with specific content on the phonetic approach, workshops for teachers, periodic visits to teachers, demonstration classes during coaching, and contextualized workbooks for 1st, 2nd, and 3rd grades. 25 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV The training for coaches (formadores) was considered to be very important. One coach explained that UPCH trained them first so they could replicate the process. The participants trained to be coaches were proud of the training program, which strengthened their abilities to help teachers think deeply about their own problems and identify solutions. One important element to underline is the number and content of workshops for teachers. Traditionally, before Soporte Pedagógico, one would expect a large number of workshops before the academic year led by MINEDU together with the regional educational bodies. Instead, Amazonía Lee implemented three workshops organized in a sequence during the school year, with connected content. The UPCH Coordinator called the modality “training stages” (“etapas formativas”), where each workshop is based on the content of the previous one and the experience of teachers after the previous workshop. The content of the workshops for teachers did not cover teaching methodologies for Math. There are two set of opinions on this topic. On the one hand, as commented by one school principal, children will learn mathematics much better if they learn to read, and there is no need for in-service training for teachers “… but if the child learns to correctly understand a message, which is the fundamental part of communication, it is easier to teach mathematics. Because the child will learn to detect what thing they are asking for, what thing they are instructing. So, that has been easier for us, we can now look for mathematical strategies and that the child could understand…” School principal On the other hand, a pedagogical specialist and former Amazonía Lee regional coordinator suggested that math can be included in the workshops24 : “Mathematics has its own didactic processes that also require starting from a formulation of problems. But it has, like is said, its strategies that also, let’s say, the teachers are also lacking…” Amazonía Lee Regional Specialist The rapid teachers’ survey confirmed that, out of 13 respondents, more than 90% attended three to four Amazonía Lee workshops during the year. Furthermore, most of the 19 respondents supported by Soporte Pedagógico attended only one or two workshops during the year. See Figure 2. 24 One KII with a UPCH specialist revealed that UPCH has included math teaching processes in its current project implemented in the Coast. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 26 Figure 2. Number of workshops attended by teachers (as a percentage of respondents) NOTE: THE SAMPLE INCLUDED 13 AMAZONÍA LEE RESPONDENTS AND 19 SOPORTE PEDAGÓGICO RESPONDENTS SOURCE: NORC, RAPID TEACHERS’ SURVEY, 2019 ELABORATION: NORC Some informants pointed out that workshops organized by the Ministry are too general in comparison to Amazonía Lee workshops, which include a preview of the demonstration class. With regard to the coaching of teachers, Peru has a vast experience with models and strategies wherein professsionals are trained and analyze their work with their peers. Given that coaching teachers is not a new strategy and the educational sector is already familiar with it, and it would be simple to conceptualize.25,26 Meanwhile, the demonstration classes, as part of the coaching process, have been regarded as novel since they allow teachers to actually observe how to administer a class and how to apply the concepts learned during the workshops. “The visit involves them entering the classroom, observing your class, and after class, doing the consultation… Or the visit also involves, that they enter your classroom, they do a demonstration session, and after that, they do the consultation about the experience [the observation].” Pedagogical Specialist “But through the demonstration sessions you learn by doing, right? The trainer, the facilitator could and had the space to concretize the theory in practice during the pedagogical hours. And the teacher observed what they had already talked about in the workshop, what they had talked about in the GIA [Inter-Learning Group]. It was somewhat cyclical, that is, everything stuck. If in the workshop, the idea did not set in, because not everyone is at the same level, the teachers, were taken to the GIA, a necessity of the school. If they did not complete the GIA, now we are going to the demonstration sessions, to the reflection consultation….” - Trainer There was also a different opinion regarding the demonstration classes: 25 Peru recognized coaching as a budgetary line in 2007, under the program PELA, Programa Presupuestal 0090 Logros de Aprendizaje de Estudiantes de Educacion Basica Regular. 26 USAID projects like AprenDes and SUMA, Canada's project PROMEB, all implemented coaching. AprenDes project supported the institutionalization of the PELA budget program in the Ministry of Economics and Finance in the 2007 budget appropriation. 52.63% 47.37% 7.69% 92.31% 0 10 20 30 40 50 60 70 80 90 100 1 to 2 3 to 4 Number of workshops attended Amazonia Lee Soporte Pedagogico 27 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV “In education, there is nothing new. Let’s start from there. We have to get it out of our heads that it is a demonstration, as we say, previously it was called a model class, after that, master class, and today demonstration. They are simply names. It is a class that comes, look… Every teacher is the author of their own session, of their own planning, through the research that he does so he can land in the classroom.” School principal The workbooks are important and possible to reproduce as needed and according to budget availability, especially for 1st and 2nd graders. UPCH commented that it would be challenging to use them if the teacher is not trained in the workshop or at school on the phonetic approach. Therefore, these sub-components (workshops and workbooks) are considered by UPCH essentially linked. It is interesting to notice that in the rapid teachers’ survey, 9 out of 13 teachers disagree with the idea that they will stop applying the phonetic approach if they do not receive the materials. That means that the workshops and coaching were effective at transferring the foundation knowledge and practice of the phonetic approach. The Evaluation Team observed in all visited schools that principals and adjunct principals’ were empowered in the pedagogical processes differentiating the approaches in teaching children how to read and write, but it was not exclusive to Amazonía Lee. DRE Directors could structure the project so that it responds to the demands of their sectors (i.e. in accordance with the type of schools and their performance). To that end, they could estimate the quantity of inputs, especially human resources, of the Amazonía Lee components and subcomponents that would be needed to fill gaps and achieve the expected results. Table 9 below summarizes this section. Table 9. Summary of Evaluation Question 1.4 ELEMENTS OF AMAZONÍA LEE’S CAPACITY BUILDING COMPONENT ATTRIBUTES OF AMAZONÍA LEE FOR PRESENTATION TO DRES Teaching approach Insertion of phonetic awareness appropriate for the Amazonian student population Coaches Importance of the selection process and training of coaches Workshops for teachers Sequence of contents (“training stages”) related to the academic year. Include other areas besides reading improvement like mathematics Coaching Frequent coaching visits with demonstration classes to practice the pedagogical processes reviewed during the workshops Educational materials Workbooks contextualized to Amazonian regions ready for printing together with teacher guides Training for school principals and DRE-UGEL education specialist Empowered in pedagogical processes to support teachers Coverage of potential program Select prioritized schools (number and type), and analyze feasibility to support components and subcomponents needed. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 28 EQ2: HOW CAN APPLICABLE COMPONENTS OF AMAZONÍA LEE BE REPLICATED IN OTHER AMAZONIAN REGIONS OF PERU? The assessment of replicability of the applicable components is understood as the possibility of the regions to implement the necessary processes to provide the ingredients of the Amazonía Lee model. This is possible if the regional educational agencies (DRE and UGEL) have the capacity to manage the implementation and if the regional government at large is able to finance the components. 2.1 ARE THERE CERTAIN KEY COMPONENTS THAT ARE MORE REPLICABLE THAN OTHERS? WHY AND WHY NOT? Among the four components, Capacity Building, which includes training activities and the availability of contextualized materials, would be the component that requires more attention. Therefore, the DRE and UGEL have to be able to put together a cadre of human resources needed for training and secure the financing for the rollout. CAPACITY BUILDING COMPONENT – SUBCOMPONENT RELATED TO WORKSHOPS FOR TEACHERS AND COACHING This is the key subcomponent of the Amazonía Lee model and, at the same time, the one that brings the highest challenge. First, as commented during the KIIs, the project had to adjust the offered salary more than once to maintain the coaches’ interest in working on the project. Second, UPCH adjusted the profile of the coaches the second year and hired additional professional to cover the open vacancies because the first group selected by the region did not have enough knowledge and skills. Third, there are not enough professionals in the market to cover those profiles, unless the best teachers transfer temporary to this employment opportunity, but again their schools are left without a good teacher. This topic is discussed further in EQ 2.3. The workshops also absorb a big amount of the budget (costs of transportation and per diem), especially if teachers have to get together three times a year. CAPACITY BUILDING COMPONENT – SUBCOMPONENT RELATED TO CONTEXTUALIZED EDUCATIONAL MATERIAL Informants commented that not all UGELs are prepared to design textbooks like Amazonía Lee’s on their own. However, no new educational material design is foreseen because contextualized workbooks and materials for training were already validated and used in San Martín and Ucayali. According to UPCH, teachers must be trained in properly using these materials, especially the workbooks. The books could be available online together with a training session on how to use them. According to KIIs with UPCH staff and a review of the teaching materials, there is a need to clarify the copyright of the books and their free availability online for printing purposes. OTHER COMPONENTS The Community Participation Component and the non-monetary Incentive System for Teachers Component are feasible to replicate because of their low budget. However, they require the commitment of UGEL staff and school principals to use their time to organize the activities and monitor the teachers’ performance on a permanent basis. Table 10 below summarizes this section. 29 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Table 10. Summary of Evaluation Question 2.1 COMPONENT (– SUBCOMPONENT) REPLICABILITY Capacity Building Component - Teacher Coaching Key ingredient, in which the salary represents 70% or more of the budget. Challenges to find enough coaches with the proper profile, and difficulties due to the territorial dispersion. Capacity Building Component - Workshops Challenge to organized and finance transportation and per diem for sessions that last 120 hours for training face-to-face education modality Capacity Building Component - Contextualized educational material No cost for designing, contextualized for the whole Amazonian region. Ready for printing, but it is not clear if there is need to clear copyright to share online. Community Participation Component Organized by UGEL and supported by school principals. No problem is foreseen for replication Regional (Non-Monetary) Incentive System for Teachers Component No problem is foreseen for replication 2.2 BASED ON PERU’S DECENTRALIZATION POLICY, DO THE COSTS OF THE APPROACH ENABLE REPLICABILITY? ARE THERE CERTAIN ELEMENTS OF THE MODEL THAT SHOULD BE PRIORITIZED? The education sector is financially decentralized; the regional governments are in charge of human resources (71% of the education budget)27. The balance of investments, goods and services is split between MINEDU and regional governments. However, MINEDU leads the policy decision making in areas that affect Amazonía Lee implementation: ● Human resources: MINEDU is in charge of the design, development and monitoring of the teachers’ tenure track (Carrera Pública Magisterial). Regions are responsible to implement the processes of public calls, test administration (design at the national level), and final hiring. ● Educational materials like books are under the responsibility of MINEDU. Contextualized books have not yet been produced, but an informant at MINEDU expressed their interest in beginning to work on that effort. ● Special programs like Soporte Pedagógico often originate in MINEDU and are later transferred to regions, or the budget and responsibilities are shared. ● Information and technology systems like the Information System for the Support of the Management of Schools (Sistema de Información de Apoyo a la Gestión de la Institución Educativa, SIAGIE) and National Standard Examinations like ECE, both used in all the regions nationwide. As a consequence, most of the cost of a possible replication would be borne by the regions’ budgets; it will depend on how the regions are thinking to finance a potential replication. SETTING THE COVERAGE TO EVALUATE REPLICABILITY COSTS The first decision regions would have to make, as observed earlier, is to assess the demand of their region, vis à vis the financial feasibility of providing the supply to meet that demand. Since it would be difficult to replicate the Amazonía Lee model in all schools, the DREs could make a 27 Includes the payroll of the deconcentrated UGELs in Lima city, normally included in the national budget. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 30 technical decision to target a group of schools and prioritize selected components. The Capacity Building component could be prioritized, including the phonetic approach, workshops, coaching, demonstration classes, and contextualized materials with proper contents. For simplification purposes, the replication of the project could start in schools in communities where Spanish is the main language, and work with four possible options: ● Option 1: Polidocente multigrado schools with no previous intervention from the project (mostly rural). ● Option 2: Polidocente completo schools with no previous support from Soporte Pedagógico (mostly urban). ● Option 3: Polidocente completo schools with previous Soporte Pedagógico support (mostly urban). ● Option 4: A combination of the above. Based on the positive impact of Soporte Pedagógico28, it is very likely that the schools with Soporte Pedagógico are better performers than the rest of schools. For that reason, it would be more efficient to invest in schools with no previous interventions (options 1 and 2). These two options provide parameters to identify the number and locations of the schools and estimate the demand of human resources (coaches and UGEL/DRE pedagogical specialists), and the number of students and teachers to receive the educational materials (books and workbooks). The financial calculations and implications of the Capacity Building Component replication are presented next for each region (Amazonas, Madre de Dios and Loreto). To simplify the analysis and cost estimates, and to present results that are easy to understand, the Evaluation Team proposed scenarios in each region. The budget considers the list of elements presented in EQ 1.3, and unitary costs of prioritized Capacity Building component. Estimates are based on budget parameters and unitary costs of Capacity Building Component provided by UPCH; direct costs do not consider operators’ administrative cost or overhead; and SIAF analysis uses 2018 accrued budget (devengado). AMAZONAS The region of Amazonas could potentially replicate Amazonía Lee Capacity Building component in schools with no previous Soporte Pedagógico intervention and in communities where Spanish is the main language29; the estimated number of schools is 22 polidocente completo and 298 polidocente multigrado schools scattered around the region. MINEDU reports that 11 polidocente completo have been supported in Chachapoyas, and 17 polidocente completo have been supported by Soporte Pedagógico in Bagua Grande, leaving the rest of polidocente completo schools outside the city (MINEDU web page30 and Education Census and INEI 2018). 28 Chinen, M., & Bonilla, J. (2017). Evaluación de Impacto del Programa de Soporte Pedagógico del Ministerio de Educación del Perú. Grupo FORGE GRADE, Global Affairs Canada. 29 Total number of schools minus number of schools supported by Soporte Pedagógico and located in communities where Spanish is not the main language. Tables 1, 3, and 5. 30 http://www.minedu.gob.pe/soporte-pedagogico/pdf/loreto.pdf 31 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Scenario: Polidocente completo and polidocente multigrado schools ● Goal of supporting 22 polidocente completo schools with a ratio of 30 teachers per coach; and 298 polidocente multigrado schools with a ratio of 15 teachers per coach (due to the dispersion of schools); located in communities where Spanish is the main language and not supported before by Soporte Pedagógico, requiring 47 coaches. ● The other inputs remain the same: 3 workshops and 6 coaching visits, and same materials for teachers, coaches and students per year. ● The number of teachers per coach (30) might be set too high because these schools are not in the main city and more coaches might be needed due to the dispersion of schools. The model estimates that implementation of the Amazonía Lee program, over a period of three years, would cost around 1.02% of the basic education recurrent budget or 2.25 % of primary education recurrent budget. See Table 11. Table 11. Three-year budget estimate of the Capacity Building Component in Amazonas (sample of polidocente completo and polidocente multigrado schools) ELEMENTS AND ASSUMPTIONS ITEMIZED BUDGET IN PERUVIAN NUEVOS SOLES 22 polidocente completo and 298 polidocente Human Resources 9,221,400 multigrado schools Coaching activities and transportation 218,520 30 teachers per coach in polidocente completo schools 15 teachers per coach in polidocente multigrado schools Educational materials for coaches 9,095 Educational materials for polidocente multigrado teachers 36,498 3 workshops Educational materials for polidocente completo 14,484 teachers 6 visits to teachers Educational materials for students 170,745 Yearly provision of educational materials for students, coachers, principals and teachers Workshops 1,655,235 4 educational specialists Central organization support on training 217,518 One central organization Total Capacity Building Budget 11,543,495 Assumptions: Recurrent regional Basic Education budget, 3 years 1,127,078,868 ● 4,000 nuevos soles per month salary plus fringe benefits, 15 salaries Recurrent regional Primary Education budget, 3 513,885,921 years ● 2 teachers per polidocente multigrado school ● 2 classes per grade in polidocente completo Intervention budget to Basic Education budget 1.02% Intervention budget to Primary Education budget 2.25% NOTE: ACCRUED 2018 BUDGET AS REFERENCE. SOURCE: MEF’S SIAF (CONSULTED JUNE 2019) AND UPCH. ELABORATION: NORC USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 32 LORETO If Loreto region targets schools with no previous intervention and in communities where Spanish is the main language31, the project could be potentially replicated in 112 polidocente completo and 520 polidocente multigrado schools located outside the two main cities and scattered along its territory. Iquitos and Yurimaguas have 135 polidocente completo schools (108 in Iquitos and 27 in Yurimaguas), and Soporte Pedagógico has already supported 99 of these schools in Iquitos and 24 in Yurimaguas city (MINEDU web page32 and Education Census and INEI 2018), leaving very few schools to work within the cities. Scenario 1: Polidocente completo schools ● Goal of supporting 112 polidocente completo schools not supported by Soporte Pedagógico before with a ratio of 30 teachers per coach, requiring 25 coaches ● The other inputs remain the same: 3 workshops and 6 visits, and same materials for teachers, coaches and students per year. ● The number of teachers per coach (30) could be set too high because these schools are not in the main cities (Iquitos or Yurimaguas) and more coaches might be needed due to the dispersion of schools. ● The model estimates that the implementation over 3 years would cost around 0.36% of the basic education recurrent budget or almost 0.86% of primary education recurrent budget. See Table 12. Table 12. Three-year budget estimate of the Capacity Building Component in Loreto (sample of polidocente completo schools) ELEMENTS AND ASSUMPTIONS ITEMIZED BUDGET IN PERUVIAN NUEVOS SOLES 112 polidocente completo schools 30 teachers per coach 3 workshops 6 visits to teachers Yearly provision of educational materials for students, coachers, principals and teachers 4 educational specialists One central organization Assumptions: Human Resources 4,905,000 Coaching activities and transportation 50,400 Educational materials for coaches 4,838 Educational materials for teachers 44,900 Educational materials for students 86,158 Workshops 2,691,954 Central organization support on training 435,036 Total Capacity Building Budget 8,218,285 Recurrent regional Basic Education budget, 3 years 2,304,418,617 31 Total number of schools minus number of schools located in communities where Spanish is not the main language. Tables 1, 3, and 5. 32 http://www.minedu.gob.pe/soporte-pedagogico/pdf/loreto.pdf 33 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ELEMENTS AND ASSUMPTIONS ITEMIZED BUDGET IN PERUVIAN NUEVOS SOLES ● 4,000 nuevos soles per month salary plus fringe benefits, 15 salaries ● 2 classrooms per grade Recurrent regional Primary Education budget, 3 years 955,505,874 Intervention budget to Basic Education budget 0.36% Intervention budget to Primary Education budget 0.86% NOTE: ACCRUED 2018 BUDGET AS REFERENCE. SOURCE: MEF’S SIAF (CONSULTED JUNE 2019) AND UPCH. ELABORATION: NORC Scenario 2: Polidocente multigrado schools ● Goal of supporting 260 polidocente multigrado schools (50% of 520 polidocente multigrado schools) located in communities where Spanish is the main language and with a ratio of 15 teachers per coach (due to dispersion), requiring 37 coaches ● The other inputs remain the same33: 3 workshops and 6 visits, and same materials for teachers, coaches and students per year. The model estimates that the implementation over 3 years would cost around 0.43% of the basic education recurrent budget or 1.03% of primary education recurrent budget. See Table 13. Table 13. Three-year budget estimate of the Capacity Building Component in Loreto (sample of polidocente multigrado schools) ELEMENTS AND ASSUMPTIONS ITEMIZED BUDGET IN PERUVIAN NUEVOS SOLES 260 polidocente multigrado schools Human Resources 7,259,400 15 teachers per coach Coaching activities and transportation 187,200 3 workshops Educational materials for coaches 7,160 6 visits to teachers Educational materials for teachers 32,029 Yearly provision of educational materials for students, coachers, principals and teachers 4 educational specialists Educational materials for students 66,210 Workshops 1,893,1950 One central organization Central organization support on training 435,036 Assumptions: Total Capacity Building Budget 9,880,230 ● 4,000 nuevos soles per month salary plus fringe benefits, 15 salaries ● 2 teachers per school Recurrent regional Basic Education budget, 3 years 2,304,418,61 Recurrent regional Primary Education budget, 3 years 955,505,874 Intervention budget to Basic Education budget 0.43% 33 The proposal budget estimate was based on the projection using 30 coaches supporting 30 teachers each; the rest of the ingredients followed the parameters of the model. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 34 ELEMENTS AND ASSUMPTIONS ITEMIZED BUDGET IN PERUVIAN NUEVOS SOLES Intervention budget to Primary Education budget 1.03% NOTE: ACCRUED 2018 BUDGET AS REFERENCE. SOURCE: MEF’S SIAF (CONSULTED JUNE 2019) AND UPCH. ELABORATION: NORC MADRE DE DIOS The case of Madre de Dios is different from Loreto, with only 12 polidocente completo and 24 polidocente multigrado non-intervened schools and located in communities where Spanish is main language34. The small number of schools could indicate at first that there is a higher probability to implement the complete model and strategies already applied in San Martín and Ucayali. However, the city of Puerto Maldonado concentrates 15 polidocente completo schools (out of 30 urban polidocente completo supported by Soporte Pedagógico) and, according to the Soporte Pedagógico report, these schools have been already supported by Soporte Pedagógico project in 2016. It would be challenging to replicate Amazonía Lee in polidocente multigrado schools located in small communities outside the city capital in a territory considered to be the least dense in the country (MINEDU’s web page35 and Education Census, and INEI 2018). Scenario. Polidocente completo and polidocente multigrado schools ● Goal of supporting 100% of schools located in communities where Spanish is the main language and not previously supported by Soporte Pedagógico: 12 polidocente completo schools with a ratio of 30 teachers per coach; and 24 polidocente multigrado schools with a ratio of 15 teachers per coach (due to the dispersion of schools), which requires 6 coaches. ● The other inputs remain the same: 3 workshops and 6 coaching visits, and same materials for teachers, coaches and students per year. ● The number of teachers per coach (30) for polidocente completo schools could be set too high because these schools are not in the main city and more coaches might be needed due to the dispersion of schools. The same issue arises for the polidocente multigrado schools. The model estimates that implementation of the Amazonía Lee program, during three years, would cost around 0.68% of the basic education recurrent budget or 1.81 % of primary education recurrent budget. See Table 14. Table 14. Three-year budget estimate of the Capacity Building Component in Madre de Dios (sample of polidocente completo and polidocente multigrado schools) ELEMENTS AND ASSUMPTIONS ITEMIZED BUDGET IN PERUVIAN NUEVOS SOLES 12 polidocente completo and 24 polidocente multigrado schools 30 teachers per coach in polidocente completo schools Human Resources 1,177,200 Coaching activities and transportation 19,440 Educational materials for coaches 1,161 34 Total number of schools minus number of schools located in communities where the Spanish is not the main language. Tables 1, 3, and 5. 35 http://www.minedu.gob.pe/soporte-pedagogico/pdf/madre-de-dios.pdf 35 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ELEMENTS AND ASSUMPTIONS ITEMIZED BUDGET IN PERUVIAN NUEVOS SOLES 15 teachers per coach in polidocente multigrado schools 3 workshops 6 visits to teachers Yearly provision of educational materials for students, coachers, principals and teachers 4 educational specialists One central organization Assumptions: ● 4,000 nuevos soles per month salary plus fringe benefits, 15 salaries ● 2 teachers per polidocente multigrado school ● 2 classes per polidocente completo grade Educational materials for polidocente multigrado teachers 6,783 Educational materials for polidocente completo teachers 13,036 Educational materials for students 30,138 Workshops 358,416 Central organization support on training 217,518 Total Capacity Building Budget 1,823,691 Recurrent regional Basic Education budget, 3 years 269,910,192 Recurrent regional Primary Education budget, 3 years 100,834,278 Intervention budget to Basic Education budget 0.68% Intervention budget to Primary Education budget 1.81% NOTE: ACCRUED 2018 BUDGET AS REFERENCE. SOURCE: MEF’S SIAF (CONSULTED JUNE 2019) AND UPCH. ELABORATION: NORC Table 15 below summarizes this section. Table 15. Summary of Evaluation Question 2.2 ELEMENT REPLICABILITY Decentralization MINEDU holds authority on human resources policies, educational materials, policy and promotion of national programs, information and technology systems, and national standardized exams; all of these affect Amazonía Lee’s components. Beneficiary schools Regions would be able to set the key variables for prioritizing schools (location, type of schools, previous intervention, performance, etc.), and decide how many school they can cover with the Amazonía Lee model. Dispersion will affect replicability unless the location of the coach is adjusted. Amazonas case Small number of polidocente completo and large number of disperse polidocente multigrado (2 and 298 respectively). It will require 47 coaches to cover all of them, and an additional 2.25% of the recurrent primary education budget. Loreto case Large number of polidocente completo and polidocente multigrado schools (112 and 520, respectively). It will require 25 coaches for the polidocente completo schools, and 37 coaches to support polidocente multigrado schools. It is estimated to cost an additional 0.86% of the recurrent primary education budget to cover polidocente completo schools; and/or an additional 1.03% of the recurrent primary education budget to cover polidocente multigrado schools. Most challenging scenario among regions. Madre de Dios case Small number of polidocente completo and polidocente multigrado schools (12 and 24, respectively). It will require only 6 coaches, but the dispersion of schools could be a limitation. It would cost an additional 1.81% of the recurrent primary education budget. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 36 ELEMENT REPLICABILITY Human resources need Represent 70% of the financial structure of the Capacity Building Component. Loreto and Amazonas require a large number of coaches to cover the schools in Spanish-speaking communities that have not been intervened before. 2.3 WHAT ARE THE HUMAN RESOURCE NEEDS? HOW CAN THEY BE MET GIVEN MONETARY AND TIME CONSTRAINTS? Two issues stand out when discussing human resources for the Amazonía Lee model. First, it is the more resource-intensive input of the Capacity Building component; according to the budget proposal prepared by UPCH (mentioned in EQ 2.2), human resources represented 70% of the capacity building component. Second, the availability of human resources is a challenge due to the qualifications required for coaches and the dispersion of the schools where they would work. In the end, it will depend on the intensity or amount of activities and sub-components of the model that the region would like to support, especially coaching. As a reference, UPCH experienced problems the first year and had to replace some coaches the second year. The first estimates to replicate Amazonía Lee done in EQ 2.2 suggest that in Loreto, 25 professionals would be needed for polidocente completo schools, and 37 coaches are required to cover 50% of polidocente multigrado schools. Amazonas would require 47 coaches to support polidocente completo and polidocente multigrado schools. The case of Madre de Dios is different due to the smaller number of schools, and the estimate is only 6 coaches; however, the dispersion of the schools remains a challenge. There is more than one alternative worth exploring when considering how to incorporate coaches. The different financing/budget and contracting options might save monetary resources in the long term. They are: ● Include as a separate task group and/or, especially for polidocente multigrado schools, as is the tested Amazonía Lee model ● Prepare principals, adjunct principals and the most experienced teachers as potential coaches at school, especially for polidocente completo (KII Soporte Pedagógico School principal) ● As used in previous USAID AprenDes and SUMA projects, support resource centers with coaching responsibilities. These centers could be community centers or large and strong polidocente multigrado schools that are part of a network of neighboring schools. ● Increase the number of pedagogical specialists in UGELs and reduce their operational tasks to enhance the implementation of the Amazonía Lee model. Currently, the educational specialists that provide support to teachers not only have monitoring and pedagogical duties, but also carry out administrative functions. (FGD) Linked with the possible alternatives noted above, there is a potential opportunity to work with human resources from the regions. Amazonía Lee utilized that strategy and the regions were able to implement the project as their staff were also trained on the pedagogical approach, leadership, and school management. As one informant underlined: (KIIs in DRE and UGEL, FGD) 37 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV “The training component is oriented towards the development of people’s competencies and abilities… we are talking about the training of the regional education authorities, the specialists of the Local Education Management Units, the specialists in the Regional Education Directorate, and the teacher trainers and the principals of educational institutions. Because in the case of Amazonía Lee, the part of the teacher training, and the accompaniment in the schools was overseen by the regions, through the regional technical team that was formed by us and the teacher trainers.” – Project Manager Moreover, a Regional Project Coordinator, as a sign of ownership, recognized the educational materials as a regional product due to the region’s close participation in the validation process. The replication of the project could find that other UGELs are not ready for the new tasks, so there is a need for initial capacity building to roll out the program. The project will need an organization to provide technical assistance until the DRE and UGELs are strengthened. According to KII informants, UPCH has shared the pedagogical methodologies and approaches with four universities and pedagogical institutes: including Instituto Horacio Cevallos, Instituto Bilingüe Yarinacocha, Universidad Intercultural de la Amazonía, and Universidad Nacional de Ucayali. The Evaluation Team could not confirm if those institutions were able to adjust their syllabi to incorporate the phonetic approach or would be ready to carry on technical assistance activities, but they represent potential alternatives to support the rest of the regional universities and institutes from outside San Martín and Ucayali regions.36 Table 16 below summarizes this section. Table 16. Summary of Evaluation Question 2.3 HUMAN RESOURCES NEEDS HOW TO MEET GIVEN MONETARY AND TIME CONSTRAINTS Important level of budget resources to allow regions to hire prepared coaches Possible schemes to explore new financing options: task force as Amazonía Lee implemented, school principals strengthened in the approach, and pedagogical and teaching strategies, networking for polidocente multigrado schools, more participation of UGEL educational specialists (need to increase number or decrease administrative tasks) High-level profile of coaches needed Could be covered partially with UGEL education specialist and trained school principals (whose posts are already in the budget, or might be easier to fund) Initial technical assistance to DRE/UGEL for the rollout Conduct a market analysis and consider universities and pedagogical institutes from San Martín and Ucayali for the project rollout. Take advantage of UPCH’s networks (those previously created with Centro Andino, as well as the most recent ones) and trained organizations in the Amazonian regions. 2.4 WHAT STEPS ARE NEEDED TO OPERATIONALIZE THE APPLICABLE COMPONENTS IN OTHER AMAZONIAN REGIONS? If there is interest in other Amazonian regions in introducing the Amazonía Lee methodology, the following steps are recommended, as derived from interviews with key stakeholders and secondary information: 36 UPCH will present the Amazonía Lee experience and educational material at the Latin American Congress “Para el Desarrollo de la Lectura y Escritura: Leer y Escribir para Contribuir al Mejoramiento de la Calidad y Equidad de la Educación en América Latina” (CONLES) 2019 Lima, Peru: September 25-27, 2019. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 38 Identification of stakeholders and promotion of Amazonía Lee It would be advantageous for the Amazonian regional governments and their DREs to start by identifying all relevant stakeholders within MINEDU and MEF, and possibly others, and evaluate each stakeholder’s interests and understanding of the teaching approaches. The regional governments and DREs should then be able to explain what the balanced approach is, and present the costs and benefits of Amazonía Lee, in each case taking into account the interests of each stakeholder. They also need to emphasize that the balanced approach can help to develop the competencies of the National Curriculum by complementing the whole language approach teaching method. This will only be possible if the regional bodies have taken ownership for the model from the beginning. Engagement with national policies Structure a convincing message on how the phonetic approach does not oppose MINEDU’s national policies contained in the National Curriculum and other policy guides, and how easily it is interwoven with the whole language approach. Consider in the message the flexibility of its application and recognize the autonomy of the school principals and teachers. Determination of the geographical area of the model and budget analysis Avoid duplication with other similar strategies implemented at the national level and prioritize a group of schools, based on geographical location (given the dispersion of the population and transportation means), performance levels on the ECE, or a mix of both factors. For example, polidocente multigrado schools that received Amazonía Lee showed improvements in Ucayali, so more polidocente multigrado schools without previous intervention can be selected. Prioritization of sub-components and activities of Capacity Building Define the size of the project considering the parameters of the model, that is, coaches, teachers’ workshops, minimum number of teachers per coach, and minimum number of visits, including when and how to perform a demonstration class. Also, consider necessary educational materials for training teachers and workbooks for students. Repeat the exercise for different possible scenarios and budget availability. Selection of specialized operator to work with UGELs Define if all the UGELs would participate and evaluate their strengths and the availability of staff to lead and monitor the project in their territory. At the same time, identify what institution (university or pedagogical institute from the Amazonian region) could support the selection of coaches (keeping in mind their previous knowledge of the phonetic approach) and train them in the pedagogical strategies and classroom planning, etc. (elements of the Amazonía Lee model). The DREs should clearly define the roles of the operator and the UGELs. Estimation of the quantity of human resources needed Once the schools are identified for potential intervention and budget is analyzed, set the number of coaches needed and reevaluate the financial availability. Also consider the feasibility to identify, select, and hire the proper coaches. The EQ 2.2 presents scenarios with different numbers of coaches, depending on the group of schools selected. The final decision comes from the same region government and DRE. 39 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Consider different strategies to provide coaches to disperse schools, like networking with polidocente multigrado schools, or internal coaches in polidocente completo schools. Include the possibility of providing a blended training program with the online courses of UPCH. Negotiation of a budget increase Cost analysis for budgeting purposes indicates the need to increase the regional budgets to support the implementation of the Amazonía Lee program. It is necessary to evaluate the model of financing and negotiate with MINEDU and MEF, providing background information on the financial feasibility of the proposal. Administrative process in place according to procurement regulations Initiate with the rollout of the hiring process and printing of educational materials. The selection of the technical advisory organization, national or local, must comply with the procurement regulation. The time that these processes take should be considered during the planning and request of additional resources. Table 17 below summarizes this section. Table 17. Summary of Evaluation Question 2.4 CONSIDERATIONS STEP NEEDED TO OPERATIONALIZE AMAZONIA LEE IN OTHER AMAZONIAN REGIONS Stakeholder analysis and ownership Be sure all stakeholders understand what the new teaching approach means and its potential impact in reading performance Engagement with national policies Present the model to MINEDU and emphasize in the message that Amazonía Lee’s approach and methodology are interwoven with the National Curriculum Determination of the geographical area of the model and budget analysis In each region, schools should be prioritized according to performance needs but also based on geographical concentration and dispersion, keeping in mind the financial feasibility Prioritization of capacity building sub-components and activities Once schools are selected, define size of program depending how much of inputs could be managed and financed, especially human resources and educational materials Selection of specialized operator to work with UGELs Evaluate the capacity of the UGELs, search for technical assistance providers, and define roles of each. Esimation of the quantity of human resources needed Set the number of human resources needed based on previous school prioritization and size of components - subcomponents, determine number of coaches needed Determine the need for a budget increase, possible budget negotiations Assess the need of additional budget and proceed to negotiate budget increase Administrative process in place according to procurement regulations Take into account budget calendar and human resources hiring process to rollout the program USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 40 EQ3: TO WHAT EXTENT ARE THE APPLICABLE AND/OR KEY COMPONENTS OF AMAZONÍA LEE SUSTAINABLE? 3.1 WHAT FACTORS CONTRIBUTE TO THEIR SUSTAINABILITY? This section addresses the factors connected to sustainability. They are grouped into institutional sustainability (national rules and regulations affecting Amazonía Lee), economic sustainability (cost considerations), technical sustainability (DRE and UGEL capacity), and others such as motivation and the selection of an opportune moment.37 The cost considerations are addressed further under EQ 3.2. INSTITUTIONAL SUSTAINABILITY Institutional sustainability requires consistency with (or support from) the rules and regulations affecting inputs to the Amazonía Lee model, such as human resources, the national curriculum, and the teaching approach, all guided by MINEDU. Human Resources MINEDU has been implementing a merit system in teachers’ tenure track. The merit framework currently allows for raises, promotions, and sometimes bonuses for teachers and principals. In the future it will include teachers that perform research and coaches; but as of yet, no process has been instituted for coaches (KIIs). This means that while the legal framework is, for the most part, ready to support the key professionals in the sector, it is necessary to advocate for the consideration of coaches when negotiating budgets with the MEF in the short term. Regulations that limit the development of undergraduate programs for primary school teachers due to current excess of supply may jeopardize the intent of institutes offering teaching degrees (Institutos Pedagógicos) to include the discussion of different teaching approaches, including the phonetic approach, in the curricula for new cohorts of teachers and coaches. For example, a professor indicated that an Instituto Pedagógico in Ucayali is training teachers for the kindergarten level only until the ban is lifted. Curriculum and the Teaching Approach Implementation of the National Curriculum is setting the path forward for developing educational competencies during school years, with competencies understood as “the skills, attitudes, and behaviors of a learner that contribute to academic success in the classroom."38 In the current context, MINEDU is working with the regions so that the local actors first have a full understanding of the general framework and the purpose of the curriculum. KIIs and FGD indicated that Amazonía Lee contributes to the strengthening of competencies as identified in the National Curriculum. Currently, according to KII, regions are transitioning from a traditional curriculum based on content to a competency-based curriculum designed by the 37 These factors have been organized in a different conceptual framework in two previous USAID evaluation reports: USAID (2018). Evaluation of Sustained Outcomes in Basic Education. Synthesis Report. USAID (2015) Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. 38 The National Curriculum aims to build 31 competencies, understood as knowledge and skills, including relationship with society and environment, ability to read in the primary language and a second language, among others. 41 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV national government. With the change of government at the regional level, staff that participated in the development of the curriculum together with the national government have changed. Some of the former coaches or coordinators from Amazonía Lee are now Curriculum Implementers, professionals that are usually contracted by the UGELs through yearly Service Administration Contracts (Contrato por Administración de Servicio, CAS). It is important to note that MINEDU’s National Curriculum and classroom practices do not disregard the importance of phonetic awareness. For example, a teacher of a 1st grade class in a school supported by Soporte Pedagógico commented that she also works on practicing the sounds of the letters once her students get familiar with the selected words in a text. KIIs and FGDs also indicated that Amazonía Lee contributed to the strengthening of the competencies identified in the National Curriculum. TECHNICAL SUSTAINABILITY Although technical staff recognize the importance and benefits of the Amazonía Lee intervention and the teaching approach, the project was not able to introduce permanent changes in the regional organizations and processes. When the project was introduced in San Martín and Ucayali, teams were organized in each region and personnel were hired to support the training and administration of project39 . When Amazonía Lee ended in 2017, those contracts were terminated. Currently, pedagogical specialists in UGEL are dedicated to implementing the National Curriculum, that is, to providing oversight and guidance to the schools that are now defining class contents and activities that will build the knowledge, skills and values that the Curriculum intends to develop nationwide. According to KIIs and FGDs, the UGELs do not have the human resources to provide sufficient pedagogical support to schools because they also have administrative responsibilities, manage the teachers’ contracts every year, and resolve cases of failed contracts of teachers, for example. MOTIVATION AND TIMING40 One aspect critical for sustainability is the support of the national government. Therefore, for Amazonía Lee to be sustainable, it is important to frame its ingredients, especially the teaching approach, as a strategy that helps to implement the National Curriculum. ECONOMIC SUSTAINABILITY Cost considerations for economic sustainability are explored in EQ 3.2. Table 18 below summarizes this section. Table 18. Summary of Evaluation Question 3.1 AREAS OF SUSTAINABILITY AND FACTORS CONTRIBUTION TO SUSTAINABILITY Institutional Sustainability Human Resources Role of the coach considered in the teachers’ merit system. Need to advocate for the consideration of the coaches when negotiating budgets. 39 USAID (2018). Evaluation of Amazonia Lee Reading Intervention in Peru: Final Report. Pp B101. 40 USAID (2018) Evaluation of Sustained Outcomes in Basic Education. Synthesis Report. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 42 AREAS OF FACTORS SUSTAINABILITY AND CONTRIBUTION TO SUSTAINABILITY Promote the use of in-service training for teachers with diverse teaching approaches including phonetic approach, while the ban to train primary teachers are still on place Curriculum and teaching approach Amazonía Lee model objectives, methods and teaching approach contributes to the strengthening of students competencies as identified in the National Curriculum. Technical Sustainability Amazonía Lee was not able to introduce permanent changes in the regions’ organization, and the human resources contracts ended or staff returned to previous tasks. UGEL specialists support administrative burden together with pedagogical responsibilities, any Amazonía Lee replication have to consider this fact. Motivation and Timing Amazonía Lee model could be introduced as a strategy to achieve the objectives of the National Curriculum and not as a different approach, will limit the understanding of its benefits in the future if not Economic Sustainability Cost analysis scarce public to support resources economic sustainability in an environment with 3.2 WHAT ARE THE COST CONSIDERATIONS? In an environment with scarce public resources, policy makers at MEF and MINEDU need to assess the best alternative to improve educational outcomes from a cost perspective. Accordingly, the Evaluation Team analyzed the ratio of cost effectiveness (CE) of Amazonía Lee and Soporte Pedagógico to determine how much improvement, or benefit, is derived from each Peruvian nuevo sol spent on the programs. Although Amazonía Lee has a unique and clear difference in its approach to teaching children how to read and write, during the school intervention, a natural comparison group of schools appeared because the MINEDU and regional governments implemented Soporte Pedagógico at the same time. Both programs, Soporte Pedagógico and Amazonía Lee, had similar objectives and general components, and targeted similar polidocente completo schools in urban areas. The parameters used for this calculation are: ● Cost per student: Total amount spent per student in Peruvian nuevos soles41. In the case of programs that have already been implemented, all actual costs should be included.42 The method uses the total budget spent on the project by all of the organizations: UPCH and the regional governments of San Martín and Ucayali in the case of Amazonía Lee, and the MINEDU and the regional governments of San Martín and Ucayali in the case of Soporte Pedagógico. All of the data for Soporte Pedagógico come from the SIAF, identified and submitted by the MINEDU to the Evaluation Team. The main limitation in the case of Amazonía Lee is that the executed budget information did not come from a single source (see the bibliography annex). 41 To increase comparability, the Evaluation Team included the investments spent on Amazonía Lee and Soporte Pedagógico over two fiscal years. 42 In the case of Soporte Pedagógico, the costs of tutoring/training teachers in math are also included. 43 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ● Effectiveness: Net gain on the ECE among 4th graders in 2018. This student population benefited from the project’ previous years in 3rd, 2nd and 1st grades. The net gain or improvement is the difference in points obtained on the ECE’s Reading Comprehension test, in comparison to schools with no support program beyond the usual regional activities. The advantage of this indicator is that is available for both programs (Amazonía Lee and Soporte Pedagógico); the limitation, according to some educators, is that the exam does not capture the full performance of the programs.43 ● CE Ratio: Nuevos soles invested per student in the project per point of net improvement on the 4th grade ECE reading comprehension exam. It is important to underline that the CE analysis is a tool that should be considered with other evaluation instruments. Since the projects had already been implemented, the results of the cost analysis provide opportunities to identify problems of overspending and to look for future solutions. RESULTS According to the analysis of polidocente completo schools, Soporte Pedagógico had a similar level of cost effectiveness as Amazonía Lee in San Martin (with CE ratios of 15.99 versus 14.42, respectively). In Ucayali, Soporte Pedagógico was more cost effective than Amazonía Lee (with CE ratios of 13.14 versus 83.02, respectively). The results were affected by the higher scores obtained on the ECE in Soporte Pedagógico schools (472.72 versus 460.37 in San Martín, and 441.71 versus 417.10 in Ucayali). Although the cost of Soporte Pedagógico was higher in San Martín, this did not offset the results. The analysis shows that the Amazonía Lee schools in Ucayali had the lowest net gain in ECE scores. The low performance in Amazonía Lee schools in this region could have been influenced by contextual limitations. Table 19. Cost-effectiveness analysis comparing Amazonía Lee and Soporte Pedagógico in polidocente completo Schools SAN MARTIN UCAYALI AL SP AL SP 1. Incremental Project Cost, nuevos soles 9,674,147 13,168,991 4,057,648 7,672,669 2. Number of students, 1st 2nd 3rd grade under program, 2017 19,129 17,366 5,443 17,384 3. Cost per student in program, nuevos soles 505.73 758.32 745.48 441.36 4. Average score in 2018 ECE* 460.37 472.72 417.10 441.71 5. Net gain in ECE evaluation 2018** 35.07 47.42 8.98 33.59 6. CE ratio 14.42 15.99 83.02 13.14 43 Other important indicators of effectiveness (dropouts and repetitions, for example) could have been included, but the results on the ECE are the available indicator that best represents the objective of the project. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 44 *AVERAGE SCORE OF 4TH GRADERS ON THE 2018 ECE READING COMPREHENSION TEST **NET GAIN IN ECE EVALUATION COMPARED TO SCHOOLS WITH REGULAR ACTIVITIES AND NO ADDITIONAL INTERVENTION, SAN MARTÍN 425.3 POINTS, UCAYALI 408.12 POINTS ELABORATION: NORC Figure 3. Average Score on ECE Reading Comprehension Test and Net Gain in Scores on ECE 2018 in polidocente completo Schools by Region and Program ELABORATION: NORC In the case of polidocente multigrado schools, the comparison of Amazonía Lee with Soporte Pedagógico was not possible, because Soporte Pedagógico only worked in polidocente completo. However, it is worth reviewing the effectiveness indicator to compare the results of polidocente multigrado schools and polidocente completo schools that received the Amazonía Lee project in both regions. Amazonía Lee showed better results in net gains in scores in polidocente multigrado schools (46.32) than in polidocente completo schools (8.98) in Ucayali. However, in San Martín, polidocente completo schools showed better results (35.07) than polidocente multigrado schools (29.04). In terms of costs, polidocente multigrado schools are more expensive than polidocente completo. The almost 2:1 differential between them is not unexpected and can be explained not only by the low number of students, but also by the higher number of coaches needed in polidocente multigrado (while one coach supports 30 teachers in polidocente completo, they would only be able to support 15 teachers in polidocente multigrado due to lengthy travel times). Table 20. Cost effectiveness results comparing Amazonía Lee in polidocente completo and polidocente multigrado schools SAN MARTIN UCAYALI Polidocente multigrado Polidocente completo Polidocente multigrado Polidocente completo Cost per student 1,058.35 505.73 1477.43 745.48 425.3 460.4 472.7 408.1 417.1 441.7 35.1 47.4 9 33.6 300 320 340 360 380 400 420 440 460 480 500 No Program AL SP No Program AL SP San Martín Ucayali Average Score on ECE Reading Test 2018 Net Gain in Scores on ECE 2018 45 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV SAN MARTIN UCAYALI Polidocente multigrado Polidocente completo Polidocente multigrado Polidocente completo Score points in Students' Census Evaluation 443.53 460.37 400.98 417.10 Net gain in ECE evaluation 2018 29.04 35.07 46.32 8.98 SOURCE: MINEDU, ECE 2018. ELABORATION: NORC MINEDU AND MEF VISION ON HOW TO PROMOTE INNOVATIONS IN EDUCATION The intervention can only be financially sustainable if it is included in the annual recurrent budget of each region. The analysis for EQ 2.2 presented scenarios according to potential priorities and dimensions of the project in each region and the additional resources they would represent. If replication of Amazonía Lee is expected to be financed with a recurrent budget of the regional government (Pliego Regional) and to be approved by the MEF, its policies need to be supported by MINEDU’s national guidelines or be aligned with MEF’s programs of budgetary incentives. For example, the need to increase the average level of learning in the regions might create an incentive to work in polidocente completo schools, which reach a majority of students. Table 21 below summarizes this section. Table 21. Summary of Evaluation Question 3.2 COST AND PERFORMANCE CONSIDERATIONS RESULTS Comparison of cost effectiveness (cost per student in two fiscal years/net gain in scores on the ECE standardized student evaluation) of two programs (Amazonía Lee and Soporte Pedagógico) with the same objective of improving students’ reading performance The analysis of cost effectiveness (understood as how much improvement is derived from each Peruvian nuevo sol spent on each program) post-implementation allows for review and improvement if replicated in the future. Cost effectiveness comparison among polidocente completo schools Soporte Pedagógico is more cost effective than Amazonía Lee in Ucayali, and both have similar cost effectiveness in San Martín. Comparison of effectiveness among Amazonía Lee schools only in Ucayali Polidocente multigrado schools show important improvement - net gains in ECE scores, however polidocente completo schools showed very low improvement Comparison of effectiveness among Amazonía Lee schools only in San Martín Polidocente completo schools showed better results than polidocente multigrado Comparison of costs per student in Amazonía Lee schools As expected, the cost per student is higher in polidocente multigrado schools than in polidocente completo schools due to lesser number of students 3.3 IF THEY ARE NOT SUSTAINABLE, WHAT STEPS WOULD THE MINEDU AND DRES NEED TO TAKE TO MAKE THEM SUSTAINABLE? The following steps would be useful in order to improve the sustainability of the Amazonía Lee program: USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 46 ● Advocate to include the role of the coach in the recurrent budget of the region/UGELs (KIIs in San Martín region). ● In the medium to long term, as USAID expected when the Amazonía Lee project was designed44, the regions would have to hire more full time personnel (“nombrados”) in UGELs to improve pedagogical support to school on a permanent basis. However, this is possible only through negotiating with MEF the creation of posts (“plazas”) to be funded by the fiscal treasury. The process would begin with the regions adjusting their CAP (Cuadro de Asignación de Personal), and then they could follow the usual process they are familiar with when negotiating more plazas for teachers. ● Interested regions and their DREs could begin to institutionalize the ideas of Amazonía Lee. In a KII, a school principal commented that Amazonía Lee was a “good reference” and that the regional government enacted a guideline promoting activities like the Reading Houses and Reading Marathon. However, she also saw some inconsistencies because not all schools had received training on the phonetic teaching approach. Table 22 below summarizes this section. Table 22. Summary of Evaluation Question 3.3 LEVEL STEPS NEEDED FOR SUSTAINABILITY MINEDU and regional governments Advocate to include the role of coaches from the UGEL in the recurrent regional education budget Regional governments Advocate and initiate the process to request budget for full-time civil servants in the role of pedagogical specialists from the UGEL Regional governments and DREs Insert activities tested in Amazonía Lee in regional plans and budget like Reading Houses and Reading Marathons as examples 3.4 HOW CAN THE DRES CONTINUE TO UPDATE AND DEVELOP RELEVANT MATERIALS SIMILAR TO THOSE DEVELOPED UNDER THE PROGRAM? UPCH and DRE Directors confirm that San Martín and Ucayali both have the electronic version of the Amazonía Lee student workbook and that UGEL staff have been trained to use them. However, informants in KIIs commented that not all UGELs are ready to work on the development of new educational materials like those prepared with UPCH, implying that they would need technical assistance. The materials, as per recommendation of UPCH, will not be useful if teachers are not trained to use them with the phonetic approach. Visits to schools revealed that teachers are currently making copies of the workbooks to continue working with the phonetic units. As one school principal pointed out, teachers are charging parents five nuevos soles per month to get the copies.45 This is consistent with survey results in Amazonía Lee schools, in which 9 out of 13 teachers disagreed with the idea that they 44 USAID (2018). G2G in Education Case Study: Peru. 45 This practice is not uncommon in public schools; the visit to Soporte Pedagógico classroom also showed that the teacher was writing exercises in each student’s notebook. 47 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV will not be able to teach using the phonetic approach if they do not receive Amazonía Lee workbooks, while only four agreed or strongly agreed with the statement. During a conversation with principals of Amazonía Lee schools, they noted the importance of adapting books and workbooks to the local context, for example, “working with images of endangered animals from the region is more appealing than showing an elephant.” The Evaluation Team also observed during a classroom visit that the same concepts could be applied to the wall decorations, where images from other geographical areas are still used to identify the letters. Principals from Soporte Pedagógico schools also pointed out that even though MINEDU’s material is relevant, they complement these books with references to their regional context. Table 23 below summarizes this section. Table 23. Summary of Evaluation Question 3.4 EDUCATIONAL MATERIAL ACTION NEEDED TO MAKE RELEVANT Contextualized workbooks and teacher guides in San Martín and Ucayali Regions could print more books, UPCH provided copy for printing New educational material Not all UGELs have the technical capacity to develop new material, so capacity development is necessary. Complement of available materials Schools look for local resources (regional materials) to complement MINEDU books 3.5 WHAT FURTHER STEPS COULD THE MINEDU, DRE, AND SCHOOLS TAKE IN THE FUTURE TO ENSURE THE APPROACH STAYS RELEVANT AND RESPONDS TO THE NEEDS OF THE REGION, TEACHERS, AND STUDENTS? CONTINUOUS ADVOCACY AND RESPONSIVE ENGAGEMENT WITH MINEDU Even though MINEDU showed concern about the regions working independently to develop their own regional curriculum, they were also open and positive about future opportunities to collaborate with the regions. MINEDU is expecting that once the implementation of the National Curriculum starts, processes and activities will begin to adjust and changes will require a lot of work, so there is a need to join forces with regional governments because MINEDU cannot do it alone. The MINEDU also understands that all the regions are not at the same level of development and the question is, according to them, how to work together to improve teachers’ capacities and strategies on how to teach children to read and write. FLEXIBILITY DURING APPLICATION OF THE MODEL The application of the model in San Martín and Ucayali demonstrated that the relative weight of the two approaches depends on the diagnosis of the students and the strategies that the teachers (and principals) decide for their students (individually or as a group). Interviews with school principals on Amazonía Lee and Soporte Pedagógico reveal that institutionalizing Amazonía Lee might also require looking at and incorporating the successful elements of other strategies, such as Soporte Pedagógico, without omitting the phonetic awareness approach. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 48 COLLABORATION AND ENGAGEMENT Holding Forums: During KIIs, informants indicated that the DRE Directors of all Amazonian regions held their first Summit in October 2018. One informant expected that these Directors would remain after elections and advocate for Amazonía Lee since they had already expressed interest in the model. The Evaluation Team found that the previous DRE Directors of San Martín and Ucayali were still working at the DRE as pedagogical specialists and were still interested in the Amazonía Lee model. San Martín and Ucayali also organized Inter-Amazonian Communities (Comunidades Interamazonicas, CIAM), a promising forum for Amazonía Lee intervention, as one pedagogical specialist commented: “Even, we, with Ucayali, we have already set a time and had a meeting of all… In the framework, of the CIAM, to speak about the educational issue. For example, we have united Amazonas, San Martín, Loreto, Ucayali and Madre de Dios. Because we almost have the same characteristics, do we not?” Engagement with other stakeholders from the educational sector. During KIIs, informants commented that the project did not engage with stakeholders that could have had an impact on the education policies. Among those institutions, two stand out, the National Council of Education (Consejo Nacional de Educación, CNE) and Foro Educativo. The CNE is an independent advisory unit to MINEDU, and its Board, which is appointed by MINEDU, is made up of recognized educators, social scientists, and other professionals interested in the education sector. The Council has a very active agenda promoting dialogue around educational policies with civil society organizations, NGOs, and donors such as UNICEF. Currently CNE is closing a long participatory process of dialogue in all regions regarding the priorities of the National Education Project (Proyecto Educativo Nacional al 203646); it is a good opportunity to propose policies and actions to enable the improvement of teacher’s teaching methodologies, like the ones practiced in Amazonía Lee. Foro Educativo is a private association independent from MINEDU that promotes research and discussions around educational policies. There is sometimes an overlap of membership with CNE. Its independence, ongoing publications, and active agenda make it another option to explore. Additionally, a promising opportunity is emerging as UPCH is organizing an international summit on the improvement of teaching and reading skills in September 2019. UPCH will present Amazonía Lee educational materials and analyze the experience (KII). Table 24 below summarizes this section. Table 24. Summary of Evaluation Question 3.5 FURTHER STEPS BY THE MINEDU, DRE, AND SCHOOLS HOW? Continuous advocacy and responsive engagement with MINEDU The DREs would consider MINEDU’s openness to work with regions on curriculum implementation 46 http://www.cne.gob.pe/evento/grupos-impulsores-construyendo-el-proyecto-educativo-nacional-al-2036/ 49 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV FURTHER STEPS BY THE MINEDU, DRE, AND SCHOOLS HOW? Flexibility during application of the model Schools would continue complementing teaching approaches depending on students diagnosis, and other strategies that complement Amazonía Lee model Collaboration and Engagement DRE Directors would maintain the network in regional forums with all Amazonian DREs and other national associations like Foro Educativo, and organizations like CNE 3.6 WHAT STEPS CAN THE MINEDU TAKE TO ENSURE THE DRES FEEL SUPPORTED AND/OR EMPOWERED TO ADAPT THE GENERAL CURRICULUM TO INCORPORATE APPLICABLE KEY COMPONENTS OF AMAZONÍA LEE? One of the main concerns of MINEDU is that working with any approach and scaling it up could encounter obstacles when trying to reconcile it with the philosophy of the National Curriculum. The recommendation from MINEDU is to first understand the National Curriculum, and once the regions take ownership for it, they can make contextual adjustments and adapt the regional curriculum as appropriate. An informant from MINEDU added that the National Curriculum is the primary tool to organize the system and educational materials; in this respect, the books edited by MINEDU follow its objectives. MINEDU’s recommendation is to work top-down, from the National Curriculum to the strategies, and not bottom-up, trying to fit an approach and strategies to the National Curriculum. KIIs in MINEDU revealed that the regions should provide evidence that the Amazonía Lee approach responds to MINEDU’s expectations and does not duplicate or conflict with its policies. In this way, it could even be considered a strategy to accomplish the National Curriculum's objectives. For their part, KII and FGD informants in the regions think that the strategies of the balanced approach contribute positively to developing the National Curriculum’s competencies. Respondents are interested in developing new strategies and not a new curriculum. A school principal commented: “… The balanced approach, as we know it, the phonetic part… always has been geared to the national curriculum; we have not left the national curriculum. We have only tried the strategies that the balanced approach has given us… we know it better by that name… it has helped us to achieve those objectives…. Within the curriculum, we respect the competencies, we respect the performances, we respect everything that the curriculum tells us …” School principal Table 25 below summarizes this section. Table 25. Summary of Evaluation Question 3.6 STEPS BY THE MINEDU AND DREs HOW? Guidance on Curriculum implementation Continue communicating the objectives, methodologies, teaching approaches and establish dialogue on how Amazonía Lee novel practices could develop the National Curriculum’s competencies Negotiation on how to combine teaching approaches or consider strategies Establish the areas where Amazonía Lee elements could be inserted as strategies USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 50 5. RECOMMENDATIONS APPLICABILITY The evaluation assessed the applicability of Amazonía Lee to other regions of Peru, taking into account the different elements of the model such as the pedagogical approach to teach children how to read, project implementation components, and strategies, as well as the characteristics and needs of the Amazonian regions. The Amazonía Lee teaching approach—based on the use of phonetic awareness in combination with the whole language approach—is recommended for Amazonian regions and poor settlements; but some considerations must be taken into account based on the implementation experience: the types of schools where the model was tested and the main language of the communities where the schools are located. ● Any level of government or agency interested in the application of the teaching approach should consider first the type of school where the project was implemented (polidocente completo or polidocente multigrado) and the location of the school (urban or rural), and language of the community when deciding on applicability. Peru’s Amazonian regions have communities with native languages other than Spanish, while the project implemented the components, including the production of learning materials, in Spanish. To explore applicability, the interested region should coordinate with MINEDU’s General Directorate of Intercultural Education (Dirección General de Educación Intercultural, Bilingüe y Rural, DIGEIBIR) before introducing the Amazonía Lee approach in schools where Spanish is not the primary language. DIGEIBIR has issued guidelines in 2018 on how and when to introduce Spanish in schools of these communities. The training modules for teachers have been successfully applied focusing on the methodology for teaching children to read. However, different actors remark that it would be useful to include more topics of the curriculum. ● It is recommended that pedagogical processes for teaching mathematics be included in the workshop curricula and training visits. Similar supporting programs of the central government, such as the coaching activities of PELA and Soporte Pedagógico, also cover the areas of science and humanities in their training workshops. The feasibility of adding more school subjects to the workshops needs to be assessed. Amazonia Lee had distinctive elements supported by an experienced technical assistance agency, which used the opportunity to implement a continuous learning process, in which elements were adjusted and improved and learning materials validated with the same regional stakeholders. The Capacity Building Component was structured around four-day workshops three times a year (120 hours) organized as a sequence with continuous content across the workshops, and concurrent teacher practice with coaching support; in addition to a standard of 30 teachers coached in polidocente completo schools, mainly in urban areas, and 15 teachers coached in dispersed, rural polidocente multigrado schools; and six coaches’ visits during the academic year. ● It is recommended that the same structure of the model and approach, including the main components and the sequence of activities be maintained, but it is suggested that different operational arrangements be developed to implement the components. This is recommended especially for the Capacity Building Component. The inputs, such as the 51 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV number of hours of training and number of coaching visits, could be maintained, but alternative implementation methods could be used, such as online training or blended modalities (online and in-person). Additionally, different types of educational professionals could serve as coaches, including full-time regional specialists from UGELs, school principals, or qualified teachers. REPLICABILITY Under certain circumstances, it is possible to replicate the Amazonía Lee model in other Amazonian regions. Once any region is interested in the teaching approach and project structure, its decision makers must evaluate the region’s ability to manage implementation directly (or contract it out) and its budgetary capacity to finance the project. ● Each region should assess the size of intervention to replicate the implementation of Amazonía Lee model. It is recommended to begin with targeting the number and type of schools that the regions are able to manage and finance with the regional budget. It is suggested to maintain the activities, especially from the Capacity Building Component, and the number of inputs, like coach visits and the number of coaches necessary to cover those visits. If not feasible, regions would need to assess if a smaller number of coach visits still produce the same outcomes like Amazonía Lee or to reduce the number of schools and scale up once the replication has been successful. ● The Amazonía Lee structure requires additional inputs to those of the Soporte Pedagogico model, namely workbooks for teachers and students, and more teacher workshops during the academic year. Additionally, if the model is replicated, stakeholders recommend adding pedagogical training for math and social studies, which might require additional hours during the workshop. Regions that are looking to replicate the model should consider all these factors. The Amazonian regions have geographical challenges; even though some have demographic concentration in a capital city, all Amazonian regions have low density in the rest of their territory in comparison to the rest of country. This issue drives up costs in education and other public services. If the applicable component such as capacity building requires the mobilization of human resources for training and coaching, the distance is a key variable to set the budget. Likewise, distant communities also have smaller schools with fewer students, making the investment per student higher than in urban areas. ● Consider location as a crucial variable in the selection of schools; the dispersion of schools in the Amazonian regions will determine how many schools and coaches the region is able to finance and manage. It is recommended to construct several scenarios with different number, types and locations of schools (which also influence the number and cost of school visits, workshops, and number of teachers per coach, among others) to help assess the cost, availability of human resources and budget, and management capacity. ● Select schools based on cost effectiveness and efficiency. For example, schools with the highest need, as reflected in their poor student performance, could be prioritized. This may even mean that schools supported previously by Soporte Pedagógico or another project could be selected for roll-out of Amazonía Lee. The Cooperative Agreement with UPCH in Amazonía Lee brought more than 10 years of experience implementing pedagogical innovations in several regions in Peru and Andean USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 52 countries. The implementation of Amazonía Lee in the two regions also created a network of professionals that could potentially provide technical assistance in other regions. ● Regional governments, or its bodies like the DRE and UGEL, that are interested in the teaching approach could look for technical support to roll out the Amazonía Lee model. They should perform market analysis to find technical assistance institutions in the Amazonian regions previously trained by UPCH, and also seek advice from UPCH and its network of academic institutions or NGOs with knowledge of the phonetic approach or the balanced approach. It is recommended that the Capacity Building Component be adapted with strategies that can reduce costs. As human resources are the most expensive input of the model, using current full time human resources as coaches not only secures the annual current budget for training and coaching but also could save costs such as transportation and working days. ● Such measures could include the use of internal coaches for large polidocente completo schools (which was tested by Soporte Pedagógico); another appealing option is the use of a Resource Center and networking among scattered polidocente multigrado schools (part of USAID earlier experiences, namely AprenDes and SUMA projects). The replicability of the Capacity Building component could be enhanced through other measures. ● Depending on the size and location of the schools, explore the option for UGELs to work directly with school principals, specifically in the use of the Amazonía Lee workbooks and the word-decoding exercises, together with the readings of stories. ● Schools with large numbers of students per teacher, like Ucayali, should consider the possibility of including the strategy of after-school tutoring (refuerzo pedagógico) to complement the differentiated instruction strategy during classes. ● Prioritize the availability of the workbooks and the teacher guides online. Overall, to initiate the process of replicating the Amazonía Lee approach in a context where central government has an important role in educational and human resources policy, the following steps are recommended for regional governments or other interested agencies: ● Stakeholder analysis and clarification of different approaches to teaching children how to read ● Association of the proposed approach (balanced approach) with national policies, especially with the National Curriculum ● Prioritized geographical areas and type of schools based on cost effectiveness or unitary costs per student ● Prioritization of sub-components and activities of the key Capacity Building Component ● Development of different scenarios and budget estimates ● Estimation of the number of coaches needed 53 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ● Analysis of the market and search for specialized technical operator ● Possible budget adjustment ● Negotiations with MINEDU and MEF SUSTAINABILITY The evaluation assessed to what extent the applicable components were sustainable. The analysis covered factors such as institutional rules and regulations, and economic and technical sustainability. It is important to consider that, although the regional governments represent an important part of educational budget, the central government sets the majority of the educational policies in Peru. ● Regional governments should strengthen the capacity of the regional educational specialists in UGELs to implement pedagogical support activities, especially in the training of the methodologies and the approach on how to teach children to read and write. In addition to being a better option to obtain budget financing, it builds sustainability. ● Regional governments could advocate for the use of a teaching approach that promotes phonetic awareness to improve teaching processes of the National Curriculum. To do this, the regional governments and DREs need to simplify and clarify what “balanced approach” means; it might be helpful to present the approach as a teaching strategy rather than as a different approach from the one currently promoted by the MINEDU.  When presenting the Amazonía Lee model, DREs should emphasize its innovative sub￾components and activities, like the training modules for school principals and regional educational staff. They can underline how the participation of regional educational staff in the design and adjustment of the workbooks created ownership in the regions where it was implemented.  They can mention past USAID projects like AprenDes and Centro Andino as examples as well, taking advantage of the fact that the current MINEDU authorities are familiar with the model and approaches of those projects. ● DREs need to continue the strategic technical discussion at the national and regional levels on the methods of teaching children to read and write. DREs can pursue discussions with actors of the education community, like the CNE and Foro Educativo; UPCH’s network is a strength for this purpose. The Congreso Latino Americano de Escritura y Lectura (CONLES) and other regional fora are additional options to share the approach. ● Regional governments could also advocate for the financing of coaches as part of the teachers’ tenure track (Carrera Pública Magisterial). They can look for opportunities to promote, together with other stakeholders, the function of the coaches, while the education programs from universities and teaching institutes (institutos pedagógicos) improve their curricula and are allowed again to train primary level teachers with better skills in the areas promoted by Amazonía Lee. ● DREs can provide technical assistance to interested schools so they can present the Phonetic Approach as a pedagogical innovation and apply for funds to the Peruvian National Education Development Fund (Fondo Nacional de Desarrollo de la Educación Peruana, FONDEP). Even USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 54 though schools might get a very small amount of financing, it is an important opportunity to showcase the model and its contribution. 55 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ANNEX I. STATEMENT OF WORK OF THE FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE I. PURPOSE OF THE EVALUATION The purpose of this final evaluation is to summarize and validate key findings to provide an evidence base for decision-makers and stakeholders to help inform future education programming, particularly in regions with similar teaching and learning challenges in Peru. The main objective of this performance evaluation is to assess the applicability, replicability, and sustainability of key components of the Amazonía Lee (Amazon Reads) program. The results of this performance evaluation will be presented to the Ministry of Education (MINEDU), the Regional Education Directorates (DRE) of San Martin and Ucayali, the University of Peru￾Cayetano Heredia (UPCH), USAID, and other donors, to help inform future education programming in Peru. II. SUMMARY INFORMATION The performance evaluation will evaluate the overall Amazonía Lee program implemented in San Martin and Ucayali. Amazonía Lee consisted of four mechanisms: ● Liderando los Aprendizajes de Niñas y Niños (LANN) with the regional government of Ucayali (GOREU) ($2.5 million, December 2014–December 2017), Government-to￾Government (G2G) agreement. ● Programa Enseñar Es Liderar (PEEL) with the regional government of San Martin (GORESAM) ($4.4 million, June 2014-December 2017), G2G agreement ● Capacity Development and Engagement (CDE) Cooperative Agreement AID-527-A￾15-00003 with Universidad Peruana Cayetano Heredia (UPCH) ($5 million, January 2015– January 2018). ● Amazonía Lee impact evaluation, implemented through the LAC Reads Evaluation contract by Mathematic (completed January 2018). While the evaluation will assess all components (Figure 1) of Amazonía Lee to identify key elements of the program, special emphasis will be given to the capacity-building component, in particular, the materials and human resources subcomponents. The evaluation team should consider the context in which the project is implemented to interpret and analyze data. III. BACKGROUND A. DESCRIPTION OF THE PROBLEM AND CONTEXT USAID celebrated 55 years in Peru in 2016 and has contributed to the development of the education sector since it began its programming in the country. Beginning in 2001, USAID has focused on improving the quality of education in Peru and supporting implementation of Peru’s 2002 law to decentralize and delegate education (and other) responsibilities to regional governments. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 56 In 2011, USAID released its Education Strategy for 2011-2015 which was recently extended to December 2018. The Strategy’s three objectives are to improve early-grade reading skills, youth workforce development opportunities, and access to education in crisis and conflict. Since 2011, under the Education Strategy, USAID’s Bureau for Latin America and Caribbean (USAID/LAC) targeted the first two objectives through programs such as the regional LAC Reads Program--a suite of programs and mechanisms with partners that work collaboratively to increase the availability and understanding of evidence for improving early grade reading and boosting the capacity of key stakeholders to implement evidence-based, cost-effective practices in early-grade reading. In Peru, under the Educational Strategy and USAID/LAC’s regional approach, education programming has concentrated on improving early-grade learning and reading outcomes in rural and marginalized areas. This programming was a natural continuation of previous education work in Peru which sought to improve the learning of reading and writing in primary school, and strengthen teaching instruction, pedagogical processes, and management capacities of the regional governments and schools as part of Peru’s decentralization process through programs such as Innovaciones en Descentralización y Escuelas Activas (AprenDes), Apoyo a la Reforma de la Educación Básica de Calidad (SUMA), and others. As a result, under the 2012 Country Development Cooperation Strategy, USAID began to implement components of the Amazonía Lee program in January 2015. At this time, USAID is phasing out of the education sector in Peru. As part of closeout, USAID would like to conduct a final performance evaluation of Amazonía Lee to assess the applicability, replicability, and sustainability of the Amazonía Lee program to help inform future education programming in Peru. To further refine the evaluation statement of work and ensure it would be useful to key stakeholders, USAID/Peru met with UPCH, the DREs from Ucayali and San Martin, and Amazonía Lee school directors to refine the evaluation focus and questions. Two overarching themes emerged from these discussions: first, Amazonía Lee has been effective and regional stakeholders would like to continue using the Amazonía Lee approach and methodology; the DREs and UPCH will need to present the program to newly elected and appointed regional and MINEDU officials to ensure key elements of the Amazonía Lee approach are included and approved in the MINEDU’s general curriculum and regional Public Investment Projects (PIP). DRE officials noted that despite decentralization, they often feel the MINEDU develop the general curriculum without input from the regions.47 In this regard, the DREs have expressed that they can be more effective in advocating to the MINEDU if the different Amazonian regions were to unite under one education strategy. This theme further supported the focus of the performance evaluation and questions to assess the applicability and replicability of key Amazonía Lee components in other Amazonian regions. The second major theme to emerge was sustainability. While the extension of the CDE cooperative agreement was focused on fostering sustainability of the program, all parties believe that further steps can and should be taken to further sustainability of the key components of the program. In particular, DREs expressed wanting to know how to ensure the Amazonía Lee approach can be adapted in the future depending on the evolving needs of the region and how to develop and update relevant materials for teachers and students in the years to come. Based on these input, USAID refined the performance evaluation purpose, scope, and key questions. 47 UNDER THE DECENTRALIZATION FRAMEWORK, DRES HAVE THE AUTHORITY TO ADAPT THE GENERAL CURRICULUM DEVELOPED BY THE MINEDU TO THE NEEDS OF THE REGION. 57 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV B. DESCRIPTION OF THE INTERVENTION TO BE EVALUATED In-line with goal one of the USAID Education Strategy, Amazonía Lee sought to improve early￾grade reading skills in two Amazonia regions of Peru: Ucayali and San Martín. As noted in Section II, Amazonía Lee consisted of four mechanisms: ● G2G agreement with the regional government of San Martin ● G2G agreement with the regional government of Ucayali ● CDE cooperative agreement with UPCH ● Amazonía Lee impact evaluation The two G2G Amazonía Lee programs share four main components, described in Figure 1 below, and the same technical assistance provider, UPCH. Following Figure 1 are brief descriptions of the two education G2G agreements and the cooperative agreement designed to provide technical assistance to the G2Gs. FIGURE 1. AMAZONÍA LEE COMPONENTS AND EXPECTED OUTCOMES PEEL The purpose of this G2G was to develop technical and managerial capacities of San Martin’s DRE to increase the percentage of students reading at grade-level standards. The program included: improvement of the quality of teaching, education materials, coaching, training, curriculum and evaluation of learning achievement; strengthening of education systems that support key areas of the quality of teaching and learning; improvement information systems of the DRE and Local Education Management Units (UGELs) in order to use administrative data effectively; sensitization of the education community to support reading and promote behavior change. LANN USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 58 The purpose of the G2G was to leave in place education sector technical and managerial capacities within the regional government of Ucayali to increase the percentage of second grade students reading at grade level, thereby ensuring sustainability of the interventions’ results and local capacity building. The program had the same elements as PEEL. CDE The purpose of this activity was for UPCH to provide expert technical assistance to the regional governments of San Martin and Ucayali to improve the percentage of children performing at grade-level standards in reading. The program included: strengthening the regional curriculum and improving instructional materials for primary school principals, teachers, and students; establishing a customized, demand-driven training program for principals, education specialists, and teachers; strengthening management and administrative leadership of the regional school system by strengthening the capacity of DRE principals and specialists; awareness raising related to reading, gender balance, and assessment; and private sector engagement. The CDE cooperative agreement was originally scheduled to close in December 2017. However, in order to solidify the gains made under Amazonía Lee, USAID extended CDE through July 2018. The priority for the extension period was to implement a sustainability strategy that will institutionalize key elements of Amazonía Lee in the two regions and at the MINEDU and disseminate the experience to ensure demand for sustained support. See Annexes 1-6 for more information on the activities, outcomes, and monitoring and evaluation of all four Amazonía Lee mechanisms. C. PROJECT OR ACTIVITY MONITORING, EVALUATION, AND LEARNING (MEL) PLAN See the CDE Monitoring, Evaluation, and Learning Plan (MEL) in Annex 7. IV. EVALUATION QUESTIONS 1. HOW APPLICABLE IS AMAZONÍA LEE TO OTHER AMAZONIAN REGIONS OF PERU? In answering this question, the following probes (at a minimum) should be considered: ● Which elements of Amazonía Lee are applicable to other Amazonian regions of Peru? Why? ● How similar or different are the needs of teachers and learners in other Amazonian regions to the Amazonía Lee regions? ● To what extent could a single Amazonía Lee strategy be developed for the region while remaining applicable to the needs of each region? Do certain Amazonía Lee elements lend themselves to a single strategy better than others? ● How could Amazonía Lee, or key components of it, be packaged to present to the DRE? 2. HOW CAN APPLICABLE COMPONENTS OF AMAZONÍA LEE BE REPLICATED IN OTHER AMAZONIAN REGIONS OF PERU? In answering this question, the following probes (at a minimum) should be considered: 59 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ● Are there certain key components that are more replicable than others? Why and why not? ● Based on Peru’s decentralization policy, do the costs of the approach enable replicability? Are there certain elements of the model that should be prioritized? ● What are the human resource needs? How can they be met given monetary and time constraints? ● What steps are needed to operationalize the applicable components in other Amazonian regions? 3. TO WHAT EXTENT ARE THE APPLICABLE AND/OR KEY COMPONENTS OF AMAZONÍA LEE SUSTAINABLE? In answering this question, the following probes (at a minimum) should be considered: ● What factors contribute to their sustainability? ● What are the cost considerations? ● If they are not sustainable, what steps would the MINEDU and DREs need to take to make them sustainable? ● How can the DREs continue to update and develop relevant materials similar to those developed under the program? ● What further steps could the MINEDU, DRE, and schools take in the future to ensure the approach stays relevant and responds to the needs of the region, teachers, and students? ● What steps can the MINEDU take to ensure the DREs feel supported and/or empowered to adapt the general curriculum to incorporate applicable key components of Amazonía Lee? V. EVALUATION DESIGN AND METHODOLOGY The evaluation team shall use a mixed-methods approach to answer the evaluation questions, integrating qualitative and survey data and a desk review of program-related materials. The evaluation team shall review program documents including UPCH’s reports, UPCH’s teacher training materials, DRE implementation reports, the Evaluacion Censal de Estudiantes (ECE), the G2G Case Study, the Amazonía Lee Impact Evaluation, and other materials. USAID will provide non-publicly available materials to the evaluation team. This final performance evaluation plan should be informed by and in accordance with ADS 201 and USAID Evaluation Policy so that reliable data will be produced to support evidence-based findings, and be a sound basis for analysis that will lead to conclusions and recommendations that are both useful for assessing the progress to date and capable of providing actionable recommendations relevant to the evaluation’s target audience. The resultant report will adhere closely to the policy’s criteria to ensure the quality of the evaluation report. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 60 The evaluation team will develop and propose a methodology that minimally draws and expands on the following methodologies: ● Primary Data Analyses: − Focus Groups and Interviews: The evaluation team will conduct interviews and/or focus groups with key stakeholders within each region and Lima, including: DRE project coordinators, Amazonía Lee trainers, school directors, teachers, MINEDU staff, and UPCH’s technical assistance team. Interviews will help identify the key elements, achievements, and results of Amazonía Lee as well as assess the institutional capacity transferred to the DRE and the institutional gaps that are necessary to address to make the program sustainable and replicable. A list of key informants will be provided by USAID, but the evaluation team should not necessarily consider that list complete or sufficient for data saturation. The evaluators may interview any other relevant individuals necessary to answer the above research questions. − Cost Analysis: In order to examine the possibilities for replicability and sustainability of the model, the evaluation team should propose a methodology to examine the cost of the model and to determine which elements of the model may be replicable and sustained after the end of Amazonía Lee. The evaluation team should use the cost analysis to make suggestions about how local resources could be used to sustain the most valuable pieces of the program even in a way that may look different from the initial program. ● Secondary Data Analysis / Desk Study: The evaluation team will analyze the data collected by UPCH, Mathematica, the government of Peru, and other sources to answer the key evaluation questions. This data includes, but is not limited to: − 2014, 2015, and 2016 ECEs − G2G Agreements with the regional governments of San Martin and Ucayali − CDE Cooperative Agreement − G2G Case Study − Amazonía Lee Impact Evaluation − UPCH quarterly, annual, and draft final reports − UPCH MEL Plan VI. DELIVERABLES AND REPORTING REQUIREMENTS 1. EVALUATION WORK PLAN Within one (1) week of the award of the contract, the evaluation team shall complete and present a draft work plan for the evaluation to the COR in Word using Gill Sans font size 12. USAID will receive the Evaluation Work Plan via electronic mail and provide comments no later than two (2) working days after receiving the document. 61 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV The work plan will include: ● Draft schedule and logistical arrangements; ● Members of the evaluation team, delineated by roles and responsibilities; ● Evaluation milestones; ● Anticipated schedule and locations of data collection efforts; ● Proposed evaluation methodology including selection criteria for comparison groups, if applicable; and ● Evaluation Report outline (if different from the attached template). The contractor will update the Evaluation Work Plan (the lists of interviewees, survey participants, the schedule) and submit the updated version to the COR on a monthly basis. The Evaluator will be responsible for all logistics support under this SOW, including field office administration, all travel arrangements (with required USAID clearances), team planning facilitation and appointment scheduling, coordination with all partners and stakeholders involved, administrative services (computer support, printing and copying), report editing and dissemination, and compliance with provisions set forth in this SOW. USAID will provide limited support to the Evaluation Team. This support, if needed, may include assistance in arranging high-level meetings; access to the U.S. Embassy compounds as necessary; and access to all reports, data, and other relevant documents created by Amazonía Lee. USAID representatives may accompany the Evaluation Team for some or all the evaluation, especially in high-level meetings. The Evaluation Team is expected to consider this when making logistical arrangements. 2. EVALUATION DESIGN Within ten (10) working days after receiving USAID comments on the Evaluation Work Plan, the evaluation team must submit an evaluation design to the COR in Word using Gill Sans font size 12. USAID will receive the Evaluation Plan via electronic mail and provide comments no later than seven (7) working days after receiving the document after which the evaluation team is expected to return the final evaluation design within seven (7) working days. The design will become an annex to the evaluation report. The evaluation design will include: ● Amazonía Lee description and logic (change theory/development hypothesis); ● Evaluation design, and the explanation of why one design or mix of designs proposed is the most appropriate, its limitations, and how these limitations will be addressed; ● Detailed evaluation design matrix that links the Evaluation Questions from the SOW to data sources, methods, and the data analysis plan; USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 62 ● Data Management Plan describing the capture of data (for example, interview notes or live recording), storage and transfer, and how all data will be handled in such a manner as to protect the identities of informants in any situations where their comments could potentially have a negative impact on their employment or security; ● Draft questionnaires and other data collection instruments or their main features, consent forms48, etc; ● List of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include sampling methodology and methods, including a justification of sample size and any applicable calculations); ● Limitations to the evaluation design; and ● Dissemination Plan (designed in collaboration with USAID). If the Evaluation Plan includes key informant interviews, surveys, and/or focus group discussions, the Evaluation Plan should include the following information: ● How the interviews/surveys will help to answer the evaluation questions; ● Who will conduct the interviews/surveys and why they are qualified to do so; ● What the rationale and methods are for deciding the number, timing, and location of the interviews/surveys; ● How the participants will be selected and recruited; ● How the interviews/surveys will be recorded; and ● How the interview/survey data will be analyzed and presented. Unless exempted from doing so by the COR, the evaluation design will be shared with partner country stakeholders as well as with the implementing partners for comment before being finalized. The data analysis plan should clearly describe the evaluation team’s approach for analyzing quantitative and qualitative data (as applicable), including proposed sample sizes, specific data analysis tools, and any software proposed to be used, with an explanation of how/why these selections will be useful in answering the evaluation questions for this task. Qualitative data should be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. Gender, geographic, and role (beneficiary, implementer, government officials, NGO, etc.) disaggregation must be included in the data analysis where applicable. All dissemination plans should be developed with USAID and include information on audiences, activities, and deliverables, including any data visualizations, multimedia products, or events to 48 IF UNDERAGE PERSONS (LESS THAN 18 YEARS OLD) WILL PARTICIPATE IN THIS PERFORMANCE EVALUATION, THE EVALUATION TEAM MUST MAKE SURE TO COMPLY WITH ALL NATIONAL REGULATIONS RELATED TO CHILD PROTECTION 63 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV help communicate evaluation findings. See the Evaluation Toolkit for guidance on Developing an Evaluation Dissemination Plan. If applicable based on the Disclosure of Conflict of Interests Forms submitted with the awardee’s proposal, the evaluation design will include a conflict of interest mitigation plan. 3. IN BRIEFING Within one (1) working day of arrival in Peru, the evaluation team will meet with the USAID/Peru team for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the SOW, if necessary. 4. EXIT BRIEFING The evaluation team is expected to hold a final exit briefing to discuss the status of data collection and preliminary findings. This presentation will be scheduled as agreed upon in the work plan. 5. FINAL PRESENTATION The evaluation team is expected to hold a final presentation by virtual conferencing software to discuss the summary of findings and conclusions (and recommendations, if applicable) with USAID before delivery of the draft evaluation report. This presentation will be scheduled as agreed upon in the evaluation work plan. 6. DRAFT EVALUATION REPORT The draft evaluation report should be consistent with the guidance provided in Section IX, Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the Evaluation Work Plan. Once the initial draft evaluation report is submitted, USAID will have seven (7) working days in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the evaluation team. The evaluation team may then be asked to submit a revised final draft report within seven (7) working days, and again USAID will review and send comments on this final draft report within seven (7) days of its submission (this second review may be option depending on the quality of the first draft of the final evaluation report). 7. FINAL EVALUATION REPORT The evaluation team will be asked to take no more than seven (7) working days (or as agreed upon in the work plan) to respond to and incorporate final draft evaluation report comments from USAID. The evaluation team lead will then submit the final report to the COR. 8. SUBMISSION OF DATASET(S) TO THE DEVELOPMENT DATA LIBRARY Per USAID’s Open Data policy (see ADS 579, USAID Development Data) the contractor must also submit to the COR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable, non-proprietary format, a copy of any dataset created or obtained in USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 64 performance of this award, if applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. Please review ADS 579.3.2.2 Types of Data To Be Submitted to the DDL to determine applicability 9. SUBMISSION OF FINAL EVALUATION REPORT TO THE DEVELOPMENT EXPERIENCE CLEARINGHOUSE Per USAID policy (ADS 201.3.5.18) the contractor must submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. VII. EVALUATION TEAM COMPOSITION The evaluation team should consist of a Team Leader, a small number of Research Assistants and/or Data Analysts (up to but no more than three). The evaluation team may make use of short-term, task-related consultants as needed and appropriate, given the finalized scope of work and budget constraints. The evaluation team must have an appropriate mix of technical skills to conduct the evaluation. Recognized experts with appropriate academic credentials should have some combination of the following skill sets, in order of priority: (i) deep knowledge of Peru and the LAC region, (ii) strong knowledge of qualitative research methodologies in the social sciences, (iii) strong knowledge of early-literacy work and research, and (iv) adequate knowledge of a combination of public sector processes and policies, or institutional capacity interventions in education ● The Evaluation Team Leader must have: − Full English and Spanish language fluency − At least 10 years of experience leading qualitative research and performance evaluations in education − Appropriate advanced degree (Master's Degree or higher) − Outstanding written and verbal communication skills in English − Superior understanding of and prior experience with education policy work (education and/or other social sectors) in the LAC region − Demonstrated success in program evaluation and report preparation − Outstanding research skills and ability to synthesize large amounts of disparate information into clear, succinct, and readable prose ● Research Assistant(s)/Data Analyst(s) must have: − Full English and Spanish language fluency − At least three (3) years of experience conducting qualitative and quantitative research and/or data analysis 65 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV − B.A. or higher in a relevant field − Demonstrated strong desktop research skills and highly organized In rounding out the evaluation team, the wealth of local expertise that exists in Peru in the areas of education policy, public administration processes, and evaluation and research should be fully explored. The evaluation team should make appropriate use of local expertise for analysis as well as data collection. The contractor must provide information about evaluation team members, including their curricula vitae, and explain how they meet the requirements in the evaluation SOW. Submissions of writing samples or links to past evaluation reports and related deliverables composed by proposed team members are highly desirable. Per ADS 201.3.5.14, all team members must provide to USAID a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project or activity being evaluated (i.e., a conflict of interest form). Proposed key personnel are expected to be the people who execute the work of this contract. Any substitutes to the proposed key personnel must be vetted and approved by the COR before they begin work. USAID may request an interview with any of the proposed evaluation team members via conference call, Skype, or other means. VIII. EVALUATION SCHEDULE The total estimated duration of this evaluation is 12 weeks with 4-6 weeks of fieldwork and data capture. The evaluation team shall propose a more detailed timeline and evaluation plan using these general parameters in the Evaluation Work Plan as required under VI.1 above. IX. FINAL REPORT FORMAT ● Abstract ● Executive summary ● Evaluation purpose ● Background on the context and the strategies/projects/activities being evaluated ● Evaluation questions ● Methodology ● Limitations to the evaluation ● Findings, conclusions, and (if applicable) recommendations ● Annexes See the Evaluation Toolkit for the How-To Note on Preparing Evaluation Reports and ADS 201mah, USAID Evaluation Report Requirements. An optional Evaluation Report Template is also available in the Evaluation Toolkit. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 66 The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be 2–5 pages and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions (plus recommendations and lessons learned, if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methods (e.g., in sampling; data availability; measurement; analysis; any potential bias such as sampling/selection, measurement, interviewer, response, etc.) and their implications for conclusions drawn from the evaluation findings. Annexes to the report must include: ● Evaluation SOW (updated, not the original, if there were any modifications); ● Evaluation methods; ● All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● All sources of information or data, identified and listed; ● Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable; ● Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest; and ● Summary information about evaluation team members, including qualifications, experience, and role on the team. X. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity; ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly; ● The Executive Summary should present a concise and accurate statement of the most critical elements of the report; ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID; ● Evaluation methodology should be explained in detail and sources of information or data properly identified; 67 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ● Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions; ● Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence; ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females; and ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Checklist and Review Template from the Evaluation Toolkit for additional guidance. XI. OTHER REQUIREMENTS All modifications to the required elements of the SOW of the contract/agreement, whether in evaluation questions, design and methodology, deliverables and reporting, evaluation team composition, schedule, and/or other requirements will be agreed upon in writing by the COR. Any revisions made will be noted in the SOW annexed to the final Evaluation Report. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 68 ANNEX II. DATA COLLECTION INSTRUMENTS IN-DEPTH INTERVIEW ADMINISTERED TO MINEDU’S REGULAR BASIC EDUCATION DIRECTOR AND PEDAGOGICALSPECIALIST APPLICABILITY ● In your view, what are the challenges and limitations that a change in the teaching methodology faces in Peru? − Probe: What knowledge do you have about Amazonía Lee (AL) and the phonetic approach? − Probe: What do you think about applying this approach in the Amazonian regions of Peru? ● [For officials who know Amazonía Lee] Based on your knowledge observing the material of Amazonía Lee and the approach, do you think that it can be adapted and used as part of the package of materials MINEDU assigns to all schools? [Same question regarding coaching, and if coaching promoted by MINEDU also incorporates mathematics] − Probe: Can we consider Amazonía Lee materials as additional to MINEDU's materials? Or does it completely replace the communications materials in all three grades (1st to 3rd)? − Probe: Does MINEDU includes Amazonía Lee schools in the list of schools who receive MINEDU materials? REPLICABILITY ● [For officials who know Amazonía Lee] With the current national curriculum, which reflects the possibility of regions to adapt the curriculum to their context, what elements of Amazonía Lee could the regions incorporate? − Probe: According to the regulations, regions have to submit a proposal of the adapted or improved regional curriculum. Is the teaching approach also considered as a possibility? ● What elements could contribute to the Amazonía Lee approach being replicable to other Amazonian regions? ● In the current socio-political context of Peru [changes of regional governors], do you think that it would be possible to introduce the Amazonía Lee approach? Why or why not? ● What steps should MINEDU take to operationalize Amazonía Lee in other Amazonian regions? − Probe: Does MINEDU have to approve or give clearance to the contents of educational materials if they are linked to the teaching methodology/approach? − Would MINEDU continue to deliver educational materials to schools if only part of the content of the curriculum (i.e. Communications) is changed? − Probe: Is there a procedure for the delivery of additional materials to the UGEL? 69 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV − Probe: What would be necessary for the MINEDU to incorporate an element of teacher training articulated with the Amazonía Lee approach? SUSTAINABILITY ● Amazonía Lee has shown positive outcomes, and MINEDU also supports innovation. What is your opinion about the possibility of funding this initiative, especially materials, with MINEDU’s recurrent budget, and what could be the possible steps for DREs and UGELs? ● Are there cases where MINEDU has supported teaching innovations and supported the institutionalization of innovations at the regional or local level? What were the steps the regions took to insert the changes in their management and planning documents? ● If Amazonía Lee is applied to the Amazonian regions, what kind of strategies could MINEDU use to achieve the sustainability of Amazonía Lee? − Probe: How could the sustainability of Amazonía Lee be achieved in regional contexts where there is high mobility of human resources and lack of funds for materials? IN-DEPTH INTERVIEW ADMINISTERED TO MINEDU DECENTRALIZED MANAGEMENT OFFICE AND PLANNING AND BUDGET OFFICE APPLICABILITY ● In your view, what are the challenges and limitations that an education project -implemented by a regional government- would face if the project supports a different teaching approach from the one supported by MINEDU at the national level? [Elaborate on the differences] − Probe: What level should be responsible for setting the teaching approach in the country? ● Considering the decentralization policy and levels of distribution of responsibilities among levels of government, do you think that the DREs of the Amazonian regions could apply Amazonía Lee? [Elaborate about the elements of the project] − Probe: What level of government has responsibility for establishing the contents of the books? − Probe: What do MINEDU and regional governments understand by "flexible regional" curriculum? REPLICABILITY ● In the current socio-political context of Peru [changes of regional governors] do you think it would be possible to introduce the Amazonía Lee approach or other innovations? Why or why not? ● What steps could MINEDU take to operationalize Amazonía Lee in other Amazonian regions? − Probe: What is the procedure MINEDU has for the production and delivery of additional materials or other materials to the UGEL? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 70 − Probe: Are there institutional limitations for a region, in keeping with decentralization policies, to replicate Amazonía Lee in their regions? SUSTAINABILITY ● Given that Amazonía Lee has shown positive outcomes and MINEDU also supports innovation, what is your opinion about the possibility of funding this initiative with MINEDU recurrent budget and what could be the possible steps for DREs? ● Are there cases where MINEDU has supported teaching innovations and supported the institutionalization of innovations at the regional or local level? What were the steps the regions took to insert the changes in their management and planning documents? ● If Amazonía Lee is applied to the rest of the Amazonian regions, what kind of strategies could MINEDU use to achieve the sustainability of Amazonía Lee? − Probe: How could the sustainability of Amazonía Lee be achieved in regional contexts where there is high mobility of human resources and lack of funds for materials? IN-DEPTH INTERVIEW ADMINISTERED TO USAID PROJECT MANAGER OR DIRECTOR APPLICABILITY ● What were the challenges that USAID experienced during negotiations and during implementation of Amazonía Lee in regard to the application of a new teaching approach? And how were these challenges confronted? − Probe: Did MINEDU set conditions to maintain its applicability together with the whole language approach? − Probe: Was there a discussion about the possibility of implementing the project in other regions? − Probe: Did USAID advocate for the new approach with international evidence, including experience in other Spanish-speaking contexts? ● Was there an understanding about how the materials of Amazonía Lee would be introduced and the need to complement MINEDU’s educational materials (teaching, teacher’s training and school management)? − Probe: Was MINEDU concerned about the connection of the approach with the rest of the educational and school management materials? REPLICABILITY ● Does the agreement with UPCH limit the use of the educational materials (produced or used in the project) in the future only to San Martín and Ucayali? Or can the materials be used in other regions? 71 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV − Probe: Provide comments on the complete process (copyright, editing, peer review, clearance from national and regional specialists, and layout), and not only reproduction and distribution. ● Do you know of other institutions or organizations (other than UPCH) that could provide the regions technical assistance on the phonetic approach? ● After the project closed, did the Regional governments, DREs and UGELs – in San Martín and Ucayali – indicate that they are willing to continue the application of the Phonetic Approach or a variation like balanced approach in their schools? − Probe: USAID mentioned in the Scope of Work that DRE representatives got together to look at the possibility of replication in their regions. Did representatives of other Amazonian regions participate? ● Based on your experience implementing Amazonía Lee successfully in San Martín and Ucayali, what would be the lessons learned to help introduce the approach to the rest of the Amazonian regions regional governments? − Probe: What kind of support would they need from MINEDU, and what support would they no longer need? − Probe: What other types of support will they need and where might they get it? SUSTAINABILITY ● Based on USAID experience supporting decentralization efforts in several sectors in Peru, what lessons might be applied to the respective roles of central and regional governments in replicating a new policy in the education sector? − Probe: Do you have any suggestions for how best to engage the central government (MINEDU) going forward on a change of policy? ● What problems did regional governments and UPCH encounter in contracting full time employees in regional governments to deal with the new teaching methodologies? − Probe: Were regional governments able to obtain MINEDU and MEF approval of their staff as new civil servants -- as that implies human resources benefits and pension payments? − Probe: Do these employees have additional responsibilities or were they exclusively employed in order to maintain the new teaching methodology? − Probe: Besides the PIP -investment project feasibility study- did the regional governments, DREs or UGELs mention the possibility of institutionalizing with current and not investment budget? ● What strategy did USAID or UPCH use with other stakeholders in the education sector to advocate for the Amazonía Lee project and to advance the institutionalization of the phonetic approach? − Probe: How about coordination with the National Education Council or another key ally? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 72 − Probe: How about coordination with FONDEP (Fondo Nacional de la Educación Peruana) advocating for teaching innovations? ● Does USAID have a strategic approach to work with the newly-elected regional government authorities? Alternatively, will the strategy be directed or mediated by the national government? − Probe: Given that MINEDU has launched an intensive campaign about the application of the national curriculum with DREs and UGELs nationwide at the beginning of the 2019 school year, what would be the possible implications to sustain the phonetic/balanced approach? IN-DEPTH INTERVIEW ADMINISTERED TO USAID EDUCATION SPECIALIST APPLICABILITY ● In your view, what are the challenges and limitations does an education project -supporting a change in the teaching methodology like Amazonía Lee- face in countries like Peru? ● What were the factors that led the Mission to support the implementation of Amazonía Lee in the Amazonian regions of Peru and not other regions? − Probe: Did the Mission have discussions with MINEDU about the possibility to apply phonetics in bilingual settings in Amazonian regions [like use with Awajun] or Quechua in the Sierra region? − Probe: What were the considerations the Mission took regarding the types of schools or preferred areas (like urban instead of rural)? ● Could you elaborate about the agreement with MINEDU to apply a blended approach combining the phonetics approach with the “whole language” approach? ● During conversations with MINEDU’s authorities or specialists – before or during the implementation of the project- what were their concerns about the implementation of the approach, mainly regarding MINEDU’s policies, and concerning the school and classroom organization? − Probe: During the conversations with MINEDU before project implementation, how did USAID foresee the work of teachers combining materials of Amazonía Lee with the ones MINEDU sends to schools nationwide, like planning guides and Mathematics educational materials? REPLICABILITY ● In regards to the Amazonía Lee project components, what elements do you think can be separated from the project package and be implemented without the support of the other elements? − Probe: Can the specialized Amazonía Lee coaching for teachers be separated from the specialized Amazonía Lee materials? What other elements can be replaced by the current MINEDU inputs in Soporte Pedagógico? Do you think Amazonía Lee does not need tutoring 73 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV for kids performing below the class average? Can this be integrated into the Amazonía Lee package? ● Based on your experience in other educational projects, and in the success achieved in San Martín and Ucayali, what lessons are there on introducing the approach to the rest of the Amazonian regional governments, − Probe: Do you think that maybe they have to renounce to MINEDU technical assistance (like the coaching workshops that foster “whole language” approach”). SUSTAINABILITY ● Considering USAID’s experience supporting the implementation of similar social projects – what are the key factors that can encourage central and decentralized governments to introduce policy changes, especially in very sensitive areas such as teaching methods? ● Based on USAID’s experience supporting decentralization efforts in several sectors in Peru, what are the findings that USAID has regarding the distribution of responsibilities of educational policies among the levels of government? − Probe: What level should be responsible for setting the teaching approach in the country? − Probe: What level of government has the responsibility of setting the contents of the books? − Probe: What do MINEDU and regional governments understand by "flexible regional" curriculum? ● Does USAID have a strategic approach for working with the new regional government authorities? Also, is it expected that work with them would be directed or mediated by the national government? REQUEST FOR INFORMATION/DOCUMENTS/MATERIALS: ● USAID reference, research evidence-based, of the benefits of the phonetic approach over other approaches/methodologies, which is used to support technical assistance ● Agreement with MINEDU concerning the application of the balanced approach ● Any recommended study laying out responsibilities among levels of government in education in Peru (e.g., PRODES, MINEDU, CNE) ● Copy of regional curriculums with UPCH inputs IN-DEPTH INTERVIEW ADMINISTERED TO AMAZONÍA LEE PROJECT DIRECTOR AND PEDAGOGICALSPECIALIST, UPCH OVERVIEW ● What are the main components of the whole Amazonía Lee project? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 74  Probe: What are the main differences between the phonetic approach and the balanced approach and the methods traditionally used in MINEDU and in Soporte Pedagógico MINEDU?  Probe: What are the main similarities between the phonetic approach and balanced approach and the methods traditionally used in MINEDU and in Soporte Pedagógico MINEDU? APPLICABILITY ● Is the model easier to implement in certain types of schools?  Probe: Has the model been applied to all types of schools? [unidocente, multigrado, polidocente]  Probe: Given that the government is promoting a policy of bilingualism, can the model of teaching to read, using phonetics, be applied to other languages in the Amazonian regions? [be ready for examples: Awajun, other]  Probe: Has the model been applied to schools located in rural areas?  Probe: Can you provide examples of difficulties? REPLICABILITY MATERIALS ● Please describe UPCH’s work on the contents and design of the books for the project. Did some of the materials and/or some of the content already exist or were they created for the project? ● Approximately, how many weeks/months did it take to deliver the books to schools, from the creation of the content to the actual distribution to schools? ● Did the content of the books require review and/or clearance from MINEDU or DRE or UGEL? ● What do you think were the advantages and disadvantages of UPCH and the Amazonía Lee Project in comparison to MINEDU in the design, editing, clearance, printing and distribution of books and materials? − Probe: In your opinion, what was the hardest part of producing the teaching material? − Probe: Could the regions reproduce/print the books or does someone else hold the copyright? − Probe: How long did it take to edit and print the books? − Probe: How long did it take to distribute the books and other materials? 75 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV SPECIFIC QUESTION FOR THE LOGISTICS STAFF OR ADMINISTRATOR: ● Did UPCH contract the layout and printing? How long did this task take? [get the dates with examples] ● Please describe the process of distribution. HUMAN RESOURCES ● Did the specialists that trained coaches (during project implementation) work at the university or were they associated with the regional government DREs or UGELs? ● What is the profile of the pedagogical specialists you were looking for? ● What is/was their hiring status? ● Based on your knowledge, was it easy to get candidates for coaches during the implementation of the project? How long did it take? ● In your opinion, what was the hardest part of contracting human resources like specialists and coaches? ● [For the pedagogical specialist, if any at the UPCH] If you need to prepare a specialists, how many hours of training were needed? [You can organize the conversation by modules if it is easier] ● [For the pedagogical specialist, if any at the UPCH] Was it easy to find coaches in the regions? How long did it take? ● What is the profile of the coaches you were looking for? − Probe: What would be the main reasons why the coaches left the project? [These could be the answers, do not direct, it is going to be triangulated with FGD with coaches: wanted tenure, distance of schools, schedules and logistics for accommodation if went to rural schools, did not understand or liked the phonetic approach, went to Soporte Pedagógico] SUSTAINABILITY ● What is your opinion about MINEDU’s approach and openness to making changes in the national curriculum and the method of teaching how to read? ● What are your expectations for the future in regards to including the phonetic strategy as an option for teaching how to read and to be blended with MINEDU’s approach ● Are there other universities with graduates with the knowledge and skills you would need to act as a specialist or coaches? REQUEST FOR INFORMATION/DOCUMENTS/MATERIALS: ● List of schools implementing the phonetic approach under Amazonía Lee. ● Financial statement by component and activities and specific information on the costs of the complete production process of books and materials. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 76 ● Documentation with information about the type of contracting of human resources: specialists and coaches ● Videos of demonstration classes IN-DEPTH INTERVIEW ADMINISTERED TO TEACHING MATERIALS SPECIALIST, UPCH APPLICABILITY ● What was the process to adapt the contents of the books to the needs of the Amazonian regions? Please provide comments for: educational and guides materials for teachers during coaching, educational and guides materials to be used in the classroom by teachers, books and working notebooks for students. − Probe: Was there a need to edit the content of existing materials or did UPCH have to design a completely new set of materials? − Probe: If the materials had to be designed, how long did it take to produce the concept (or content) of the materials? − Probe: Did the UPCH team aim to design book content that can transition to a more blended approach (like a balanced approach) or that can be connected with the rest of MINEDU’s materials? − Probe: Do educational materials go through a process of review, editing, and clearance in the UPCH? Or some type of peer review? How long did it take? − Probe: How long did the lay-out and production (printing) take? REPLICABILITY ● To what extent do you think it will be necessary to edit or adjust materials in case other regions besides San Martín and Ucayali want to apply the phonetic approach? ● How were the books used by UPCH distributed? ● What was the role of MINEDU, DRE or UGEL in the clearance of the books and other materials? − Probe: Does USAID or UPCH hold the copyright to the educational materials produced? ● What do you think were the advantages and disadvantages of UPCH and the Amazonía Lee Project in comparison to MINEDU in the design, editing, clearance, printing and distribution of books and materials? SUSTAINABILITY ● Would it be possible to have approximate values (costs) of the services and materials used to produce the educational materials? ● Does UPCH have a timeframe of durability of books? For example, three years? 77 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV REQUEST FOR INFORMATION/DOCUMENTS/MATERIALS: ● List of the materials produced by the UPCH, for training specialist, for coaches, for teachers, and guides for the classroom, books for the children and guides for families. ● Observe the materials in office of interviewee, check quality of materials, number of pages, sings of copyrights. ● Financial documentation that shows costs of educational materials. IN-DEPTH INTERVIEW ADMINISTERED TO REGIONAL PROJECT COORDINATORS IN REGIONS APPLICABILITY ● To what extent was the project implemented in this region? ● Is the model easier to implement in certain types of schools? − Probe: In your region, was Amazonía Lee implemented in all type of schools, from polidocente to unidocente? − Probe: Was there any school you couldn’t reach because of distance? Any other reason? REPLICABILITY HUMAN RESOURCES ● Was it easy to find specialists / coaches (formadores) in the regions? ● In your opinion, what was the hardest part of contracting human resources like specialists and coaches? ● Was there a difference between the coaches who had to go to remote schools than the ones that go to urban schools? ● Did some of the coaches have previous experience in coaching financed by PELA? How was their performance? MATERIALS ● Where were all the books and workbooks produced? ● What was the process of approval of the contents, including the role of the UGEL and/or the DRE? ● How and by whom were the books distributed? SUSTAINABILITY ● How would you describe the willingness and ability of the DRE and UGEL to accept changes in the curriculum and the method of teaching how to read during project implementation? Currently, how would you describe their willingness? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 78 ● What do you think could have been done differently -during project implementation - to advance the work with MINEDU and incorporate the phonetic approach as an option in MINEDU’s strategies? ● Based on your knowledge, what could the DRE and UGELs do right now to incorporate the phonetic approach in the MINEDU teaching strategies? ● What is your opinion regarding the possibility of using some of the books and materials for teachers – such as guias metodológicas - prepared by MINEDU in the phonetic approach or combined balanced approach? ● To what extent are DRE and UGEL ready to manage the training and coaching process? ● Are there other universities or institutes in the region that might have graduates with the knowledge and skills needed to act as a coach with Amazonía Lee profile? And would they be available to be hired by the regional government units? REQUEST FOR INFORMATION/DOCUMENTS/MATERIALS: ● Please describe the training process. Can you tell us about the modules of workshops or courses and number of hours required? ● Modules for specialists ● Modules for coaches ● Modules for teachers ● Visits and demonstration class sessions (videos) ● Can you provide us with figures on the costs of your activities? [Obtain the amount spent in each training model, including lodging, food, premises, transportation, etc] − Probe: Did the project finance the contextualized materials (material visual y concreto)? − Probe: How much was the average cost to travel to schools outside the city? − Probe: How far did the coaches have to travel to get to the project schools? − Probe: Who paid coaches? Who paid specialists? IN-DEPTH INTERVIEW ADMINISTERED TO GENERAL MANAGER OF REGIONAL GOVERNMENT OR DRE APPLICABILITY ● Based on your knowledge, are there any schools in the rest of Amazonian regions, other than San Martín or Ucayali, similar to the ones supported by Amazonía Lee which could be part of an intervention similar to Amazonía Lee? ● What is your opinion regarding the capacity of other DREs and UGELs in Amazonian regions, others than San Martín or Ucayali? 79 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV REPLICABILITY ● Was it necessary to enact rules and regulations specified for the project? ● What were the special arrangements you had to make in your institution to implement or supervise the project? − Probe: Did you have to make organizational changes to accommodate the project? Or, were the professionals monitoring the implementation of Soporte Pedagógico or coaching financed by PELA also in charge of the Amazonía Lee project? ● To what extent is it feasible for the DRE to replicate this project in the future? ● Did you observe if some UGELs performed better than others with regard to the implementation of the project? What were the factors that may have facilitated better performance? SUSTAINABILITY ● One of the objectives of the project was to institutionalize it within the Region. What are the challenges in doing that? − Probe: What are the challenges to committing full time personnel (nombrados) to working for a project? − Probe: What are the challenges to creating full time posts in your region? What would be the institutional and legal matters that your institution has to follow? ● [For DRE only] Was your institution involved in the review, clearance, and/or approval of educational materials? − Probe: If yes, what were the challenges? If not, do you think you should have been? ● Is the DRE a Budget Operational Unit? (Unidades Ejecutoras)? Are all UGELs Budget Units? Or is there any UGEL that participated in Amazonía Lee that is only an operational unit? ● Was the Budget and Planning Office of the regional government involved in the transfers of funds for project activities to the UGELs? Was this a standard process or was it different? − Probe: Did you observe any difficulty financing the activities of the Amazonía Lee? ● Is the Investment Office of the regional government preparing a Feasibility Project for replication? What are the challenges this office has in regards to this PIP? − Probe: Was the PIP approved and when will the funds be available? IN-DEPTH INTERVIEW ADMINISTERED TO BUDGET AND PLANNING OFFICER OF REGIONAL GOVERNMENT APPLICABILITY USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 80 ● What are the challenges of financing a type of innovation for teaching at the schools – like Amazonía Lee- with sources from an institutional investment project (PIP) and not with current budget as in the other areas at schools? − Probe: Can the challenges and limitations be overcome for future initiatives in other Amazonian regions? ● What is your opinion regarding the administrative capacity of investment, budget and planning offices in other Amazonian regions to manage investment projects like Amazonía Lee? REPLICABILITY ● During Amazonía Lee implementation, what were the main budgetary items covered by the PIP? ● What were the challenges coordinating the availability of funds from the PIP -financing Amazonía Lee- with the rest of the current budget? − Probe: Any problems with the payment calendar? Missed payments of teachers, coaches? − Probe: Were there challenges at the school level? (Where part of inputs were financed with current resources, and part with investment funds) ● If you had to choose between financing innovations with investment project funds or with current budgetary funds, which one you would prefer? Why? SUSTAINABILITY ● Considering your knowledge and experience, were there cases where investment projects like Amazonía Lee were institutionalized and began to be financed with current sources from the annual budget? ● During Amazonía Lee implementation, was there any relevant difference - in structure or costs – between the current funds and PIP funds when financing schools or other educational activity? ● Was there any difference in terms of authority for decision making or flexibility using funds from an investment project like Amazonía Lee and recurrent funds? ● If DREs or UGELs would have to produce books, what part of the process could/would have to be procured following the national regulations of procurement? − Probe: Assuming that the design and content is done institutionally, does your region have regional or local contractors in the printing industry large enough to produce books for school primary grades? ● Is your office preparing or participating in the design of a second PIP for Amazonía Lee or similar activities? 81 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV IN-DEPTH INTERVIEW ADMINISTERED TO CHIEF OF UGEL OR PROFESSIONALS THAT HAD THAT POST DURING AMAZONÍA LEE PROJECT APPLICABILITY ● Based on your knowledge, are there any schools in the rest of Amazonian regions, other than San Martín or Ucayali, similar to the ones supported by Amazonía Lee which could be part of an intervention similar to Amazonía Lee? ● What is your opinion regarding the capacity of other DREs and UGELs in Amazonian regions, others than San Martín or Ucayali? REPLICABILITY ● Was it necessary to enact rules and regulations specified for the project? ● What were the special arrangement you had to make in your institution to implement or supervise the project? − Probe: Did you have to make organizational changes to accommodate the project? Or, were the professionals monitoring the implementation of Soporte Pedagógico or coaching financed by PELA also in charge of the Amazonía Lee project? ● To what extent is it feasible for the UGEL to replicate this project in the future? SUSTAINABILITY ● One of the objectives of the project was to institutionalize it within the Region. What are the limitations and challenges in doing that? − Probe: What are the challenges to commit full time personnel (nombrados) to working for a project? − Probe: What are the challenges to create full time post (nombrados) in the regions? What would be the institutional and legal matters that your institution has to follow? ● Is your UGEL a Budget Operational Unit (Unidad Ejecutora)? Do you undertake procurement for the schools? ● Did you receive funds from the regional governments for the project? ● Was your institution involved in the clearance of educational materials? ● If not, do you think you should? If yes, what were the challenges? ● Do UGELs have to get clearance from MINEDU to produce teaching materials? − Probe: What happens with all the teaching guides sent by the MINEDU, are they sent to Amazonía Lee too?? REQUEST FOR INFORMATION/DOCUMENTS/MATERIALS: ● Type of contract and salaries of specialists and coaches USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 82 ● Sample of improved curriculum with the inserted improvement of Amazonía Lee IN-DEPTH INTERVIEW ADMINISTERED TO PEDAGOGICAL SPECIALIST AT DRE AND UGEL APPLICABILITY ● To what extent was the project implemented in this region? ● Is the model easier to implement in certain types of schools? − Probe: In your region, was Amazonía Lee implemented in all types of schools, from polidocente to unidocente? − Probe: Was there any school you couldn’t reach because of distance? Any other reason? ● What were or are the challenges to apply Amazonía Lee in a context where all the rest of the educational content and calendar is guided by MINEDU? ● What were or are the challenges to coordinating the regional curriculum with the national curriculum, especially if the majority of educational materials are furnished centrally? REPLICABILITY ● How was the phonetic approach inserted into the regional curriculum? How was it possible to establish a new method or blended approach like the balanced approach? HUMAN RESOURCES ● In your opinion, what was the hardest part of contracting and training human resources like coaches? − Probe: Was it easy to find coaches (formadores) in the regions? ● Was there a difference between the coaches who had to go to remote schools and the ones that go to urban schools? (especially in terms of pedagogical skills and knowledge) ● Did some of the coaches have previous experience in coaching financed by PELA or part of the Soporte Pedagógico? How was their performance? ● Were all teachers happy with the change of the teaching approach? What were the concerns teachers raised to oppose a change towards a phonetic approach? ● Can you describe the type of training received by coaches and teachers? Please explain the cascade process: UPCH specialist to DRE and UGEL specialist, UPCH specialist to coaches, coaches to teachers. − Probe: Are you still participating in coaching to teachers in the whole language approach financed by PELA or as part of Soporte Pedagógico? ● How many hours of training do you think teachers applying Amazonía Lee got during coaching? Is that different from the “whole language” approach coaching? 83 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV MATERIALS ● What was the process of approval of the contents, including the role of the UGEL and/or the DRE? ● Were teachers and students able to use MINEDU’s books to complement the phonetic approach? SUSTAINABILITY ● What is your opinion of the intensive training the MINEDU is rolling out on the national curriculum? And what could the DRE and UGELs do right now to incorporate the phonetic approach in the MINEDU teaching strategies? − Probe: Do you think it is going to limit or interfere with the plans to replicate and sustain the phonetic approach? ● Have you updated your regional curriculum for 2019 or have you maintained the same curriculum while Amazonía Lee was being implemented? ● What else do you think could have been done - during project implementation - to advance the work with MINEDU and incorporate the phonetic as an option for MINEDUs strategies? ● What is your opinion regarding the possibility to use some of the books and materials for teachers- guias metodológicas - prepared by MINEDU in the phonetic approach or combined balanced approach? ● To what extent are DRE and UGEL ready to manage the training and coaching process performed during Amazonía Lee implementation? ● Are there other universities or institutes in the region that might have graduates with the knowledge and skills needed to act as specialist and coach with Amazonía Lee profile and be available to be hired by the regional government units? ● Please describe the training process. Can you tell us about the modules and number of hours required? (Obtain printed information if possible) − Modules for specialists − Modules for coaches − Modules for teachers − Visits and demonstration class coaches (videos) ● Can you provide us with figures on the costs of your activities? [Obtain the amount spent in each training model, including lodging, food, premises, transportation, etc.] − Probe: Did the project finance the contextualized materials (material visual and concreto) − Probe: How much was the average cost to travel to schools outside the city? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 84 − Probe: How far did the coaches have to travel to get to the project schools? IN-DEPTH INTERVIEW ADMINISTERED TO SCHOOL PRINCIPALS UNDER AMAZONÍA LEE PROJECT APPLICABILITY ● Based on your experience with the Amazonía Lee Project, what is your opinion about the benefits of the teaching “phonetic approach” in how to read in comparison to the MINEDU’s supported “whole language” approach? ● To what extent did the Amazonía Lee project enrich the phonetic approach and create the balanced approach? If yes, why and how? − Probe: Is it because teachers and students can also use all or part of MINEDU educational materials in teaching how to read? Or other reasons? ● Have you observed or received information or comments from your teachers (1st, 2nd, and 3rd grades) on how the phonetic approach interacts or complements with the rest of the MINEDU’s curriculum? − Probe: How do teachers work with mathematics? Do they use MINEDU’s books and workbooks? − Probe: Do teachers receive coaching for Math teaching in other events? ● What is your opinion on the possibility of applying Amazonía Lee components to all type of schools? [refers to unidocente, multigrado and polidocente schools] ● What is your opinion on the possibility of applying Amazonía Lee with the phonetic approach in other Amazonian regions? Are the schools similar to yours? REPLICABILITY ● What is your opinion about the materials that your school received from Amazonía Lee? Are they used exclusively for the phonetic approach or are they used complementing with the whole language approach? − Probe: Do teachers combine educational materials from Amazonía Lee and MINEDU? − Probe: Do your teachers use the planning and schools guides from MINEDU? − Probe: Would you please show us the materials of each source [Amazonía Lee and MINEDU, Soporte Pedagógico if different] in the library or classroom? − Probe: Do you have pictures or videos of the classroom materials and walls in your files for 1st, 2nd, and 3rd grades? − Probe: Is your school receiving MINEDU’s materials for Communications this 2019 school year for 1st, 2nd, and 3rd grades? 85 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ● What is your opinion about Amazonía Lee training events for you and your 1st, 2nd and 3rd grade teachers, including like coaching for teachers, demonstration classes, and workshops for principals? − Probe: Do you think they accomplish the same purpose as the events of Soporte Pedagógico? − Probe: Please tell us about the training that you received. How many hours did it last? − Probe: How many hours of coaching did the teachers from your school receive? [Number of hours, number of teachers] − Probe: How often do teachers have the “grupos de inter-aprendizaje”? − Probe: How many demonstration classes did teachers have? [For example 1 or 2 per teacher, number of teachers that had the demonstration class] − Probe: Do you have videos of demonstration classes? − Probe: How do you think that teachers that received coaching from Amazonía Lee perform in comparison to teachers who only received PELA or Soporte Pedagógico support? SUSTAINABILITY ● What is your opinion about the possibility of continuing the use of the phonetic approach? − Probe: Do you think the DRE and/or UGEL would support you? − Probe: Do you think MINEDU [through Soporte Pedagógico] could cover the area of teachers support? And workshops for principals? ● What is your opinion about the durability, in terms of number of calendar years, of the educational materials prepared and provided by Amazonía Lee? − Probe: Can you compare it to the durability of books provided by MINEDU? ● Do you have challenges to continue with the phonetic approach, like any new rule or regulations from DRE, UGEL or MINEDU that promotes the whole language approach? − Probe: Have you been required to participate in the online or face-to-face training in the national curriculum for the 2019 school year replicated by the UGEL? IN-DEPTH INTERVIEW ADMINISTERED TO SCHOOL PRINCIPALS UNDER SOPORTE PEDAGÓGICO SUPPORT APPLICABILITY ● Have you heard or are you familiar with the phonetic approach, which is different from the whole language approach supported by MINEDU? ● If yes, what is your opinion about the benefits of the phonetic approach in how to read in comparison to the MINEDU’s supported “whole language” approach? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 86 ● Do educational materials (you receive from MINEDU) for the Communications subject complement the rest of material of the curriculum (like the materials of Mathematics)? Or can they be used independently in 1st, 2nd, and 3rd grades? REPLICABILITY ● Would you please show us the materials your school receives from MINEDU [and Soporte Pedagógico if different] in the library or classroom? − Probe: How are the materials of planning and classroom organization used? − Probe: Do you have pictures or videos of the classroom materials and walls in your files for 1st, 2nd, and 3rd grades? − Probe: Is your school receiving MINEDU’s materials for Communications this 2019 school year for 1st, 2nd, and 3rd grades? ● Please tell us about the training you have received from Soporte Pedagógico? How many hours was the training you received? − Probe: How many hours of coaching did the teachers from your school receive?[Number of hours, number of teachers] − Probe: How often do teachers have the “grupos de inter aprendizaje”? − Probe: How many demonstration classes did teachers have? [For example 1 or 2 per teacher, number of teachers that had the demonstration class] − Probe: Do you have videos of demonstration classes? − Probe: How many hours of tutoring do teachers provide per week or month? [Discuss the annual projection of hours with the Principal, and the size of tutoring groups] − Probe: How do you think that teachers that received coaching from Amazonía Lee perform in comparison to teachers who only received PELA or Soporte Pedagógico support? SUSTAINABILITY ● What is the durability of your educational materials (like books and workbooks) in terms of calendar years? [Books last three years, workbooks last one year?] ● Is your school participating in the online or face-to-face training about the national curriculum for the 2019 school year replicated by the UGEL? 87 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV RAPID SURVEY FOR TEACHERS Region: ______ School supported by Amazonía Lee ____, or supported by Soporte Pedagógico or coaching financed by PELA ____ Please provide the following information: a. Gender: F_______ M_______ b. Years of teaching experience: ____ c. Years of experience teaching 1st, 2nd, or 3rd grades: _______ d. What grade or grades are you teaching? _____ e. What grade(s) did you teach last year? _____ About your participation in previous events of formación docente: f. How many formación docente workshops did you attend last year? 0 1 2 3 4 [Answer G only if F is greater than zero] g. On average, how many hours did each workshop last? Please estimate. Number: [Answer H only if F is greater than zero] h. The workshops allowed me to improve my ability to: Mark all that apply Organize and planning my classes Design the content of my classes Understand MINEDU’s educational policies/guidelines Understand DREs or UGELs’ educational policies/guidelines Teach children to read Teach Math and Science Use or improve my use of educational material Other: Please explain briefly i. During the last year, how many times has a coach visited you? Number: [Answer J only if I is greater than zero] j. During the last year, how many of those visits were to observe your class and provide technical advice? Number: [Answer K only if J is greater than zero] k. The technical advice received after class observations was useful in helping me to: Mark all that apply Organize and plan my classes Design the content of my classes Understand DREs or UGELs educational policies/guidelines Teach children to read Teach math and science Use or improve my use of educational material Other: Please briefly explain USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 88 l. During the last year, how many demonstration classes did you receive? Number: [Answer M only if L is greater than zero] m. The demonstration classes allowed me to improve my ability to: Mark all that apply Organize and planning my classes Design the content of my classes Understand MINEDU’s educational policies / guidelines Understand DREs or UGELs’ educational policies/ guidelines Teach children to read Teach Math and Science Use or improve my use of educational material Other: Please explain briefly [FOR SCHOOLS WITH AMAZONÍA LEE SUPPORT ONLY] Please let us know if you agree with the following statements, being 5 “totally agree” and 1 “totally disagree”. n. It is easy to differentiate between the “phonetic approach” from the “whole language” approach when teaching children how to read Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 `I had been successful teaching children how to read before I learned about the “phonetic approach” or received Amazonía Lee support Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 o. I have combined the “phonetic approach” to my previous experience with the “whole language” approach Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 p. I observed my students improving their reading skills with the phonetic approach Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 q. My students would be behind in comparison to today if we had continued with the whole language approach Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 r. I can use and incorporate MINEDU’s reading materials into the phonetic approach Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 s. I can use and complement MINEDU’s educational material of Mathematics and Science with the Communications/Reading material of the “phonetic approach” Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 89 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV t. If I do not receive educational material to apply the phonetic approach I will not be able to teach with this approach in the future Totally disagree 1 2 Neither agree or disagree, 3 4 Totally agree 5 Thank you! USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 90 CLASSROOM VISIT GUIDE REMINDER FOR THE OBSERVER Explain the specific objective of the visit. Ask permission to take pictures of the materials, books and decorated walls. You would like to know the use of materials in connection with the activity she/he organizes every day. For first and second grade classroom, take time to talk about the books and workbooks and other materials they use for: (1) teaching guides (2) communications and reading subjects. Explain to the teacher that you would like to differentiate among the materials they receive from Amazonía Lee, Soporte Pedagógico program, and other sources like the books in the classroom libraries sent by MINEDU. It does not matter the year in which the school received the material if the teacher is still using the material. Also it is important to know: what are the materials each student needs, what books and materials are shared in the classroom and materials shared in the school. As you go around the classroom, between stations, ask these questions: ● Is it possible to use part of Soporte Pedagógico or MINEDU’s books and workbooks in the phonetic approach? ● Is the classroom using only the Amazonía Lee materials for reading and Soporte Pedagógico or MINEDU for Mathematics and other areas? − Probe: What is the biggest challenge to manage her/his strategies with the phonetic approach? − Probe: What are enabling factors that have supported the use of the phonetic approach? Find below a list of materials you can check during the interview or immediately after you leave the classroom. To facilitate the observation, differentiate the materials that belong to each student, materials shared in the classroom and materials shared in the school. The list also includes a list of small equipment. 91 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV TABLE 1. CLASSROOM VISIT CHECKLIST CATEGORY USED BY NUMBER DURATION (MONTHS) FROM AMAZONÍA LEE FROM SOPORTE PEDAGÓGIC O OTHER FROM MINEDU VALUE PER UNIT (S/.) MATERIALS FOR STUDENTS Curricular Area Textbooks Package Per Student Student Notebooks/ worksheets Package per student Student Reading plan book package Package for school Student Fungible school package for student Package per student Student STUDENT MATERIALS FOR CLASSROOM AND SCHOOL Concrete module material for Communication Package Classroom Concrete material module for Mathematics Package Classroom Fungible material package for the classroom Package per classroom Classroom Physical education module Package Classroom Art/music/theater module Package Classroom Civic education module Package Classroom USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 92 MATERIALS FOR SCHOOL Basic school library Package of 14 books Schools Media library materials Package Schools Concrete module of special needs Package Schools MATERIALS FOR TEACHERS Basic library for teachers, Package of 17 books Schools Teacher’s guide: CI, LM, CA, PS, package Classroom Curricular area textbooks, Package Classroom Notebooks / worksheets, Package Classroom PEDAGOGICAL EQUIPMENT Classroom Furniture Classroom Classroom Computers, Innovation Module Classroom TV Schools DVD Schools Photocopier Schools 93 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV MATERIALS FOR SCHOOL Radio Recorder Schools Multimedia Projector/ Screen Schools Photographic Camera Schools Package-Outdoor Games, Package Schools USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 94 FOCUS GROUP GUIDE TO BE ADMINISTERED TO AMAZONÍA LEE COACHES ABOUT THE APPROACH AND COMPONENTS ● What are the main components of the whole Amazonía Lee project? − Probe: Share your perspectives about the phonetic approach, the balanced approach and the methods used traditionally in MINEDU and in Soporte Pedagógico. − Probe: Do you have any opinions or suggestions? APPLICABILITY ● How well has the model been applied? Why? − Probe: Do you believe the phonetic approach would be easier to implement in certain types of schools? Why? − Probe: What are your thoughts on bilingualism? In rural areas? REPLICABILITY MATERIALS ● What are your perspectives on the contents of the materials and books? − Probe: What are your perspectives on the design of the materials and the books? − Probe: How were the materials created? Did UPCH support? If yes, how? − Probe: What is the possibility of the regions reproducing the materials and books? − Probe: Are there more you would like to share about the materials and the books? HUMAN RESOURCES ● Was it easy to get candidates for coaches like you in your region, with the Amazonía Lee profile? How long did it take? ● If you need to prepare somebody for being a coach with Amazonía Lee profile, how many hours of training are needed? ● How was the relationship between the specialist and the coaches? Do you think they were prepared to teach the phonetic approach? ● How was the relationship between the coaches and the teachers? Was there any reluctance to be coached in the new phonetic approach? SUSTAINABILITY ● What is your opinion about MINEDU’s approach and openness to changes in the national curriculum and in the method of teaching how to read? − Probe: What are your expectations for the future? 95 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV − Probe: Are there other universities with graduates with the knowledge and skills you would need to act as a specialist or coaches with Amazonía Lee profile? USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 96 ANNEX III SCHEDULE OF KEY INFORMANT INTERVIEWS, FOCUS GROUPS AND CLASSROOM VISITS NAME TITLE INSTITUTION DATE KII / FGD / CV Marisol Perez Pedagogical Specialist USAID 03/25/2019 KII Martín Castro Project Manager USAID 03/25/2019 KII Raquel Villaseca Former Project Director UPCH 03/26/2019 KII Mercedes Karina Ortiz Former San Martín Education Regional Director San Martín Regional Government / MINEDU 03/27/2019 KII Cecilia Ramirez Former Director of Regular Basic Education Ministerio de Educación 03/29/2019 KII Miriam Choy Project Manager USAID 03/29/2019 KII Wilson Guerrero Former Project Coordinator Amazonía Lee San Martín Sam Martín Regional Government 04/01/2019 KII Jorge Vazguez Pedagogical Specialist UGEL Moyobamba 04/01/2019 KII Wilson Quevedo DRE Specialist - Amazonía Lee Coordinator DRE San Martín 04/02/2019 KII Genaro Nuñez Principal Director of Amazonía Lee school IE N° 00479 Juan Clímaco Vela Reyes 04/03/2019 KII SCHOOL 1 Amazonía Lee (PEEL) school IE 00479 Juan Clímaco Vela Reyes 04/03/2019 CV Tedy Rojas José Lozada Virinic Lucana Arnaldo Lopez Formadores DRE San Martín 3/04/2019 FGD SCHOOL 2 Soporte Pedagógico IE 00475 María Lizarda Vásquez López 04/04/2019 CV Rodolfo Carrasco Principal Director of Soporte Pedagógico School IE N° 00475 María Lizarda Vásquez López 04/04/2019 KII Roxana Atoche Former San Martín Regional technical UPCH 04/04/2019 KII 97 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV NAME TITLE INSTITUTION DATE KII / FGD / CV team – Pedagogical Specialist Juana Tello Ucayali Education Regional Director DRE Ucayali 04/08/2019 KII Ángela Villacorta Former Ucayali Education Regional Director DRE Ucayali 04/08/2019 KII Jose Wagner Former Project Coordinator Amazonía Lee Ucayali DRE Ucayali 04/09/2019 KII Blanca Franco Former San Martín Regional technical team – Pedagogical Specialist UPCH 04/09/2019 SCHOOL 3 Amazonía Lee (PELA) school IE 65099 Nuevo Amanecer 04/10/2019 CV Mercedes Pacaya Principal Director of Amazonía Lee school IE Nuevo Amanecer 04/10/2019 KII Elena Catayama Head of the Pedagogical Management Area UGEL Coronel Portillo 04/102019 KII Judith Davila Karen Cardenas Tedy García Bnajamin Zavala Formadores DRE Ucayali 04/10/2019 FGD SCHOOL 4 Soporte Pedagógico School IE 64012 Miguel Grau 04/11/2019 CV Auristela Torres Principal Director of Soporte Pedagógico school IE Nuevo Amanecer 11/04/2019 KII USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 98 ANNEX IV. DOCUMENTS CONSULTED Amazonía Lee. Mejoramos los aprendizajes en Lectura y Escritura en la Educación Primaria. Brochure. Edited by UPCH. AprenDes. Acompañamiento pedagógico: Haciendo camino al andar. Bulletin. AprenDes Edition Nº 16 Year 4 / 2008 June - July. https://pdf.usaid.gov/pdf_docs/PNADX612.pdf Balarin, M. (2016). Evaluación del proceso de implementación de la intervención de Soporte Pedagógico del Ministerio de Educación del Perú. Grupo FORGE, GRADE, Global Affairs Canada. Chinen, M., & Bonilla, J. (2017). Evaluación de Impacto del Progarma de Soporte Pedagógico del Ministerio del Educación del Perú. Grupo FORGE, GRADE, Global Affairs Canada. Ferrer, Guillermo (2007). Enseñanza de la lectura y escritura. Edited by UPCH, Centro Andino, and USAID. FONDEP (2018). Recomendaciones de política para la innovación educativa. FONDEP. Institucional Brochure. Cofinanciando innovación para una mejor educación. FONDEP. Brochure on co-financing. Instituto Nacional de Estadística e Informática (June 2018). Perú: Crecimiento y distribución de la poblacion 2017. Primeros resultados. Instituto Nacional de Estadística e Informática (2018). Perfil Sociodemográfico 2017. Ministerio de Educación (2017). Curriculum Nacional de la Educacion Básica 2016. Ministerio de Educación (2018). Censo Educativo 2018. Ministerio de Educación (2019). Evaluación Censal de Estudiantes (ECE) 2018. Ministerio de Educación (2015). Qué y cómo aprenden nuestros estudiantes. Área Curricular, Comunicación III ciclo, 1ro y 2do grado de Educación Primaria. In: Rutas del Aprendizaje. USAID, Ministerio de Educación (2007). Los materiales educativos. AprenDes. Una experiencia replicable. Propuesta Pedagógica Integral y Apoyo a la Descentralización Educativa No 3. https://pdf.usaid.gov/pdf_docs/PNADX645.pdf USAID (2010). Evaluación de los Programas en Educación de la USAID/Peru: AprenDes y Centro Andino. Summary Evaluation Report. USAID (2018). Evaluation of Sustained Outcomes in Basic Education. Synthesis Report. USAID (2018). Evaluation of Amazonía Lee Reading Intervention in Peru: Final Report. Mathematica Policy Research. USAID (2018). G2G in Education Case Study: Peru. 99 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV USAID and Universidad Peruana Cayetano Heredia. (n/d) Centro Andino de Excelencia para la Capacitación de Maestros. Una experiencia internacional exitosa y replicable. USAID and Universidad Peruana Cayetano Heredia (2017). Desarrollo de Capacidades y Compromiso: Amazonía Lee. Annual Report 2017. USAID and Universidad Peruana Cayetano Heredia (2018). Desarrollo de Capacidades y Compromiso: Amazonía Lee. Final Report, July 2018. AGREEMENTS Implementation Letter No. 527-0426-U-01, under Assistance Agreement No. 527-0426 for the Amazonía Lee 2014-2017 Project. December 2014. USAID SOAG No. 527-0423, USAID SOAG No. 0426, Implementation Letter No. 527-0423- SM-19, Implementation Letter No. 527-0426-SM-08. December 2014. Letter to UPCH with Cooperative Agreement AID-527-A-15-00003. Capacity Development and Engagement (CDE). January 2015. FINANCIAL INFORMATION MEF MEF. Sistema Integrado de Administración Financiera. Consulta Amigable de Ejecución del Gasto. http://apps5.mineco.gob.pe/transparencia/mensual/default.aspx?y=2018&ap=ActProy GORESAM Oficio 474-2016-GRSM/ORA. May 2016. Formatos para Liquidación No 01 de Fondos del Proyecto “Enseñar es Liderar”. January-March 2016. Public Voucher for Purchases and Services other than Personal. Oficio 700-2016-GRSM/ORA. July 2016. Formatos para Liquidación No 02 de Fondos Proyecto “Enseñar es Liderar”. April - June 2016. Public Voucher for Purchases and Services Other Than Personal. Oficio 1140-2016-GRSM/ORA. October 2016. Formatos para Liquidación Parcial No 03 de Fondos Proyecto “Enseñar es Liderar”. July - August 2016. Public Voucher for Purchases and Services Other Than Personal. Oficio 1268-2016-GRSM/ORA. November 2016. Formatos para Liquidación Final No 04 de Fondos Proyecto “Enseñar es Liderar”. July - August 2016. Public Voucher for Purchases and Services Other Than Personal. Dirección Regional de Educación. 01 de setiembre al 31 de diciembre 2016. Convenio de donaciones No. 527-0423. Estado de Rendición de Cuentas Recursos USAID. Presupuesto del año 2016 por Proyecto “Proyecto Ensenar es Liderar”. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 100 Dirección Regional de Educación. 01 de enero al 31 de diciembre 2017. Convenio de donaciones No. 527-0423. Estado de Rendición de Cuentas Recursos USAID. Presupuesto del año 2017 por Proyecto “Proyecto Enseñar es Liderar”. GOREU Contraloría General de la República. Informe 771-2017-CG/AFI-AF. Auditoría Financiera al Convenio de Donación 527-0426 Financiado por USAID. Estado de Rendición de cuentas Amazonía Lee: Liderando los Aprendizajes de Niños y Niñas 2014-2017. January 1 – December 31, 2016. Contraloría General de la República. Informe 922-2018-CG/SUE-AF. Auditoría Financiera al Convenio de Donación 527-0426 Financiado por USAID. Estado de rendición de cuentas Amazonía Lee: Liderando los Aprendizajes de Niños y Niñas 2014-2017. January 1 – December 31, 2017. UPCH USAID. Summary Executed Budget per year. June 2015- July 2018. 101 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ANNEX V. KEY TEAM MEMBERS BETTY ALVARADO DE LOZAN, TEAM LEADER Ms. Alvarado is a senior development specialist with more than 20 years of experience in social sectors (education and health), governance, decentralization, and urban development, primarily in Peru. She is specialized in policy design and project cycle, including ex-ante evaluation, monitoring and ex-post evaluation, and has conducted multiple studies and evaluations in Peru in areas such as education, health, nutrition, and decentralization. She has served in long and short-term posts as technical leader and manager, in national and international organizations, and as researcher and university professor. Ms. Alvarado has proven experience in project management of USAID financed projects, having served as Director of Finance and Decentralization in Education under USAID’s Quality Basic Education Reform Project in Peru, the predecessor project to Amazonía Lee. In addition, she has expertise conducting applied research for policy analysis and analyzing data through quantitative and qualitative methods. Ms. Alvarado has extensive experience designing and administering different tools and guides for data collection such as surveys, in-depth interviews, participatory evaluation, focus groups, and process analysis. Moreover, she has expertise executing strategies for advocacy and constructive engagement with authorities and public officials from different levels of government, academia, nonprofit and private sectors. With her rich experience in both government and international donor programs, Ms. Alvarado has a sound understanding of the context for all the interviews. Ms. Alvarado holds a Master of Public Administration (Fulbright Scholar) and Master of Philosophy from New York University. KATHARINE MARK, PROJECT DIRECTOR Katharine Mark has more than twenty five years of international experience in evaluation, performance management, and local government, working in more than 20 countries in Latin America, Africa, Eastern Europe, and Asia. Ms. Mark currently manages NORC’s role in USAID mission-wide M&E&L Activities in El Salvador and Senegal. In El Salvador, she has managed evaluations of a higher education project, a regional trade program, and an impact evaluation of USAID’s justice and crime prevention place-based strategy, and a review of the education sector. She recently led multi-country performance evaluations of the IDB’s New Employment Opportunities (NEO), a regional youth workforce development project in Latin America, and of USAID’s global Electoral and Political Process Strengthening project. Ms. Mark has provided extensive technical assistance and training on performance monitoring and evaluation. In Peru, Ms. Mark conducted a review of the Government’s 2009 Budget for Results initiative, focusing on education, health and nutrition, providing recommendations on measures to facilitate effective roll-out to regional and local government, and earlier, made presentations to the Ministry of Economics and Finance (MEF) and to Congress introducing results-based governance. Ms. Mark has extensive experience in local government and decentralization issues. She is currently working on local government projects in Honduras and Liberia, providing specialized support in monitoring and evaluation. Earlier, she worked on a project to help the government of El Salvador to develop a decentralization strategy with a focus on education, health, and water. In Ethiopia, she directed the World Bank-funded Deepening Decentralization project in 2005-2006 to assist regional governments and local governments to develop capacity in key areas such as municipal service delivery, financial management, housing and land management, performance management and urban development. In Honduras she directed the local government component of USAID’s Greater Transparency and Accountability in Government project. She has worked on other USAID local government USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 102 projects in Albania. Bosnia, Bulgaria, and Croatia. Between 1991 and 1999 she served as Chief of Party for a decentralization, local government, and housing project in Hungary including activities on increasing local transparency and accountability, infrastructure development, financial management, housing policy, and housing finance. Ms. Mark holds an MSc in Economics from the London School of Economics. CARLA MARTÍNEZ, QUALITATIVE RESEARCHER Ms. Martínez is a Peruvian anthropologist with over twelve years of experience researching and managing social projects in education, health, social inclusion, gender, job performance evaluation, peasant economy, and informal economy with vulnerable populations in Peru’s coastal region, Andean Highlands, and Amazonian area. In recent years, she has used qualitative methods to obtain evidence to support public policy through in-depth, semi-structured and focused interviews, focus group, group discussions, participant observation, semi-structured observation, anonymous observation, and participatory workshops. She has participated in research projects and evaluations regarding incentive tools for teacher performance management, the incorporation of gender in state social programs, and rural education management in Peru and Honduras. At the Superintendencia Nacional de Educación Superior Universitaria in Peru, Ms. Martínez Ramirez worked on the methodological design, supervision, systematization, and analysis of the adaptation of the University Law No. 30220. Ms. Martínez Ramirez holds a Masters in Methodology of Social Science Research from Universidad Complutense de Madrid in Spain, and a B.S. of Social Science with a major in Anthropology from Universidad Nacional Mayor de San Marcos, in Lima, Peru. INGRID ROJAS ARELLANO, PRINCIPAL RESEARCH ANALYST Ms. Rojas is a Principal Research Analyst in the International Programs Department of NORC at the University of Chicago. She has more than six years of experience conducting research in a wide range of sectors, including education, labor, security, justice, health, agriculture, transport, telecommunications, electricity, and water and sanitation. Ms. Rojas’ work has included research and evaluation projects in Latin America and Africa. Ms. Rojas has expertise conducting surveys and analyzing large datasets to carry out performance and impact evaluations. When assessing the effects of development interventions, Ms. Rojas has used both qualitative and quantitative methods. She has experience conducting in-depth interviews and focus groups, as well as using experimental and-quasi experimental techniques. Ms. Rojas is currently supporting performance and impact evaluations of a literacy and gender-based violence project, including design of survey instruments and data analysis. She has also evaluated the impact of a conditional cash transfer program on labor and educational outcomes, and designed a school-to-work transition pilot project in Mexico. Ms. Rojas’s experience in Peru includes diagnosing the education and health sectors, and developing a 5-year plan to increase labor productivity in the country, for the Competitiveness National Agenda 2014 – 2018. She has also developed problem and objective analysis, and formulated multiannual investment plans for the education sector, as part of her production of Development Plans for Peruvian rural communities. In addition, Ms. Rojas has conducted fieldwork in the coastal, Andean and Amazonian regions of the country. Furthermore, she has interviewed representatives of sub￾national governments as well as beneficiaries of social protection programs in each of these regions. A native Spanish speaker, Ms. Rojas is also fluent in English. She holds a Master in International Development Policy from Georgetown University, a Master of Public Administration from Syracuse University, and B.S. in Economics from Universidad del Pacifico, in Lima, Peru. 103 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV ANNEX VI. DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Betty Alvarado de Lozan Title Project Lead Organization NORC at the University of Chicago Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) Contract No.: ID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Evaluation of Amazonía Lee, Peru I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 104 Name Betty Alvarado de Lozan projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 06/06/2019 105 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Name Katherine Mark Title Principal Research Scientist Organization NORC at the University of Chicago Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) Contract No.: ID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Evaluation of Amazonía Lee, Peru I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 106 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 06/06/2019 107 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Name Carla Martínez Title Qualitative Researcher Organization NORC at the University of Chicago Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) Contract No.: ID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Evaluation of Amazonía Lee, Peru I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 108 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 06/06/2019 109 | FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU USAID.GOV Name Ingrid Rojas Arellano Title Principal Research Analyst Organization NORC at the University of Chicago Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) Contract No.: ID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Evaluation of Amazonía Lee, Peru I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. USAID.GOV FINAL PERFORMANCE EVALUATION OF AMAZONÍA LEE IN PERU | 110 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 06/06/2019