This publication was produced at the request of the United States Agency for International Development. It was prepared independently by DevTech Systems, Inc. with support from SSG Advisors d.b.a. Resonance, and was authored by Kristin Rosekrans (Team Leader), Daniela Cardozo, Ana Roxanda Rodriguez Cabrera, Mario Alas, and Cefas Asencio. USAID/LRCP (FEREMA) EVALUATION Midterm Performance Evaluation: The Latin America and Caribbean Reads Capacity Program September 2019 Disclaimer: This report is made possible by the support of the American people through the United States Agency for International Development (USAID) under Contract Number GS-10F-0048L/7200AA18M00024. The contents of this report are the sole responsibility of DevTech Systems, Inc. and do not necessarily reflect the views of USAID or the United States Government. MIDTERM PERFORMANCE EVALUATION: The Latin America and Caribbean Reads Capacity Program September 30, 2019 Submitted to: Barbara Knox-Seith, Contracting Officer’s Representative USAID Bureau for Latin America and the Caribbean, Office of Regional Sustainable Development (LAC/RSD) Submitted by: DevTech Systems, Inc. Contract GS-10F-0048L/7200AA18M00024 (October 1, 2018–September 30, 2023) Cover photo: First grade children in Tegucigalpa, Honduras read together as part of the campaign supported by LAC Reads Capacity Program called ¡Leer Nos Cambia la Vida! Photo by USAID/LRCP Fundación para la Educación Ricardo Ernesto Maduro Andreu ABSTRACT The Latin American and Caribbean (LAC) Reads Capacity Program (LRCP) is a six-year program initiated in 2014 with the purpose of increasing the impact, scale, and sustainability of early grade reading (EGR) interventions in the LAC region. LRCP provided ministries of education and other relevant stakeholders with the necessary awareness, knowledge, and tools to increase EGR achievement. LRCP has been implemented primarily in the Dominican Republic, Guatemala, Haiti, Honduras, Jamaica, and Nicaragua. This performance evaluation was undertaken to inform midcourse adjustments in program implementation for the remainder of the LRCP and any possible extensions, as well as to inform ongoing and future regional and bilateral USAID programming. It addresses two questions: (1) How effective has the LRCP been in achieving its goals by result and by country to date? and (2) What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy-level interventions? Quantitative and qualitative data gathered for the evaluation revealed that LRCP has consolidated and disseminated EGR research and resources from the LAC region, yet the use of them by key stakeholders remains limited. EGR building the capacity of ministry of education staff, teachers, members of NGOs, and other stakeholders was reported to be the most important LRCP contribution. Bases established for sustainability include a critical mass of EGR stakeholders, a momentum for EGR improvement, alliances, and a regional network for EGR research. i | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV CONTENTS Abstract ........................................................................................................................................................................ ii Acronyms..................................................................................................................................................................... iii Executive Summary.................................................................................................................................................... iv Program Background............................................................................................................................................. iv Evaluation Methods and Limitations................................................................................................................... iv Findings and Conclusions...................................................................................................................................... iv Recommendations................................................................................................................................................ vii Evaluation Purpose and Evaluation Questions ....................................................................................................... 1 Evaluation Purpose ................................................................................................................................................. 1 Evaluation Questions.............................................................................................................................................. 1 LRCP Program Background....................................................................................................................................... 2 LRCP Objective, Expected Results, and Theory of Change............................................................................ 3 Evaluation Methods and Limitations......................................................................................................................... 5 Sampling Process..................................................................................................................................................... 6 Data Collection....................................................................................................................................................... 7 Data Analysis Methods.........................................................................................................................................10 Limitations..............................................................................................................................................................12 Findings........................................................................................................................................................................13 Evaluation Question 1..........................................................................................................................................13 Evaluation Question 2..........................................................................................................................................38 Conclusions................................................................................................................................................................42 Evaluation Question 1..........................................................................................................................................42 Evaluation Question 2..........................................................................................................................................43 Recommendations.....................................................................................................................................................44 Evaluation Question 1..........................................................................................................................................44 Evaluation Question 2..........................................................................................................................................45 Annex 1. Evaluation Statement of Work ..............................................................................................................47 Annex 2. Evaluation Team Composition...............................................................................................................55 Annex 3. Data Collection Instruments..................................................................................................................57 Annex 4. Sources of Information............................................................................................................................79 Annex 5. Disclosure of Any Conflicts of Interest................................................................................................83 USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | ii TABLES Table 1. Evaluation Design Matrix............................................................................................................................ 6 Table 2. Data collection methods and number of respondents, by country and timeline ............................ 8 Table 3. Number of online survey respondents by country and category of employer...............................11 Table 4. Overview of the LRCP intervention in each country..........................................................................14 Table 5. Budget amount by country.......................................................................................................................15 Table 6. LRCP website views, Apr. 2016–Feb. 7, 2019......................................................................................17 Table 7. Database use, Apr. 2016–Sep. 2018 (AIR Data Analytics)..................................................................17 Table 8. Downloads from LRCP website, Apr. 1, 2016–Feb. 7, 2019 .............................................................17 Table 9. Usefulness of information on the LRCP website (online survey)......................................................22 Table 10. Capacity-building activities by country.................................................................................................24 FIGURES Figure 1. The LRCP Theory of Change ................................................................................................................... 4 Figure 2. Online survey respondents by title .......................................................................................................11 Figure 3. Distribution of financial resources by country ....................................................................................15 Figure 4. How did you learn about the LRCP? (online survey) .........................................................................20 Figure 5. How did you learn about the LRCP? (by country) .............................................................................20 Figure 6. Number of articles mentioning EGR (M&E data) ................................................................................21 Figure 7. Usefulness of information on the LRCP website (online survey) .....................................................22 Figure 8. Number of organizations that participated in training (M&E Data) .................................................24 Figure 9. Number of stakeholders who completed professional development activities (M&E data) ........25 Figure 10. Motives for participating in LRCP activities (online survey) ..........................................................26 Figure 11. Usefulness of LRCP Events, by percentages (online survey) .........................................................26 Figure 12. Reasons for usefulness of LRCP events (online survey) ..................................................................27 Figure 13. Percentage of respondents familiar with LRCP products (online survey) ....................................30 Figure 14. Usefulness of LRCP products ...............................................................................................................32 Figure 15. Reasons an LRCP activity was not useful (online survey) ...............................................................33 iii | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ACRONYMS AIR American Institutes for Research ASIES Asociación de Investigación y Estudios Sociales CESESMA Centro de Servicios Educativos en Salud y Medio Ambiente CETT Centers of Excellence in Teacher Training CIASES Centro de Investigación y Acción Educativa Social EGL Early grade literacy EGR Early grade reading EGRA Early Grade Reading Assessment ELP Early language program ESC Eastern and Southern Caribbean FEREMA Fundación para la Educación Ricardo Ernesto Maduro Andreu FGD Focus group discussion FY Fiscal Year GCNPE Gran Campaña Nacional para la Educación KII Key informant interview LAC Latin America and the Caribbean LAC/RSD/EDU Regional Sustainable Development Office of the Bureau of Latin America and the Caribbean Education Team LRCP LAC Reads Capacity Program M&E Monitoring and evaluation MINERD Dominican Republic Ministry of Education MOE Ministry of Education MOU Memorandum of Understanding NGO Non-governmental organization OECS Organisation of the Eastern Caribbean States PREAL Partnership for Educational Revitalization in the Americas RedLEI Central American and Caribbean Early Grade Literacy Network RFP Request for Proposal SOW Statement of Work UCA Universidad Céntroamericana UCR Universidad de Costa Rica UPNFM Universidad Pedagógica Nacional Francisco Morazán USAID U.S. Agency for International Development UVG Universidad del Valle de Guatemala USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | iv EXECUTIVE SUMMARY The United States Agency for International Development (USAID) requested a midterm performance evaluation of the Latin American and Caribbean (LAC) Reads Capacity Program (LRCP). The evaluation addresses two questions: (1) How effective has the LRCP been in achieving its goals by result and by country to date? and (2) What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy-level interventions? PROGRAM BACKGROUND USAID initiated the LRCP in 2014 to increase the impact, scale, and sustainability of early grade reading (EGR) interventions in the LAC region. LRCP is implemented in the Dominican Republic, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Peru, and selected countries in the sub-regional Eastern and Southern Caribbean (ESC) by American Institutes for Research (AIR), with sub-contractor Juárez and Associates. The $23 million, six-year LRCP was awarded September 30, 2014 and ends June 30, 2020. LRCP aims to provide relevant stakeholders with the necessary awareness, knowledge, and tools to increase EGR achievement. EVALUATION METHODS AND LIMITATIONS The evaluation methodology consisted of document review, an online survey, an in-person survey using a structured questionnaire, in-depth key informant interviews (KIIs), and focus group discussions (FGDs). The evaluation team collected data in Guatemala, Honduras, and Nicaragua and remote data for the other countries. Informants included USAID, implementing and partner organizations, ministries of education (MOEs), universities, local and international non-governmental organizations (NGOs), and early grade literacy (EGL) consultants and researchers. The evaluation team collected data from a total of 392 people. It conducted KIIs and FGDs with 52 people from LRCP implementing and partner organizations, the MOE, NGOs, and universities. Data was also collected through a structured questionnaire with 18 LRCP implementers and an online survey from 335 LRCP beneficiaries. The team analyzed survey data using frequencies and cross tabulations and triangulated data by comparing them by source (e.g., KII data, project reports) and method (e.g., survey and KII data). Limitations included incomplete questionnaire and online survey responses, selection bias of online survey respondents, and potential bias in online survey and questionnaire responses. FINDINGS AND CONCLUSIONS Evaluation Question 1: How effective has the LRCP been in achieving its goals by result and by country to date? LRCP EXPECTED RESULT 1: EVIDENCE ON EGR PRACTICES AND OUTCOMES IS COLLECTED, CONSOLIDATED, AND SYSTEMATIZED FOR PRACTICAL USE BY STAKEHOLDERS IN THE LAC REGION LRCP consolidated evidence in the LAC region through a systematic literature review, an LRCP database, and stakeholder analyses. Survey and KII/FGD data from all categories of informants reported the stakeholder analysis as the most valuable, because it identifies key EGL actors so that interventions can catalyze and complement EGL improvement efforts. For members of partner organizations, the stakeholder analysis and database-creation generated ownership and fostered learning opportunities. About half of KII/FGD informants reported having used resources in the database. v | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Awareness of and demand for evidence-based EGL information remain relatively low in the LAC region. The monitoring and evaluation (M&E) and KII/FGD data reflect some demand for and use of certain LRCP resources, such as articles from the online database, yet most downloads were in the U.S. from April 2016–February 2019 (874 compared to an average of 174 for each of the five participating LRCP countries). Survey data reveal that only 46 percent of beneficiaries who participated in LRCP events are familiar with the website, 16 percent with the database, 13 percent with the systematic review, and 25 percent with the stakeholder analysis. The systematic review set a precedent in the region for a rigorous analysis of available EGL research and gaps; however, it lacked practical application for most beneficiaries. The majority of LRCP implementing partners interviewed and four KII respondents familiar with the systematic review indicated that it was too technical to be useful for most targeted beneficiaries of the LRCP. They also noted that the process of producing it could have been more inclusive and included some practical applications for using the produced information. LRCP EXPECTED RESULT 2: DISSEMINATION OF UP-TO-DATE KNOWLEDGE ABOUT EGR PRACTICES IS TARGETED TO DIVERSE AUDIENCES AND STAKEHOLDERS LRCP targeted EGL knowledge to diverse audiences, reinforcing other efforts to make EGL more central on the agendas of the MOEs and civil society. All five participating countries have held conferences on EGL targeting MOE staff, teachers, teacher coaches, professors, researchers, and NGOs. Partner organizations have participated in policy discussions, sent out bulletins, targeted media, and used social media to promote EGL. KII/FGD data from all categories of informants in all countries pointed to LRCP’s activities as being instrumental in EGL becoming a higher national priority. There was not a cohesive communication strategy at the beginning to help guide partner organizations with unified messages and strategies and more widely diffuse LRCP compilations and products. Members of partner and implementing organizations said the communication strategy lacked guidance and regional integration for the first two years of LRCP, resulting in a less uniform and effective dissemination strategy. Respondents reported that LRCP materials were disseminated through events, yet scarcely through other means. LRCP is seen as a platform to test different methodologies for literacy development and use evidence to inform the discussion and find common ground. Informants from all categories and countries reported that LRCP has illuminated debates on EGL methodological approaches, which could be further developed through evidence and discussion forums provided by LRCP. LRCP EXPECTED RESULT 3: CAPACITY IS STRENGTHENED AT INSTITUTIONS, SO THEY CAN MORE WIDELY IMPLEMENT PROVEN APPROACHES TO IMPROVE EGR OUTCOMES FOR POOR AND DISADVANTAGED CHILDREN LRCP has been strengthening the capacity of MOE personnel, partner organizations, NGO staff, and other beneficiaries across all the LRCP countries. This has reached beyond the capital cities into different regions of the countries. Survey and KII/FGD data show that informants from all categories indicate having gained new knowledge about teaching methodologies, and roughly half (53 percent) indicated that they have put it into practice, such as guiding teachers in EGL instruction. A third of respondents indicated that they had replicated or shared what they learned. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | vi Training events that built upon one another and had practical applications were the most useful for participants. According to KII/FGD and survey data, most training events lacked follow-up, which limited the process of knowledge and skill development. Beneficiaries who participated in training with continuity, such as the workshops in Honduras to document EGL best practices and the regional online EGL training, indicated that these were the most useful for developing knowledge of evidence￾based EGL practices and how to implement them. LRCP EXPECTED RESULT 4: SUSTAINABILITY PLATFORMS ARE IN PLACE TO CONTINUE AND STRENGTHEN EGR INTERVENTIONS IN COUNTRIES IN THE LAC REGION LRCP has done foundational work for the sustainability of its interventions. The five public and private universities that form the Central American and Caribbean Early Grade Literacy Network, RedLEI, are all highly regarded and can contribute to EGL research. RedLEI has established important alliances to support EGL research and influence policies. RedLEI has also designed a regional EGL master’s program and is currently designing a virtual library to continue hosting the LRCP database. Across the countries, LRCP has strengthened the capacity of a critical mass of EGL stakeholders and generated momentum for EGL evidence use and EGL improvement. Stakeholders from all categories across the countries pointed to a critical mass1 of EGL stakeholders and momentum for EGL improvement that has been generated by LRCP. However, stakeholders are concerned about this momentum being sustained beyond LRCP funding. The greatest challenges RedLEI has faced are ensuring it can generate its own funds to sustain efforts for improving EGL and training young researchers to conduct EGL research. RedLEI has successfully launched EGL research in Central America, such as the EGL research projects and mentoring of researchers, yet to sustain these RedLEI must be able to generate its own funds.2 The master’s program in EGL is an opportunity for RedLEI to work together and generate funds and to strengthen EGL capacity in the LAC region. Currently, however, both the demand for such a program and how participants will be funded without LRCP funds are unclear. Evaluation Question 2: What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy-level interventions? LRCP has helped bolster the impact and sustainability of Missions’ bilateral EGL projects. For all Missions, LRCP has strengthened the capacity of MOE personnel to improve EGL and to support the implementation and sustainability of bilateral projects. Some of the MOE staff who participate in LRCP training implement the bilateral programs, and thus gain knowledge for effectively guiding EGL efforts. The demand-driven nature of LRCP has allowed Missions to use it for their needs, in addition to benefiting from specific LRCP activities. In Peru, LRCP provided technical support to work with regional MOE offices to implement the bilateral project. In the ESC, LRCP provided technical support to baseline, midline, and endline assessments for the bilateral project. In Nicaragua, LRCP facilitated the 1 Stakeholders and beneficiaries often used the term “critical mass” to refer to the many people specializing in EGL and working to improve EGL outcomes. 2 This is discussed in detail in the sustainability assessment by O’Brien and Associates, dated November 2018. vii | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV continuity of the bilateral program despite the political crisis and inability to work with the MOE. All countries have experienced some benefits from LRCP regional sharing and capacity building. The degree of coordination between Missions and LRCP has varied across the countries. In Guatemala and Nicaragua, LRCP has had a close working relationship with the Missions’ EGR bilateral project. In Honduras, there has been less coordination between LRCP and the bilateral projects. LRCP’s organizational structure of a prime and sub-prime with multiple partner organizations has mostly worked well. Informants from implementing and partner organizations attributed this to previous working relationships between the prime and sub-prime and their experience in the countries they oversee in LRCP’s implementation. They also pointed to the strong institutional capacity and solid professional reputation of each partner organization. Given the complexity of the organizational structure, USAID noted some initial challenges which resulted in operational delays or communication gaps. It is difficult to identify indicators to measure LRCP’s effects due of the nature of the intervention, although many current M&E indicators are good proxies if rigorously measured. For example, “number of hits on the database” and “number of downloads” provide data to indicate database use and demand for evidence by country, yet these data fall short of measuring how and whether the information is used. Similarly, to measure effects of capacity strengthening, the pre- and post￾surveys measure acquired knowledge, which is helpful, but not how this knowledge is being applied. LRCP’s regional policy and systems-level approach has built capacity of key stakeholders responsible for implementing EGL, such as partner organizations, MOEs, teacher-training colleges, universities, and international and national NGOs. LRCP has strengthened the capacity of stakeholders in charge of EGL guidelines and materials and training teachers. The regional approach has allowed for cost-efficient sharing of resources and transferring capacity across countries, especially for partner organizations to strengthen knowledge and skills over time. Both partner organizations and other stakeholders have benefited from information dissemination and training; by contrast, partner organizations have benefited from sustained engagement, while this is not the case for other stakeholders. Missions can engage with LRCP to the extent they find useful, though there may be missed opportunities for creating synergies to improve EGL. LRCP allows Missions to engage to the degree they would like and how they would like. However, there is no mechanism in place to help ensure that coordination and/or joint planning takes place between LRCP and the bilateral projects, which, if it did exist, could provide structure and guidance for effective coordination. RECOMMENDATIONS The following recommendations reflect both those provided by KII/FGD informants and those that the evaluation team identified based on the analysis of findings. RESOURCES AND LEVELS OF EFFORT. For the remainder of the LRCP, focus resources on maximizing the impact from the investment in capacity strengthening by carrying out training activities to develop EGL skills based on identified gaps. Develop a training plan with applications, such as training/coaching teachers in EGL, elaborating EGL research designs, and conducting action research. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | viii RESULT 1 (A). Transfer the database to be housed by RedLEI in coordination with RedLEI universities and task the universities with regularly doing searches and reviews of new EGL research and resources. RESULT 1 (B). RedLEI should supply annual or biannual updates of the systematic review, with support from AIR for the first update. Make the review available in Spanish, with a shortened version for a more widespread audience available in both English and Spanish. RESULT 2 (A). Hold dissemination events in each country with stakeholders who are researchers to discuss the findings of the updated systematic review and define a research and training agenda. RESULT 2 (B). Make the dissemination strategy more interactive by sending out notifications of new database resources, providing summaries, and targeting resources based on stakeholder interests. RESULT 2 (C). For future regional initiatives, ensure that there is a cohesive communication strategy across the countries and partner organizations that provides uniform guidance, yet allows for country￾specific differences. Consider having a dedicated communications specialist based in the LAC region. RESULT 3 (A). USAID and implementers should learn from the successful aspects of the capacity building component to design future initiatives, such as targeting MOE and bilateral project personnel and conducting training events with continuity and practical applications, such as researching EGL practices in classrooms and analyzing them in light of evidence on effective EGL instruction. RESULT 4 (A). Hold an event in each country to present and discuss evidence on EGL methodological approaches for the context-relevant debate and define a research agenda and mechanism to share contextualized evidence. Give the MOE a leading role in this event to generate ownership. RESULT 4 (B). RedLEI should assess the demand for the EGL master’s course by consulting with and/or surveying stakeholders and base a marketing plan on the demand. Initiate discussions with MOEs, starting with Guatemala, to seek MOE support in providing scholarships for its technical staff. RESULT 4 (C). To maintain momentum, prioritize stakeholders that form part of the EGL-improvement critical mass for training and dissemination and identify ways to further strengthen an EGL network. LRCP APPROACH. Structure future similar initiatives with the same mission-demand-driven flexibility, as well as some consistency across countries such as regional sharing of technical expertise, resources, and practices. Consider regional capacity building using online learning and practical applications. FLEXIBLE INDICATORS. Measure capacity-building more systematically and rigorously by capturing specific skills and applications such as the indicator “percent of people applying content from training”. POLICY AND SYSTEMS. For future initiatives, USAID could consider a modality for translating evidence into practice through technical support to the MOE and engaging relevant stakeholders to design and conduct research that directly informs their most pertinent EGL policy and programming decisions. MISSION ENGAGEMENT. USAID’s Regional Sustainable Development Office of the Bureau of Latin America Education Team (LAC/RSD/EDU) should convene a virtual meeting with the USAID/Honduras education team to discuss how LRCP can best complement and bolster the bilateral projects’ impacts. The USAID/Honduras education team and implementers should hold monthly meetings with this aim. 1 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV EVALUATION PURPOSE AND EVALUATION QUESTIONS The United States Agency for International Development (USAID) initiated the Latin American and Caribbean (LAC) Reads Capacity Program (LRCP) to increase the impact, scale, and sustainability of early grade reading (EGR) interventions in the LAC region. The LRCP was designed to support Goal 1 of USAID’s five-year strategy released in February 2011, “Improved reading skills for 100 million children in primary grades by 2015.”3 The LRCP is a $23 million cooperative agreement (No. AID-OAA-A-14-00058) that was awarded on September 30, 2014 and ends June 30, 2020. It is implemented by American Institutes for Research (AIR) in partnership with its major subcontractor Juárez and Associates. Implementation has taken place in the Dominican Republic, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Peru, and the Eastern and Southern Caribbean (ESC). EVALUATION PURPOSE The purpose of this performance evaluation is to inform (1) midcourse adjustments in program implementation for the remainder of the LRCP and any possible extensions and (2) ongoing and future regional and bilateral USAID programming that addresses similar subjects and key stakeholders. The primary audiences for this evaluation include USAID education experts and agency implementing partners. The evaluation findings may also be of interest to regional education stakeholders. This evaluation examines the LRCP’s progress against its main goals, the extent to which expected results are being met, and the effectiveness of the approaches employed to meet them. The evaluation identifies key factors that contribute to and/or impede the program’s success, as well as the effects of the program’s interventions and how they are working together to accomplish the results. Additionally, the evaluation analyzes the underlying theory of change and mechanisms used to achieve the goals and expected results. The timing of the LRCP interventions has varied greatly across countries. Some countries, such as Guatemala, only began full implementation roughly one year ago. Thus, while this evaluation is considered a “midterm” evaluation, it was not conducted at the midpoint of the program’s operation in all countries. This evaluation of the LRCP was conducted under USAID’s LAC Education Support Contract implemented by DevTech Systems, Inc. between October 1, 2018 and September 30, 2023. EVALUATION QUESTIONS The evaluation responds to the following questions: 1. How effective has the LRCP been in achieving its goals by result and by country to date? 1.A. Where have the most resources and levels of effort focused, by result, by country, and to date? 1.B. What have been the most successful activities under each result by country to date and for what reasons? 1.C. What have been the greatest challenges or setbacks to achieving each result, by country, to date, and for what reasons? 1.D. What evidence is there that local partner institutions and key stakeholder institutions have increased capacity to improve early literacy in their home countries? 3 This strategy has been replaced by the U.S. Government Strategy on International Basic Education, though at the time of its design, the LRCP was responding to the former strategy. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 2 1.E. What have been the most valuable aspects of the LRCP for USAID Missions? 2. What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy-level interventions? 2.A. What are the strengths and weaknesses of the technical approach as structured (featuring a principal prime and principal sub-prime, as well as multiple national-level partners)? What alternative modalities might be considered for a program with similar or related goals? 2.B. What are the strengths and weaknesses of having results with flexible indicators not tied to performance indicators (such as improved literacy levels)? 2.C. What are the strengths and weaknesses of this regional literacy programming that focuses at the policy or systems levels? Given the stated objectives, the scope of activities, and investment levels, are there alternative approaches, models, or activities to supporting increased early literacy outcomes that USAID should explore? 2.D. What are the strengths and weaknesses vis-à-vis USAID Mission benefits and engagement? LRCP PROGRAM BACKGROUND The LRCP was designed to target countries in the LAC region, specifically for the early grades of primary school (typically grades one through three) to increase the impact, scale, and sustainability of EGR interventions in the region. In the LAC region, USAID has a particular interest in supporting the improvement of EGR. The population under 15 years old in the LAC region is 35 percent and growing. Children who do not develop fundamental literacy skills are much more likely to drop out of school and not work, which also puts them at risk of becoming socially isolated and involved in illicit activities and violence. Therefore, it is essential that these children develop fundamental literacy skills in order to continue learning and eventually incorporate themselves into the society and the legal economy. Over the past two decades in the LAC region, access to and enrollment in primary school has improved, though major challenges still exist in the quality of education children receive. An important building block and indicator of education quality is reading proficiency. Reading levels at the primary grades across the LAC region are extremely low and have shown little or no improvement. According to the most recent regional achievement test (2016 Third Regional Comparative and Explanatory Study, or TERCE), the majority of third-grade students scored in the lowest achievement band, meaning that they can only identify key information when it is explicitly and repeatedly stated in a highlighted part of the text and separate from other information. Low reading proficiency is the result of systemic problems in education, such as insufficient investment in education and poor teaching quality. Another challenge is the lack of availability of, demand for, and use of evidence for making decisions related to improving reading achievement. Increasingly, there is more evidence internationally on effective early literacy instruction policies and practices. While contexts vary, much is known today about what is required for EGR programs to be successful and cost-effective. However, this information has not been readily available in Spanish or easily accessible across the LAC region. When it is available, it is very uncommon for education decision makers in the region to use this evidence to inform policy and program decisions. USAID’s LAC Regional Sustainable Development Office for Education (LAC/RSD/EDU) has supported several regional programs to address challenges in teaching quality and evidence-based decision making, 3 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV the most noteworthy being the Centers of Excellence in Teacher Training (CETT) and Partnership for Educational Revitalization in the Americas (PREAL). PREAL, which began in 1995. These programs aimed to improve the quality and relevance of policy dialogue around education reform to strengthen the social demand for improved quality in education and to build political support for implementing quality-driven reforms. While PREAL was not designed to provide direct support to the LAC Missions’ bilateral programs, one of the most important roles of the LAC/RSD is supporting the work of USAID’s bilateral Missions in the region. While regional activities are not a substitute for bilateral activities, they can complement and bolster them. LRCP was developed as one of the three components of the LAC Reads Project, aiming to increase the impact, scale, and sustainability of EGR interventions in the LAC region. It was particularly designed to assist USAID bilateral efforts across the LAC region and complement other regional efforts to improve the evidence base for decision making related to improving literacy in the LAC region. It is also intended to build the capacity of people in organizations and institutions to improve EGL to use evidence in decision making. The rationale for this regional reading program was: (1) the shared, region-wide focus at USAID on improving EGR in LAC; (2) the interest expressed by LAC stakeholders in a platform that would facilitate regional sharing of information and evidence on cost￾effective approaches to improve reading, building on the precedent set by PREAL; and (3) the need for technical assistance to strengthen the capacity of key stakeholders in early literacy improvement to access, understand, and utilize evidence on EGR in order to inform interventions to improve early literacy. LRCP COUNTRIES. At the start of the LRCP, USAID had bilateral basic education programs in eight LAC countries (the Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, and Peru) and in the ESC. Because El Salvador would no longer have a bilateral EGR program when the LRCP was to begin, El Salvador was not included in LRCP. Thus, USAID implemented the LRCP in the Dominican Republic, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Peru, and selected countries in the ESC. However, Haiti developed a separate implementation plan that deviated significantly from that of the other beneficiary countries, while Peru and the ESC participated only in certain activities in a much more limited level of engagement. Thus, the countries that have participated in all components of the LRCP to date are the Dominican Republic, Guatemala, Honduras, Jamaica, and Nicaragua, with Haiti receiving a full but separate treatment. LRCP IMPLEMENTATION. AIR is officially the prime organization and Juárez and Associates the sub￾prime. However, AIR and Juárez agreed in their technical proposal to USAID that they would operate as a single team, with AIR taking implementation responsibility in Haiti, Honduras, Nicaragua, and Peru, and Juárez in the Dominican Republic, Guatemala, Jamaica, and the ESC. Partner organizations in each country have received sub-grants from AIR and Juárez. LRCP OBJECTIVE, EXPECTED RESULTS, AND THEORY OF CHANGE The overall objective of the LRCP is to provide ministries of education (MOEs) and other relevant stakeholders with the necessary awareness, knowledge, and tools to increase EGR achievement. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 4 The expected results are: RESULT 1. Evidence on EGR practices and outcomes is collected, consolidated, and systematized for practical use by stakeholders in the LAC region (systematic review of evidence; creation of evidence database). RESULT 2. Dissemination of up-to-date knowledge about EGR practices is targeted to diverse audiences and stakeholders (stakeholder engagement to assess state of EGR in each country, website, conferences, and publications). RESULT 3. Capacity is strengthened at institutions so they can more widely implement proven approaches to improve EGR outcomes for poor and disadvantaged children (technical assistance and assessments for stakeholders). RESULT 4. Sustainability platforms are in place to continue and strengthen EGR interventions in countries in the LAC region (creation of regional research network). The LRCP theory of change is that providing state-of-the-art knowledge resources and technical assistance will lead to increased awareness of, demand for, and capacity to use effective evidence-based practices that increase EGR achievement by MOEs and other key stakeholders. This will increase the impact, scale, and sustainability of EGR interventions in the LAC region and eventually lead to increased EGR achievement.4 Figure 1 illustrates the theory of change. Figure 1. The LRCP Theory of Change 4 As stated in the Request for Application of the LAC Reads Capacity Program. Inputs -Elaboration of program documents and studies -Dissemination activities -EGR workshops Outputs -Database use -Use of studies -Stakeholder participation in events and training -Creation of research network Outcomes -Increased stakeholder awareness of effective EGR evidence-based practices -Increased stakeholder demand for effective EGR evidence-based practices -Increased stakeholder capacity to use effective evidence-based practices Impact -Increased impact, scale, and sustainability of EGR interventions in the LAC region -Increased EGR achievement 5 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV EVALUATION METHODS AND LIMITATIONS In order to conduct the performance evaluation, the evaluation team used document reviews, an online survey, an in-person survey using a structured questionnaire, in-depth key informant interviews (KIIs), and mini-focus group discussions (FGDs) that consisted of fewer than five participants. The team conducted document reviews for all eight countries and ESC. Owing to the regional nature of the LRCP, which has unique implementation approaches in each country, the depth of data gathered in each country varied. ● In Guatemala, Honduras, and Nicaragua, the team collected data more deeply and broadly because LAC/RSD/EDU identified these as countries where more activity had taken place and more information on the LRCP’s effects and impact was needed. Furthermore, these three countries all have substantial ongoing EGR bilateral projects with their respective USAID Missions. ● In the Dominican Republic and Jamaica, the team collected less data because the evaluation did not include in-country data collection for these countries. ● The team did not collect data in Haiti because the LRCP goals there differ from the other LRCP countries and USAID indicated that data collection was not necessary in the case of Haiti. ● For Peru and the ESC, minimal data (limited only to several KIIs) were collected because very few LRCP activities were implemented there. A team of both international and local evaluators conducted the evaluation. The team comprised five members: the team leader, a research assistant, and three local consultants (one each in Guatemala, Honduras, and Nicaragua) to assist with in-depth data collection. The team leader, research assistant, and local consultant in Guatemala and Honduras carried out KIIs and FGDs and administered the structured questionnaire survey; in Nicaragua, the local consultant carried out the KIIs and FGDs.5 The team leader and research assistant developed and sent out the online survey with the input of the local consultants. The team leader led the data analysis, yet all team members contributed to the analysis. See Error! Reference source not found. for a summary of the evaluation design arranged by evaluation question. 5 Due to the sociopolitical situation in Nicaragua, travel to Nicaragua was not possible for the Team Leader. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 6 TABLE 1. EVALUATION DESIGN MATRIX Evaluation Question Sources of Data Data Collection Methods Analytical Approach 1. How effective has the LRCP been in achieving its goals by result and by country to date? 1.A. Where have the most resources and levels of effort gone, by result, by country, and to date? Program budget Document review (Desk review) Budget analysis Program monitoring and reporting documents Document review (Desk review) Content analysis, thematic analysis Implementers KIIs/FGDs 1.B. What have been the most successful activities under each result, by country, to date, and for what reasons? Program monitoring and reporting documents KIIs/FGDs and online survey Summary analysis and content analysis Implementers, partner organizations, and beneficiaries In-person survey with structured questionnaire Descriptive statistics, content analysis Online survey Descriptive statistics 1.C. What have been the greatest challenges or setbacks to achieving each result, by country, to date, and for what reasons? Program monitoring and reporting documents Document analysis (desk review) Summary analysis Donor, implementers, partner organizations, and beneficiaries KIIs/FGDs and online survey Content analysis, thematic analysis 1.D. What evidence is there that the capacities of local partner institutions and key stakeholder institutions have increased capacity to improve early literacy in their home countries? Implementers, partner organizations, and beneficiaries KIIs/FGDs and online survey Content analysis, thematic analysis Online survey Indicator analysis, descriptive statistics 2. What have been the most valuable aspects of the LRCP for USAID Missions? 2.A. What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy-level interventions? Donor, implementers, partner organizations, and beneficiaries KIIs/FGDs Content analysis, thematic analysis 2.B. What are the strengths and weaknesses of having results with flexible indicators not tied to performance indicators? Program monitoring and reporting documents Document review (desk review) Indicator analysis Donors, implementers, and partner organizations KIIs/FGDs Content analysis, thematic analysis 2.C. What are the strengths and weaknesses of this regional literacy programming that focuses at the policy or systems levels? What alternatives could be explored? Program reporting documents Document review (desk review) Content analysis Donor, implementers, and partner organizations KIIs/FGDs Content analysis, thematic analysis 2.D. What are the strengths and weaknesses of the program vis-à-vis USAID Mission benefits and engagement? Donor, implementers, and partner organizations KIIs/FGDs Content analysis, thematic analysis SAMPLING PROCESS For KIIs, mini-FGDs, and structured surveys, the evaluation used a purposive sampling approach. The team selected key informants from USAID, implementers, and partner organizations, as well as beneficiaries 7 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV from government institutions (mainly MOEs), other donors, and local organizations. The prime implementing organization provided an initial list of the LRCP core team members, members of partner organizations in the LRCP countries, and other informants (such as independent consultants). With the assistance of LAC/RSD/EDU, the evaluation team identified individuals from USAID Missions in the LRCP countries who had knowledge of and/or involvement with the LRCP. Thus, for USAID, implementers, and partner organizations, the team did not aim for a specific sample size per country but rather sought representation from each group (USAID, implementers, and partner organizations) across the LRCP countries, with the exception of the independent consultants and U.S.-based individuals who could speak to LRCP implementation at a regional level. The evaluation team selected all these individuals for participation in KIIs and/or mini-FGDs. The team also administered a structured questionnaire to members of the LRCP implementing and partner organizations. Additionally, beneficiaries from MOEs, local organizations, and donor organizations were identified by the implementing and partner organizations because of their involvement with and/or knowledge of the LRCP. For the three in-country data collection countries, an exact sample size was not determined due to the non-uniform nature of implementation (e.g., in Nicaragua, MOE personnel could not be interviewed). For the Dominican Republic and Jamaica, in addition to the people identified at the USAID Missions and implementing partners, several key individuals were selected from the MOEs. All these individuals were included in the sample for KIIs and FGDs. For the online survey, the beneficiary sample size could not be predetermined because survey responses were optional. However, the sample was made up of all the individuals who had attended events and/or training activities as documented by the partner organizations. Because the activities varied by outreach and participation across countries, there was not a uniform sample size per country. The evaluation team decided to send the survey to as many individuals as possible in order to increase the response rate. DATA COLLECTION The evaluation team collected data from a total of 392 people. It conducted KIIs and FGDs with 52 people from LRCP implementing and partner organizations, the MOE, NGOs, and universities. Data was also collected through a structured questionnaire with 18 LRCP implementers and an online survey from 335 LRCP beneficiaries (Table 2). USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 8 TABLE 2. DATA COLLECTION METHODS AND NUMBER OF RESPONDENTS, BY COUNTRY AND TIMELINE Country or Region Document Review 6 (Oct. 2018– Feb. 2019) Online Survey (Jan. 7– Feb. 11) Survey Structured Questionnaire (Nov. 2018) KII (Nov. 2018– Jan. 2019) Mini-FGD (Nov. 2018– Jan. 2019) ESC 26 0 0 2 0 Dominican Republic 26 72 0 2 0 Guatemala 26 63 4 17 1 (5 people) Haiti 2 0 0 0 0 Honduras 28 73 2 13 1 (3 people) Jamaica 28 63 3 1 2 (5 people total) Nicaragua 28 61 4 7 2 (5 people total) Peru 26 0 0 1 0 U.S./Other 0 3 5 4 0 Total 34 335 18 46 6 (18 people) DOCUMENT REVIEW The team conducted a document review prior to fieldwork to inform in-country data collection and fine￾tune the data collection instruments and sources. The team obtained and reviewed several documents relevant to the LRCP, including the original Request for Application, the implementers’ technical proposal, the M&E plan and data reported, quarterly reports, country-specific policy documents, and studies and other materials produced as part of LRCP’s implementation. The team reviewed additional documents throughout the evaluation. See Annex 4 for a full list of documents consulted, including a bibliography. KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS The team conducted all KIIs, surveys using the structured questionnaire, and FGDs with stakeholders in Guatemala, Honduras, and Nicaragua in person. KIIs with the LRCP core implementation team were mostly conducted in Guatemala or Honduras in person. In Jamaica and the Dominican Republic, KIIs and FGDs were conducted in person with key stakeholders when possible, or by phone when in-person interviews were not possible. 6 All documents were designed to be relevant and accessible to all countries, except in the case of Haiti where some documents were not translated to French or Creole. The country profiles and stakeholder analyses were specific to the Dominican Republic, Honduras, Jamaica, and Nicaragua. 9 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV The decision to carry out a KII versus an FGD depended primarily on the sensitivity of the topic, the context, and the participants. The team conducted mini-FGDs with five people or fewer only when feasible and beneficial, such as when more than one member of an organization shared responsibilities and goals. In the case of Nicaragua, owing to the political situation, the team decided to carry out KIIs in almost all situations. KII data captured information relevant to the evaluation questions and helped the team develop a better understanding of different perspectives around the program’s theory of change and aspects that have facilitated or interfered with effectively carrying out the intended interventions. See Annex 4 for a list of key stakeholders who were interviewed or participated in focus groups.7 KIIs were conducted in English, when possible, to decrease the amount of qualitative data (transcripts or sections of transcripts) to be translated from Spanish to English. Semi-structured and structured interview protocols were used for all KIIs and FGDs. (See Annex 3 for interview/focus group protocols and other data collection instruments.) After obtaining consent from participants, the team recorded audio of almost all the KIIs and FGDs. Interviewers also took detailed notes during the interviews and FGDs and wrote analytic memos following each KII or FGD to capture key findings related to the evaluation questions and each expected result of the LRCP. ONLINE SURVEY The online survey gauged the effects of the LRCP on beneficiaries, such as awareness and use of the products and database and knowledge and skills acquired. After considering various mechanisms for administering the online survey, the team chose Survey Monkey because it is user friendly for survey respondents and effective for organizing, processing, and analyzing data. The evaluation team emailed a link to the survey to stakeholders in the Dominican Republic, Guatemala, Honduras, Jamaica, and Nicaragua.8 The survey was in Spanish, and an English translation for Jamaican respondents captured the same information as the Spanish version. The team gathered contact information for survey respondents from lists of LRCP event participants provided by partner organizations in each country. In total, the team sent surveys to 1,011 people. Of these, 217 were undeliverable, resulting in 794 surveys received by potential respondents. Of those who received the survey, 335 responded. Thus, the response rate at the regional level was 42 percent. Online survey data captured basic quantitative information (such as the number of people who attended specific events and/or used specific documents) and more detailed feedback via Likert scales (such as degree of usefulness of information from the LRCP database and conferences). The survey also captured some open-ended responses. 7 An earthquake off the coast of the Dominican Republic debilitated the phone lines and internet connection on the day of the scheduled KIIs. The team rescheduled, but there were still problems with the phone lines and the internet. Per the respondents’ suggestion, the evaluation team sent the KII questions via email; however, they were never answered. 8 The online survey was not sent to the ESC or Peru because the LRCP activities that took place there were more limited in scope than in the other LRCP countries. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 10 The team emailed the link to the online survey shortly after in-country data collection to allow for any necessary fine-tuning based on KII and structured survey data. The survey was available for four weeks. To increase the response rate, the team sent periodic follow-up emails and reminders. To ensure anonymity, the survey asked for very little identifying information. STRUCTURED QUESTIONNAIRE SURVEY The team administered a structured survey questionnaire to LRCP core team members and key partner organization members. The survey assessed perceptions and opinions using Likert scales and open-ended questions. The survey was designed, not to assess the effects and effectiveness of LRCP implementation or its strengths and weaknesses, but to gather perceptions about the LRCP’s approach and interventions considered most and least effective for triangulation with other data. The questionnaire was administered in person during the LRCP regional meeting in Antigua, Guatemala, which members of partner organizations from all five LRCP countries attended. This provided the team an opportunity to explain the survey’s purpose and enter responses on a computer during the survey. Unlike the online survey, the survey questionnaire collected identifying information (i.e., was not anonymous). DATA ANALYSIS METHODS DOCUMENT REVIEW The initial document review was instrumental in determining the stage of completion of deliverables and results across the region and in each country and developing a better understanding of the logic, or theory of change, embedded in the program’s components. The team conducted subsequent document analysis for all countries throughout the evaluation as needed, in addition to analyzing country-level and regional data relevant to the LRCP. Document reviews involved reading and synthesizing key information to: (1) identify the specific intervention strategies in each country and their underlying and/or explicit theory of change; (2) identify initial information answering the evaluation questions and determine gaps in information that required further verification; (3) determine key aspects in which evidence produced by/about the LRCP has revealed a need for adjustment and if (or which) adjustments have been made; and (4) identify areas that require more in-depth analysis through empirical data collection. Tables and matrices allowed the team to organize and synthesize the information from the documents. Country-level tables recording activities, dates, and stakeholder involvement were especially important because goals and implementation in each country varied widely. These country-specific tables helped the team organize and visualize the goals and achievements to date, challenges and obstacles, key stakeholder/partner involvement, and adjustments made. ONLINE SURVEY The evaluation team exported survey data from Survey Monkey to Excel for data cleaning and analysis. Data cleaning involved identifying and removing duplicates, analyzing outliers and response distributions, and spot-checking for response patterns, missing data, and formatting issues. Once the data was sufficiently cleaned, the team began analysis using descriptive statistics, including frequencies, cross-tabulations, and response options where applicable. Open-ended survey responses were treated as qualitative data and 11 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV analyzed through content analysis. Survey data were disaggregated by sex, country, position/role, and type of organization. Characteristics of the survey respondents, who are also the intended beneficiaries of LRCP, are presented in Error! Reference source not found.. The majority of the responses came from educational institutions and MOEs, followed by local NGOs. Most were technical specialists, teachers, and directors, followed by project managers and teacher supervisors (see Figure 2). TABLE 3. NUMBER OF ONLINE SURVEY RESPONDENTS BY COUNTRY AND CATEGORY OF EMPLOYER Country or Region Local NGO Intl. Orgn. Other Gvtl. Orgn. MOE Educnl. Inst. Private Sector Univ. No Response Total Dom. Rep. 4 3 4 24 10 0 26 1 72 Guatemala 11 10 2 13 15 4 5 3 63 Honduras 23 6 3 7 12 7 13 2 73 Jamaica 4 0 2 22 28 2 3 2 63 Nicaragua 22 15 1 0 9 2 8 4 61 U.S./Other 0 2 0? 0 0 1 0 0 3 Total 64 36 12 66 74 16 55 12 335 Most of the survey respondents (82 percent) were female (264 females and 59 males). As for education level, most had either a bachelor’s degree (41 percent) or a postgraduate degree (39 percent). STRUCTURED QUESTIONNAIRE SURVEY Data from questionnaires were organized by question and topic, and responses were grouped and categorized. These categories were linked with topics, categories, and codes from the survey, interviews, and FGDs. In addition to grouping the findings by topic, this allowed for triangulation of data. KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS The team organized data from KII and FGD notes and analytic memos by themes using data matrices. Themes and theoretical categories were based on the evaluation questions, program goals, expected results, and intervention logic. Data displays allowed the team to systematically analyze the data, clearly identify trends and patterns, and draw conclusions by country and on a regional level. For example, matrices consolidated findings by country and result, showed similarities among responses, and allowed the team to pull representative quotations by respondent type (e.g., partner organization, MOE official, teacher). Similarly, the team documented differences among responses, with quotations and by informant type. Teacher 21% Teacher supervisor 11% Director 19% Project manager 14% Consultant 3% Technical specialist 29% Other 3% Figure 2. Online survey respondents by title USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 12 TRIANGULATION Triangulation of data sources (e.g., KII/FGD respondents and project reports), methods (e.g., online survey and KIIs/FGD), and evaluation team members’ analyses helped ensure the validity of the analysis. For example, the online survey data reflected familiarity with and use of the different LRCP products by actor and country; data from KIIs and FGDs also provided such information, but on a smaller scale. Triangulation helped the team determine whether any discrepancies existed in the data. GENDER ANALYSIS The team applied gender-analysis frameworks in several ways. First, KIIs/FGDs were intentionally carried out with both women and men from the partner organizations when possible. In MOEs, men tended to be in director positions more than women; interviews were carried out with both directors and technical staff of both genders when possible. Second, KIIs and FGDs included probing questions about the participation and role of women versus men in the program. Additionally, the evaluation team collected data on the institutional structures of the implementers and partner organizations in order to identify trends in gender roles and analyze the implications this may have for implementation. Finally, the team analyzed participation in events (such as conferences and workshops) to identify whether and/or where there are gender gaps in participation and program benefits. LIMITATIONS INCOMPLETE RESPONSES. The structured survey questionnaire was administered to all participants at the regional meeting in Antigua. Several respondents skipped some questions or did not complete the questionnaire, despite in-person follow-up. Nevertheless, most participants completed the questionnaire. In analyzing the data, rather than counting responses to questions (which would lead to uneven responses by country), the team used a more nuanced approach by identifying overall trends and trends by country. Although the online survey yielded a high response rate, a relatively high number of respondents skipped several questions. The skip logic in Survey Monkey should not have allowed for skipping any questions; however, this did not work for all questions and all respondents because of a programming issue. This was resolved by calculating percentages based on the number of responses to the question, rather than the number of survey responses. In some cases, people indicated their opinion about something even when they had also indicated that there were unfamiliar with it; the team did not count these opinions in the analysis. BIAS. Implementers and partner organizations may have been biased in answering questions about strengths and challenges of programs for they are responsible. This limitation was partially overcome by triangulating their responses with information from interviews and survey responses. The team contrasted implementers’ perceptions with beneficiary perceptions and self-reported benefits. The online survey respondents were self-selecting, leading to a potential selection bias, in that beneficiaries who have either positive or negative feedback are typically likely to respond more than beneficiaries with more neutral views. However, the 42 percent response rate is higher than the typical 30 percent rate for online surveys and respondents were from all the intended categories representing the beneficiary population, thus reducing selection bias. 13 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV DIFFICULTY MEASURING INTERVENTION EFFECTS. The LRCP has interventions aimed at building capacity to improve EGR; however, the design of the program did not anticipate how this would be measured. There is no clear learning agenda for a specified group of beneficiaries whose skills and knowledge can be assessed to determine the degree of improvement. Nor can early literacy skills of children be the proxy for assessing capacity to improve them. Thus, data for measuring program effects such as “improved capacity” were largely assessed through self-reports and opinions. This limitation could not entirely be overcome. However, it was partially mitigated by consistently asking respondents for concrete examples, such as what knowledge they have acquired from program interventions and how they have used it, as well as what LRCP resources they have used and the purpose and frequency of use. RESOURCE AND TIME CONSTRAINTS. There were issues faced in transcribing and translating all the KII and FGD data due to resource and time constraints. The evaluation team members conducted KIIs and FGDs until the end of January 2019 because of delays from the U.S. government shutdown, which was not the originally planned timeline. This made it infeasible to have the verbatim transcriptions and translations necessary for coding the KII and FGD data. Instead, the evaluation team used the KII/FGD detailed notes that captured informants’ responses, as well as summaries and analytic memos from KIIs/FGDs, in order to extract data and organize it in data charts and displays to document findings and respective evidence. Transcribing would have helped to better code data and carry out some other analyses, such as having precise counts of frequencies of specific topics and opinions mentioned and numbers of corresponding respondents. FINDINGS EVALUATION QUESTION 1 How effective has the LRCP been in achieving its goals by result and by country to date? Evaluation Question 1.A: Where have the most resources and levels of effort focused, by result, by country, and to date? LRCP implementation has varied in the specific objectives and interventions in each country. Each Mission had a different bilateral EGR program as well as its own programmatic and support needs. Jamaica and Peru had projects that came to an end during LRCP implementation, and the goals of the LRCP in Haiti changed significantly when the program expanded to include technical support to the MOE’s early literacy curriculum redesign and materials development. Thus, the Dominican Republic, Guatemala, Honduras, Jamaica, and Nicaragua are the countries actively participating in LRCP with similar scopes of work, while the other three countries have had different scopes of work under LRCP. Activities in Guatemala started later than in other countries because the partner organization, Asociación de Investigación y Estudios Sociales (ASIES), was not on board until February 2017. See Error! Reference source not found. for a summary of country-level implementation of the LRCP. Activities and achievements for each expected result vary by country. In part, this is due to the flexible nature of the program that allows it to respond to mission-specific needs rather than achieving uniform goals and results across the region. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 14 TABLE 4. OVERVIEW OF THE LRCP INTERVENTION IN EACH COUNTRY Country USAID Mission EGR Bilateral Project/ Program Organizational Arrangement for the LRCP Participation in LRCP Regional-level Interventions LRCP-specific Interventions Dominican Republic Project Reads (2015– 2020) Implementer: Universidad Iberoamericana (UNIBE) Coordinating organization: Juárez Local partner: consultant works directly with the MOE Evidence collection Dissemination Capacity Strengthening RedLEI Stakeholder analysis Country profile LRCP database Dissemination Training events Guatemala Lifelong Learning Project (2014–2020) Implementer: Juárez and Associates Coordinating Organization: Juárez Local partner: ASIES Evidence collection Dissemination Capacity Strengthening RedLEI Stakeholder analysis Country profile LRCP database National database Dissemination Training events Haiti Let’s Learn to Read and Write (2015– 2016) Implementer: FHI360 AIR expanded LRCP support to ministry of education ‘s early literacy curriculum redesign Capacity Strengthening (Sharing and exchange with ESC countries) Support to conduct country visits initially in 2016; AIR opened a technical office in Haiti to assist with curricular redesign. Honduras Honduras Reading Activity (2017–2022) Implementer: Education Development Center (EDC) Coordination Organization: AIR Local partner: FEREMA Evidence collection Dissemination Capacity Strengthening RedLEI Stakeholder analysis Country profile LRCP database Dissemination Training events Jamaica Enrichment Initiative to Increase Literacy at The Primary School Level (2013– 2016) Implementer: Digicel Foundation Coordinating organization: Juárez Local partner: Educonnect Evidence collection Dissemination Capacity Strengthening Stakeholder analysis Country profile LRCP database Dissemination Training events Nicaragua Community Action for Reading and Security (2013–2019) Implementer: DevTech Coordinating organization: AIR Local partner: CIASES Evidence collection Dissemination Capacity Strengthening RedLEI Stakeholder analysis Country profile LRCP database National database Dissemination Training Events Peru Amazonia Reads (2012–2017) Implementer: Two regional governments and the Universidad Peruana Cayetano Heredia (UPCH) Coordinating organization: AIR Local partner: the Universidad Peruana Cayetano Heredia (UPCH) Capacity Strengthening Support provided to USAID to facilitate a relationship of engagement with regional governments to UPCH on integrating gender in education; to have regional exchanges Eastern and Southern Caribbean (ESC) Early Learner’s Programme (ELP) (2015–2019) Implementer: The Organisation of Eastern Caribbean States (OECS) Coordinating organization: Juárez Local partner: There is a Memorandum of Understanding among AIR, OECS, and USAID Evidence collection Dissemination Capacity Strengthening Support provided to the ESC countries to carry out Early Grade Reading Assessment (EGRA) baseline and endline, participation in training (Bloom software); technical support to ELP 15 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Resource distribution across the region also varied. Table 5 shows the amount of resources provided to each country to date; Figure 3 shows resource distribution by percentage. Haiti has roughly double the budget of all of the other countries put together, which is primarily due to the support to the MOE for the early literacy curriculum redesign. In terms of the other countries, Guatemala has received the highest amount of financial resources, partly because there are two local partners there (Universidad del Valle de Guatemala [UVG] and ASIES). The next highest amount is for Jamaica, followed by Nicaragua and Honduras. The Dominican Republic has received a relatively small amount. Resources for all five participating countries have been used to produce the country profiles and stakeholder analyses, gather resources for the LRCP database, hold dissemination events, carry out dissemination activities such as sending out information and using media to promote EGL9 awareness, and carry out capacity￾strengthening activities. Guatemala has received more resources, as it is the hub of the LRCP activity. Furthermore, resources have gone to build RedLEI to carry out research on EGL, which has been based in Guatemala. Overall, the LRCP core team members (from the prime, sub, and partner organizations) reported that they have had the necessary human and financial resources to meet the program’s goals. In the structured survey questionnaire, all responded “yes” to the question about having sufficient human resources to achieve the goals of the LRCP, and all except one responded “yes” to the question about having sufficient financial resources to achieve the goals. The regional nature of the LRCP allows for sharing of human and material resources, which was pointed out by most members of the implementing and partner organizations. For example, one of the overall strengths of the program is the high level of technical expertise of the LRCP core team members. They have provided support to different participating countries based on particular country needs and the specialization of each LRCP member. Furthermore, implementing partners of USAID’s bilateral projects have provided support to LRCP partner organizations and beneficiaries in other countries. This is discussed further under Evaluation Question 1.E. 9 Implementing and partner organizations use ‘EGL’ instead of ‘EGR’ because of the importance they place on writing as being as fundamental to literacy development as reading. Thus, ‘EGL’ is used throughout this report. TABLE 5. BUDGET AMOUNT BY COUNTRY Country Amount Dom. Rep. $203,858 Guatemala $2,081,691 Haiti $8,571,465 Honduras $559,703 Jamaica $1,014,667 Nicaragua $628,681 Honduras 4% Nicaragua 5% Guatemala 15% Jamaica 7% Dom. Rep. 7% Haiti 62% Figure 3. Distribution of financial resources by country USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 16 Evaluation Question 1.B: What have been the most successful activities under each result, by country, to date, and for what reasons? Expected Result 1: Evidence on EGR Outcomes is Collected, Consolidated, and Systematized for Practical Use by Stakeholders in the LAC Region The LRCP core team members, members of the partner organizations, and most beneficiaries interviewed pointed to the LRCP database, stakeholder analyses and country profiles, and systematic review as concrete achievements of the program. The systematic review and database are regional products, while the stakeholder analyses were country-specific and produced in the Dominican Republic, Guatemala, Honduras, Jamaica, and Nicaragua. Survey and KII/FGD data from all categories of informants report the stakeholder analysis to be the most valuable LRCP product because it identifies key EGL actors that LRCP interventions can catalyze and complement to bolster EGL improvement efforts. The LRCP website was launched in April 2016 and resources were collected and consolidated by partner organizations to make them accessible in the LRCP website database starting in August 2017. As of September 2018, there were a total of 1,748 uploaded resources. Resources are in Spanish and English (some are in French and Portuguese) and include studies, evaluations, teaching aids, and other resources on EGL and related topics, such as bilingual learning, literacy development and learning disabilities, and pre-literacy. Additionally, in Guatemala ASIES created the database Guatemalalee.org in response to the needs of key actors to inform the general public about EGL. This digital magazine provides public access to research, materials, and pedagogical resources. Another achievement highlighted by members of the partner organizations in Guatemala, Honduras, and Nicaragua was the participation of the partner organizations in the review and consolidation of information for the database, which generated a sense of ownership by members of these organizations. Members of partner organizations interviewed have a plan for ensuring that the database will be managed and continually updated by them or another local entity. For example, in Honduras, members of Fundación para la Educación Ricardo Ernesto Maduro Andreu (FEREMA) aim to have the Universidad Pedagógica Nacional Francisco Morazán (UPNFM) manage and update the database to ensure its sustainability. Almost all the beneficiaries interviewed were familiar with the LRCP database, especially those who have had more involvement in the program. Several informants gave examples of how they have used it. For example, an MOE staff member in Guatemala who is taking the LRCP online course on EGL research needed to find studies on the topic, which he was able to do on the database. In this case, participation in the course provided an opportunity to discuss the research methodologies and findings. Another example was an MOE staff member in Guatemala who stated that “the material that was downloaded from the website [LAC Reads] has been useful for showing examples [to teacher trainers] of how to work with teachers.” One of the most common responses relative to the utility of the database was that it eases finding studies and experiences from other countries and other regions in the same country. As an MOE staff member in Honduras stated, “I have visited the database and looked at some studies from other countries, so it really caught my attention, and I started downloading information.” Learning about these other experiences gave her ideas for techniques to share with teachers to develop children’s reading fluency. Several beneficiaries from local organizations in Guatemala, Honduras, and Nicaragua also indicated having used the database to find resources. As stated by a member of a Nicaraguan NGO: “I have had access to some topics, for example, how we are working on comprehensive multi-grade literacy, as well as reading 17 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV comprehension in early learners. With this I have been able to pass on this knowledge to MOE staff and teachers. It is very useful to share with other actors.” Of the online survey respondents, 16 percent were familiar with the LRCP database and, of these, 66 percent indicated that it was “useful,” and six percent indicated it to be “very useful.” A higher percentage of stakeholders was familiar with the LRCP website: 46 percent of survey respondents indicated that they were familiar with it. The number of visits to the LRCP website is tracked by LRCP staff. This is broken down by country in Error! Reference source not found.. As the table illustrates, most website visits are from the United States. The majority (76 percent) of website visitors are female, and 90 percent are first-time visitors. LRCP staff also track database use through capturing the number of times it is consulted, or the number of “hits.” The visit time is also recorded, which was an average of one minute and 24 seconds. As can be seen in Table 7 below, database use varies widely among the countries, with the highest number of hits being outside the five participating countries. Within the LRCP participating countries, Jamaica showed the highest total number (451), followed by the Dominican Republic (215) and Honduras (110). Guatemala and Nicaragua reflect the fewest “hits.” In the case of Guatemala, this may be explained in part by the late start of LRCP activities. In the case of Nicaragua, beneficiaries also use the virtual library of Universidad Céntroamericana (UCA). TABLE 6. LRCP WEBSITE VIEWS, APR. 2016–FEB. 7, 2019 Country Page Views Percent of Total U.S. 10,141 59.97 Nicaragua 501 2.96 Honduras 1372 8.11 Guatemala 957 5.66 DR 706 4.18 Jamaica 1,368 8.09 Other 1,864 11.02 Total 16,909 TABLE 7. DATABASE USE, APR. 2016– SEP. 2018 (AIR DATA ANALYTICS) Country No. of Hits on the Database Guatemala 2017: 8 2018: 51 Honduras 2017: 15 2018: 87 Nicaragua 2017: 19 2018: 44 Dominican Republic 2017: 113 2018: 102 Jamaica 2017: 35 2018: 416 Other 2017: 569 2018: 594 Total (Regional) 2016: 35 2017: 759 2018: 1,294 TABLE 8. DOWNLOADS FROM LRCP WEBSITE, APR. 1, 2016–FEB. 7, 2019 Country Downloads Percent of Total U.S. 874 46.59 Nicaragua 78 4.16 Honduras 256 13.65 Guatemala 251 13.38 DR 87 4.64 Jamaica 194 10.34 Other LAC 118 6.26 Other 18 0.96 Total 1876 USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 18 As illustrated in Table 8, document downloads are also tracked for the program. It is important to note that these are resources that are on the LRCP website database, not ones for which links on the database redirect users. Because many resources are accessed through links, many resource downloads may go undocumented. However, based on the information available from the program database analytics, almost half of the downloads are from the U.S. (47 percent), though Honduras, Guatemala, and Jamaica each reflect more than ten percent of downloads. Though relatively low, this does show that there is some use and demand for the information on the LRCP website. Even considering the relatively low numbers of document downloads from the LRCP website, beneficiaries from all categories of informants in Honduras, Guatemala, and Nicaragua cited the web pages of the local partners as sources of information with respect to materials generated (or strategic allies in the case of Nicaragua with the UCA virtual library). The most commonly downloaded resource from the website was the systematic review (full document and executive summary), which accounted for 32 percent of downloads, followed by the country profiles (19 percent). The latter were downloaded in Spanish (for country profiles from Nicaragua and Honduras) and English (for Jamaica). The systematic review downloads, however, were in English; the executive summary that was translated into Spanish was not on the website. The production of the systematic review, stakeholder analyses, and country profiles was done intentionally at the beginning of the LRCP to provide the necessary information for effective implementation. This was perceived to be a strength of the program by all members of implementing and partner organizations interviewed as well as several beneficiaries. In particular, the stakeholder analyses provided the necessary information to initiate strategic dissemination and capacity building in each country, while the systematic review identified research gaps from which a research agenda was defined. As one informant from an international NGO in Nicaragua stated, “proposing the program from this information has been a key starting point to contribute to improving early literacy.” The first systematic review was published in 2016, and an updated version was published in 2017. For LRCP core team members and members of partner organizations, it was considered important because it set a precedent in the region of carrying out a scientifically rigorous review methodology of research on EGL from or on the LAC region.10 A few characteristics of the systematic review are: ● It offers educational policy designers and curricular and teaching specialists a guide to quantitative and qualitative research pertinent to the regional reality. ● Based on the findings, reading comprehension success and sustainability strategies are analyzed for the regional context. ● It identifies gaps in the evidence regarding EGL in LAC with respect to the best practices in the world. This gap in evidence presents challenges for key stakeholders who attempt to make evidence-based decisions. ● Recommendations provided outline the type of research that is needed to fill gaps in the evidence base. 10 In 2016, from a universe of 9,696 articles, 108 met the high-quality standards established for inclusion and were used in the review. In the update issued in 2017, 34 out of the 772 identified articles met the standards and were included in the review. 19 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV The systematic review update in 2017 identified which research gaps in the 2016 review had to some extent been addressed, based on new research from 2016. Three areas were identified: (1) research on EGL for children with disabilities; (2) pre-reading and pre-writing; and (3) reading comprehension development. Of the 13 percent of survey respondents who were familiar with the systematic review, 75 percent considered it “useful,” and three percent found it “very useful.” Its usefulness was described in interviews as “making us aware of the lack of causal research on EGL” and “making us aware of the need to use rigorous research evidence for decision making.” Several members of partner organizations and staff from MOEs (in Jamaica, Guatemala, and Honduras) mentioned that the systematic review helped them become aware of their own limitations in terms of research on EGL and motivated them to do more research on areas where there were gaps. For example, in Jamaica, EduConnect worked with MOE staff to go into schools and collect information on what is missing for effective EGL instruction. They identified several gaps that they have been trying to address, including having culturally relevant reading materials for students. The stakeholder analysis was useful for partner organizations and local stakeholders in various ways. In Honduras, FEREMA used it to develop a targeted dissemination and capacity-building strategy by differentiating actors by category. As a member of FEREMA explained, “We made a sub-classification, which was, we classified organizations and actors based on their roles; within an organization there may be people who are decision makers but also people who are technicians, so their role varies.” In the case of Jamaica, the stakeholder analysis helped the partner organization identify key private-sector stakeholders with whom to form alliances. MOE officials expressed the importance of the stakeholder analysis to them because they needed to know who is doing what to improve EGL: “I have put looking at the website as second priority. I am far more keen at looking at the stakeholder analysis, and we are very glad the LAC Reads Program has done the stakeholder analysis because it has saved the Ministry a lot of work.” Another achievement of the stakeholder analyses, as pointed out by partner organizations in all countries, was that each of them had direct participation of members of partner organizations and indirect participation of other local stakeholders. This participation generated a sense of ownership. Additionally, LRCP provided training on research methods, such as conducting focus groups, to equip staff from partner organizations to conduct data collection for the stakeholder analysis. Expected Result 2: Dissemination of Up-to-Date Knowledge about EGR Practices is Targeted to Diverse Audiences and Stakeholders Dissemination activities have varied across the LRCP participating countries, with some activities being similar across the region. All five participating countries have held conferences on EGR/EGL targeting MOE directors and technical staff, teachers, teacher coaches, university and teacher college professors and researchers, members of international and national NGOs, and others working to improve EGL (students, consultants, etc.). The LRCP was launched in all five countries; the results of the systematic review and stakeholder analyses were presented in all five countries as well. In Honduras and Guatemala, other dissemination activities include spaces to discuss education and policy issues (called “conversatorios”), in which education leaders and decision makers participated. Partner organizations in all countries have held events as one of the main ways to share information about the LRCP and the program documents. All partner organizations contributed to the LRCP database with USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 20 resources from their countries. Similarly, partner organizations in all countries have targeted the media and sent out informative bulletins and newsletters to key stakeholders in education and specifically EGL. Additionally, they have used their webpages, blogs, and Facebook pages to disseminate information about the LRCP in addition to messages about the importance of EGL, EGL evidence and practices, and other topics related to EGL promotion. In Jamaica, a television commercial was also produced. Finally, partner organizations in Guatemala, Honduras, Jamaica, and Nicaragua have also participated in national education events in order to promote the importance of EGL and evidence to inform EGL practices. In the case of Guatemala, because of the delay in starting work with a partner organization as part of LRCP implementation, dissemination began in 2018, and fewer dissemination activities have occurred. In Honduras, FEREMA carried out a very targeted dissemination approach, catering events and information to different audiences. In Nicaragua, information has been made available to organizations and the public on the UCA’s virtual library,11 which consists of 160 books and 110 digital documents, plus a list of approximately 90 references created in the country in the last ten years. One third of survey respondents learned of the LRCP by being invited to an event (see Figure 4). However, even though all survey respondents had attended an LRCP event, roughly two thirds had learned of the LRCP in another way. About 38 percent learned of it through work or colleagues. This was the case across all the countries, although it was the most prevalent in Honduras (see Figure 5). Guatemala was the country with the highest percentage who learned about the LRCP through the news, though this percentage was relatively low across all countries. There is an indication, based on KIIs and the survey questionnaire, that the program’s dissemination processes have reinforced the efforts to place EGL on the agendas of MOEs, with the exception of Nicaragua, and, through mass media and public events, LRCP has strengthened efforts for EGL to become more central for civil society. This is especially true for Jamaica and Honduras. As stated by a USAID official in Honduras: “I do not think 11 See https://www.uca.edu.ni/index.php/comunicacion/noticias/2110-uca-cuenta-con-nueva-biblioteca-virtual-con￾apoyo-de-ciases-y-lac-reads. 0 5 10 15 20 25 30 35 Dominican Republic Guatemala Honduras Jamaica Nicaragua Figure 5. How did you learn about the LRCP? (by country) Figure 4. How did you learn about the LRCP? (online survey) Public news 4% Contacted by someone in program 17% Invited to an event 32% Through colleagues 11% My organization is a member of LRCP 9% Through the organization I work with 27% 21 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV it has generated a policy as such, but it has strengthened a series of clear messages that have been happening in the country through our USAID projects. It is not new, but it does support the messages being communicated, such as the significance of literacy.” Similarly, a member of an international NGO in Nicaragua commented: “There is a greater awareness of the importance of reading and writing; this is fundamental. This process of sensitization has allowed us to think together in search of solutions, in ways to improve the results that the country has, and how we together can make these improvements.” Across the participating countries, partner organizations have aimed to get media coverage of the program’s dissemination activities and other EGL-related issues. Though not directly attributable to the LRCP, the M&E indicator “number of articles mentioning EGR” also provides a gauge for assessing the importance of EGR on the public agenda (see Figure 6). Guatemala and Honduras are the countries where there is the most notable change in the number of articles mentioning EGR, each showing a sharp increase from 2017 to 2018. According to online survey respondents, studies on EGL and on emergent literacy and program documents (e.g., stakeholder analysis, country profile, systematic review) were the most useful resources on the LRCP website (see Figure 7). This is consistent with KII and FGD data, in which these resources were the most frequently mentioned (see Table 9). About three quarters (76 percent) of respondents indicated that studies on emergent literacy were “useful” (62 percent) or “very useful” (14 percent), and 73 percent indicated that EGL studies were “useful” (63 percent) or “very useful” (10 percent). It is interesting to note that 27 percent of respondents indicated “neutral” (24 percent) or “moderately useful” (3 percent) regarding the utility of EGL resources. On a country level, the majority of these were from Nicaragua, for which 47 percent indicated “neutral” or “moderately useful,” followed by the Dominican Republic, where this number was 36 percent, Guatemala (28 percent), and Honduras (19 percent). This complements the KII data in which the topics of methodological approaches and language of instruction were mentioned as issues that were very context-specific, thus making research resources on these topics not always applicable in different countries. For example, KII and FGD informants in Nicaragua said that there is a strong need for more resources on intercultural and bilingual EGL, since that is one of the major challenges they face. In the case of Guatemala, interview informants pointed out that research on EGL and emergent literacy often did not consider bilingualism. In the case of Jamaica, informants felt that many of the EGL resources did not pertain to their context and were inaccessible owing to the language barrier. In the Dominican Republic and Honduras, there are current debates regarding the most effective methodological approach to EGL. 29 2 4 21 2 136 11 66 9 48 2 330 94 174 6 46 10 Regional Guatemala Honduras Nicaragua Jamaica Dom. Rep. 2016 2017 2018 Figure 6. Number of articles mentioning EGR (M&E data) USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 22 Figure 7. Usefulness of information on the LRCP website (online survey) TABLE 9. USEFULNESS OF INFORMATION ON THE LRCP WEBSITE (ONLINE SURVEY) Reply Program Docs EGL Emergent Literacy Other Educational Issues Research Methods Learning Assessments Teaching & Learning Teacher Education Bilingual Education Intercultural & Bilingual Education Very useful 17 11 14 17 20 24 20 20 29 30 Useful 51 68 62 40 39 40 52 38 25 21 Neutral / Mod. useful 37 29 24 34 33 27 24 29 29 29 The LRCP has contributed to more discussion on methodological approaches for EGL acquisition. In all participating countries except Jamaica and Guatemala, there are current debates and disagreements regarding the phonetic or communicative approaches. Informants from all different categories of actors pointed to the LRCP as a context to help test the different approaches and seek common ground and/or evidence supporting the most effective approach. In Honduras, informants from the national university, or UPNFM, and the MOE said that the LRCP has helped to shine light on the debate on methodological approaches and the need to look at evidence on these. Similarly, informants from local and international organizations in Nicaragua commented on the need to “test different approaches in the local context” in order to determine the most effective ones. In Guatemala and Jamaica, informants from all different categories of actors perceived the LRCP to be a platform that could potentially facilitate debates and decisions regarding multilingual education. In Guatemala, informants from local NGOs, international NGOs, and USAID emphasized the need for Program documents Studies on early grade literacy Information on research methods Teaching and learning resources Information on learning assessments Studies/information on teacher education Studies/information on intercultural bilingual education Studies on emergent literacy/pre￾literacy Studies on other education issues Studies/information on bilingual education Graphic Key = Neutral/Moderately Useful = Very Useful = Useful Each icon represents 10 responses. 23 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV further public debate on bilingual literacy development. According to a program coordinator from a local NGO, there needs to be a “common front” on how children should acquire two languages. Several informants, including staff from USAID, implementers, and the partner organizations, explained that some donors as well as the Vice Minister of Intercultural and Bilingual Education promote the learning of both languages simultaneously (as in the dual immersion model), which he confirmed. USAID promotes a transitional model, in which children develop literacy first in their first language and then transition to using a second language in school. Likewise, in Jamaica, there is a language policy debate regarding the use of Patois for early learning and whether it should be taught in its written form. A regional event on language policy supported by the LRCP shed light on this issue for the national coordinator who stated that this “discourse on policy is crucial in Jamaica due to its complex multilingual context.” It is noteworthy that the majority of respondents did not indicate “information on research methods” as being useful. By countries, Honduras and, to a lesser extent, Guatemala were the only countries where there was a higher indication of this information being useful. This is consistent with data from KIIs and FGDs, in which several MOE technical staff members as well as members of local organizations gave examples of accessing EGL studies from the LRCP database. Across the countries, the issues of early grade writing and emergent literacy came up as topics that were very important for EGL improvement but that had not received enough attention in available research literature or teaching resources. At least 70 percent of all categories of actors indicated that studies on emergent literacy were “very useful” or “useful”. This was the case for EGL studies as well, except in the case of directors. Teacher supervisors and project managers found program documents to be useful; project managers found teaching resources useful, as did project directors. For the case of information on learning assessments and information on research methods, only project managers and technical specialists tended to find these useful. Expected Result 3: Capacity is Strengthened at Institutions So They Can More Easily Implement Proven Approaches to Improve EGR Outcomes for Poor and Disadvantaged Children One of the most evident contributions of the LRCP has been to strengthen the abilities of members of partner organizations and beneficiaries through workshops, conferences, courses, symposia, and national and international meetings. This is the only component (or Expected Result) of the LRCP that has benefited all the countries, including the ESC countries, Haiti, and Peru. This will be further addressed below under Evaluation Question 1.E (“Most valuable aspects of the LRCP for Missions”). In the five participating countries, LRCP carried out capacity strengthening through: (1) workshops and other events; (2) regional and national-level sharing of experiences; (3) joint work on products such as the stakeholder analyses and the LRCP database; and (4) a regional online course on EGL. LRCP carried out workshops on EGL and related topics in all five participating countries. Workshop topics varied by country and responded to the needs identified by the partner organizations in each country (see Table 10). USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 24 TABLE 10. CAPACITY-BUILDING ACTIVITIES BY COUNTRY Dominican Republic Guatemala Honduras Jamaica Nicaragua EGR workshop Replication of EGR workshops in different regions of the country Pre-reading and pre￾writing conference Pre-reading and pre￾writing workshop Emergent literacy, transition to primary workshop Workshop on pre￾reading and pre￾writing Workshops on writing opinion articles Symposium on evidence use for EGL Workshop on evidence and EGL Replication of EGL workshops in different regions of the country Pre-reading and pre￾writing workshop EGR workshop for teachers (central and municipal levels) Workshops on systematization for MOE staff and members of international and local NGOs Early reading conference EGR training in schools Writers workshops (reading material production) for MOE personnel, teachers, editors, writers EGL workshops (central and departmental levels) Workshops on intercultural and bilingual education in EGL Pre-reading and pre￾writing workshop Workshops on reading comprehension (for teachers and teacher coaches in different regions) Workshops on promoting EGL (for parents) Workshops on EGL in multi-grade classrooms Source: LRCP Quarterly Reports The excellent reputation of the partner organizations as well as their close coordination with the MOEs has been a strength in all countries, according to USAID staff, members of the implementing and partner organizations, and personnel of MOEs. This has facilitated capacity building for MOE personnel in addition to other beneficiaries (such as members of NGOs, civil society organizations, and others).12 Across all LRCP countries, the partner organizations held workshops and conferences beyond the capital city in different regions of the country; this was also done through beneficiaries replicating the activities.13 Beneficiaries of capacity strengthening events were teachers, teacher coaches, MOE directors and technical staff, professors and researchers from universities and teacher training colleges, members of international and local organizations, and others with work or interest in EGL (e.g., consultants and university students). 12 The exception is Nicaragua, because of the political climate and inability to work with the government. 13 At the time of data collection, activities in Guatemala were initiating, as opposed to other countries where implementation had begun much earlier. After data collection, activities occurred outside of the capital. For example, as part of the rollout of the stakeholder analysis in March 2019, ASIES presented in three departments of the country, in addition to Guatemala City. Additionally, in January 2019, opinion-writing workshops were conducted in Quetzaltenango for University Panamericana master’s students studying literacy in intercultural and bilingual contexts through an agreement with the Lifelong Learning project. 60 39 14 91 57 8 26 173 47 60 26 28 10 Regional Dom. Rep.Guatemala Honduras Jamaica Nicaragua 2016 2017 2018 Figure 8. Number of organizations that participated in training (M&E Data) 25 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV As shown in Figure 8, as of 2018 members of 324 organizations had participated in capacity-strengthening activities, with Honduras reaching the highest number of organizations. As shown in Figure 9 below, Jamaica has reached the highest number of stakeholders who completed capacity-strengthening activities. Note that the number of organizations reached was highest in Honduras, while the number of stakeholders was highest in Jamaica. This is partly explained by there being more organizations in Honduras. In addition, it may be because more LRCP events in Jamaica had more attendees from the MOE. Although Guatemala has a very high number of EGL-related organizations, capacity strengthening did not begin until 2018. However, in 2018, a relatively high number of organizations and individual stakeholders were reached in Guatemala. Figure 9. Number of stakeholders who completed professional development activities (M&E data) Capacity-building activities started in 2016 and have intensified over the years. The greatest number of stakeholders participating in training was in 2018. Many more women than men have participated in LRCP training activities. This can be partially explained by the higher number of women in the teaching service and in education in general, except for in the most senior positions. Across the countries, the most common reason for participating in a training event, after “interest in the subject,” was “in order to join/form teams for EGL improvement” (see Figure 10, next page). This is mirrored by KII data, in which the members of all partner organizations and the majority of beneficiaries (from the MOE and local and international organizations) in all LRCP countries highlighted the advantages of creating networks at a national and regional (LAC) level to share experiences and resources on EGL. Frequently mentioned by beneficiaries in these categories was the importance of creating a critical mass of people working in EGL who use evidence to inform practice. Online survey data also reflect that beneficiaries value the training events for establishing these connections and working jointly to improve EGL. Regarding the usefulness of training events, survey respondents from all categories of actors indicated that conferences on EGL were the most useful, followed by conferences on emergent literacy and other 81 86 1 167 186 25 0 50 100 150 200 250 300 2017 Male Female 28 11 22 15 12 3 152 23 51 239 210 120 0 50 100 150 200 250 300 2018 Male Female USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 26 conferences with international/national experts (see Figure 11). The stakeholder analysis presentation was indicated as being less useful than the conferences, yet it was not applicable for a quarter of respondents. This is consistent with KII data in which the majority of beneficiaries had not attended the stakeholder analysis presentations. Regarding the EGL workshops, almost all beneficiaries who had attended them named them as important for providing valuable EGL knowledge and resources. Some beneficiaries provided examples, such as a staff member from the MOE in Honduras who said: “The first training helped us put into practice what we were getting from the program because we had the opportunity to share materials and bibliography with other colleagues in the region. In Guatemala, for instance, the material available on literacy was good. We learned about this material through the program and we adapted it to be implemented with the teachers in Honduras.” Another example was provided from a member of a local NGO in Nicaragua who stated, “During the forums and workshops held with the regional specialist, we work on the indicators to know whether a teacher is working on the basic skills, as well as the skills of the children, in a multi-grade classroom setting.” In Nicaragua, according to Coordinators from Fe y Alegría and Save the Children, the training received by the program strengthened their conceptual and methodological abilities to improve early literacy, giving them tools that were shared and replicated among the territories’ MOE technical staff, directors, and teachers. As Save the Children’s Education Coordinator stated: “Before, when this early literacy matter started, the teacher prepared children for the EGRA test making them practice reading, fluency being the great challenge, to see if children could read the words in 60 seconds; in that case, comprehension was left out. Nowadays, there is a better analysis, in addition to comprehension and fluency, in terms of diction, phonetics, knowledge of graphemes and punctuation. All these issues are seen more articulated. There has been a great amount of progress.” Other organizations, such as DevTech, the Fabretto Foundation, Save the Children, and Centro de Servicios 0 5 10 15 20 25 30 35 40 Dominican Republic Guatemala Honduras Jamaica Nicaragua Figure 10. Motives for participating in LRCP activities (online survey) 30 66 47 67 74 14 22 21 19 16 9 4 7 2 3 47 8 25 12 7 Online course Conf. expert Pres. Stake. An. Conf. emer. lit. Conf. EGL Very Useful Useful Neutral/MU N/A Figure 11. Usefulness of LRCP Events, by percentages (online survey) 27 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Educativos en Salud y Medio Ambiente (CESESMA), also developed skills to apply learning assessments (EGRA), which allowed them to involve MOE technical staff and directors in early literacy learning measurement. The workshops on systematization that were carried out in Honduras were also reported as being very useful for participants due to their continuity and practical application of learning to systematically document best EGL practices. Most of the beneficiaries interviewed in Honduras highlighted these as one of the program’s main achievements due to the skills it developed in participants, as well as the resulting products. According to MOE personnel who attended the workshops, “the space to systematize those [isolated] experiences, training experiences, successful experiences, classroom experiences, was generated through the [LAC Reads] program.” According to a professor from the UPNFM, “All of the colleagues who participated in the three workshops on systematization thought they were phenomenal, and we each did a systematization project and sent the reports to FEREMA.” This was one of the activities in the region that had practical applications and continuity, as there were three workshops that built on each other. Furthermore, in coordination with the MOE, these were replicated in various departments and municipalities with regional MOE staff and teachers. Another example of training with practical application is the workshops in Guatemala for writing opinion pieces, which resulted in published articles. Participants in these workshops referred to them as being very useful for gaining practical and useful skills. Survey respondents were asked about the utility of training events (see Figure 12). Out of the 134 respondents who answered that question, all selected “I have gained new knowledge on teaching methodologies,” 53 percent (71 out of 134) indicated that they have put what they learned into practice, and 34 percent indicated that they have shared and/or replicated what they learned with others. Another 37 percent indicated that they gained new knowledge about education research. Figure 12. Reasons for usefulness of LRCP events (online survey) Project managers, teachers, and technical specialists indicated that they put what they learned to practice with the second highest frequency (after gaining knowledge of teaching methodologies), followed by replicating what they learned for teachers and project managers. The second highest frequency for directors and teacher supervisors was “knowledge of education research”. In all categories, some participants indicated that they put what they learned into practice and replicated it. 0 5 10 15 20 25 30 35 40 45 50 Consultant Director Project Manager Teacher Teacher Supervisor Technical Specialist Knowledge teaching methodologies Knowledge measuring learning Knowledge Ed. research Put what was learned into practice Replicated/shared what was learned USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 28 The online course in EGL is offered regionally to members of partner organizations, MOE staff, and staff from local organizations. There are 40 people in total taking this course, coming from the Dominican Republic, Guatemala, Honduras, and Nicaragua. Four informants (from KIIs) were taking the course, all of whom said that it offered excellent readings and that they were learning a lot. One informant from the MOE in Guatemala pointed to the value of the course’s practical applications, such as participants observing a class and documenting the EGL practices in order to share and discuss with participants from other countries. Expected Result 4: Sustainability Platforms in Place to Continue and Strengthen EGR Interventions in Countries in the LAC Region Sustainability platforms to strengthen EGR interventions are sought through LRCP in several ways. Per the indicators in the M&E plan, the establishment of alliances or networks to support EGL improvement is considered a proxy of sustainability, as is the creation of laws or policies to improve EGL. Sustainability is also sought through the institutionalization or establishment of a platform to continue the interventions of the LRCP and the promotion of EGL improvement, such as strengthening EGL research capacity, maintaining the database, and carrying out EGL research in the LAC region. The platform is conceived as a network of universities, which began to form at the end of 2017 through a sub￾award with the UVG. The main achievements for this expected result are: (1) the formal establishment of RedLEI; (2) the establishment of strategic alliances; (3) the official launching of RedLEI and its website; (4) the realization of a study by the RedLEI universities on teacher pre-service curriculum for EGL; (5) the funding of research fellows with the support of the Global Reading Network; and (6) the design of a master’s program in EGL. A memorandum of understanding (MOU) was signed between the UVG and the five founding members of RedLEI, which are the UCA in Nicaragua, UPNFM in Honduras, Universidad de Costa Rica (UCR), the UCA in El Salvador, and Coordinación Educativa and Cultural Centroamericana. All universities are highly regarded nationally and regionally and, as such, bring prestige and quality to RedLEI. Through these universities, RedLEI is carrying out a regional study on what teachers need to know and be able to do to effectively teach children to read and write. This will inform teacher pre-service curricula and thus make a valuable contribution for improving EGL in the region in the long term. In the short term, it can improve EGL practices through informing in-service teacher training. The funding and technical support obtained by the Global Reading Network is an achievement that offers the opportunity to generate research from the region that responds to the needs identified in the systematic review. It also offers an opportunity to further develop the capacity of the researchers via mentoring from RedLEI researchers. In the first phase of support, ten research fellows were selected to receive financial and technical support to carry out studies on key EGL research gaps in Central America as identified in the systematic review. These include three studies in Guatemala, one in El Salvador, two in Honduras, two in Nicaragua, and two in Costa Rica. The second phase was funded by RedLEI and consisted of scholarships to produce and disseminate knowledge related to EGL through carrying out and publishing research studies. Scholarships were awarded to nine fellows across the RedLEI countries: one in El Salvador, one in Honduras, two in Nicaragua, and five in Guatemala. 29 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV In terms of establishment of alliances or networks to support EGL improvement, a proxy of sustainability, the LRCP reporting reflects a total of 14 to date. These alliances include the RedLEI MOUs, the MOU with the Organisation of the Eastern Caribbean States (OECS) to carry out the baseline and endline EGRAs of the USAID bilateral program, alliances with MOEs, and alliances with other networks and organizations. As evidenced in interviews with partner organizations and the MOE personnel in Guatemala, Honduras, and Jamaica, the alliances with the MOEs have helped to establish strong working relationships with various departments and levels (e.g., curriculum specialists, in-service teacher training, special education). As a teacher coach in Honduras commented: “FEREMA has met my expectations, and they have done innovative things in our community to the point where I feel at home. They have been my strongest alliance to be able to grow in the process of (teaching) reading and writing.” In Jamaica, EduConnect established alliances with the private sector, such as one with a book publishing company who partially supported the launching of LRCP. This has helped give more visibility to the program. In Guatemala, the strategic alliance formed with the Gran Campaña Nacional para la Educación (GCNPE), or in English, Great National Campaign for Education, is considered an achievement by the implementers, USAID staff, members of the partner organization, and UVG staff. The GCNPE was initiated in 1999 by civil society organizations with the aim of demanding that the government increase the education budget in order to improve education quality. It is currently made up of 70 organizations and entities including universities, churches, media networks, foundations, research organizations, private organizations, and international and local NGOs.14 According to the UVG’s Vice Rector for Research, RedLEI can provide the GCNPE with information to influence policies. There is no evidence of laws or policies being created as a result of the LRCP to date, but other actions have been taken to influence government priorities and plans.15 There was variation in the opinions of implementers and stakeholders regarding how important these were for effecting change in EGL. According to data from the structured questionnaire survey, participants from all countries thought this should be the first or second priority out of seven options, except for Guatemala, where it was listed it as one of the lowest priorities. Survey respondents, however, indicated this to be the lowest priority for improving EGL. In Guatemala, various informants including USAID staff and a high-level MOE official, expressed that they did not think it was important to establish a policy, but rather implement existing laws and policies to strengthen education quality. In this instance, ASIES is developing a proposed policy agenda for the next government in which EGL will be highly emphasized. While not a law or policy, this action could directly affect implementation of new programs and resources to strengthen EGL. Similarly, USAID staff in Nicaragua highlighted the impact that Centro de Investigación y Acción Educativa Social (CIASES) had on the national education plan; its contributions led to the incorporation in the plan of early literacy learning assessments. Since then, the MOE has performed standardized tests, although the results are not publicly known or discussed. The universities in RedLEI, both public and private, have and can assume roles to help sustain the LRCP’s interventions. For example, the UCA in Nicaragua has been responsible for hosting the database resources as a virtual library on its website. Beneficiaries across the categories of actors interviewed pointed to the UCA’s website as a valuable resource. Implementers, partner organizations, and UVG staff have envisioned a similar role for the UVG. In Honduras, the UPNFM can play a key role 14 La Gran Campaña Nacional para la Educación (Fundaungo, 2015). 15 It is important to note that, despite the monitoring and evaluation reporting of a total of four laws or policies being established, it is not clear from the quarterly reports or KII/FGD data what these are. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 30 in strengthening research capacities and carrying out research where there are gaps, similar to the UVG. As a UPNFM professor pointed out, as a university, they can access funds for research if it will be carried out under their leadership. Another achievement cited by members of implementing and partner organizations, the UVG, and RedLEI is the design of the master’s program in EGL, which is scheduled to launch in September 2019. The master’s program has the potential to offer rigorous training on EGL theory and practice, which could help continue strengthening the MOE staff, staff from organizations implementing EGL programs/training, and other key stakeholders working to improve EGL. RedLEI’s sustainability has been examined in recent months, in tandem with a study that was carried out to analyze and recommend ways to achieve this.16 While some members of the UVG stated that they think the master’s program can generate revenue toward RedLEI’s sustainability, the sustainability study points out that this will not be sufficient revenue and that RedLEI must carefully define its research agenda. In addition, a business approach has not been employed until now, which could have been beneficial from the start. Finally, one of the biggest steps toward the sustainability of EGL platforms is the strengthened capacity of a critical mass of people working toward improving EGL who are conscious of the need to use evidence for informing practice. According to informants from implementing and partner organizations and local and international NGOs in Guatemala, Honduras, and Nicaragua, this critical mass of people has begun to form with LRCP interventions. Evaluation Question 1.C: What have been the greatest challenges or setbacks to achieving each result, by country, to date, and for what reasons? Expected Result 1: Evidence on EGR Practices and Outcomes is Collected, Consolidated, and Systematized for Practical Use by Stakeholders in the LAC Region LIMITED USE OF EVIDENCE-BASED INFORMATION. One main challenge that LRCP faces is ensuring that stakeholders in the region use the evidence-based information generated by the program. Though most key informants among the beneficiaries indicated being familiar with the database, this does not seem to extend beyond those who have been directly involved in the LRCP implementation, such as MOE staff. Furthermore, though most MOE personnel interviewed were familiar with the database, few said that 16 Sustainability Assessment: Central American Network of Early Grade Reading, RedLEI. November 2018, O’Brien and Associates International. 13 16 25 87 84 75 Systematic Review LRCP Database Stakeholder Analysis Familiar Not familiar Figure 13. Percentage of respondents familiar with LRCP products (online survey) 31 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV they had explored it or used its resources. The most common reason given for this was a lack of time. As mentioned earlier, according to online survey data, the majority of respondents (54 percent) were not familiar with the LRCP website. However, as shown in Figure 13, a much lower percentage of online respondents was familiar with the specific LRCP products, even though these respondents had participated in LRCP events. Out of 255 online respondents who answered this question, 210 (87 percent) indicated that they were not familiar with the systematic review, and 84 percent were not familiar with the LRCP database. Overall, more people across the region were familiar with the stakeholder analyses than the systematic review or database, making up a quarter of the respondents across the region. LACK OF AWARENESS AND USE OF THE SYSTEMATIC REVIEW. Though information on website use reflects that the systematic review (full document and executive summary) has been downloaded more than other documents, this does not necessarily mean that it is for use in the LAC region, given that roughly half of downloads are from the U.S. Furthermore, the systematic review (full version) is not available in Spanish, and though there is a Spanish version of the executive summary, it is not on the website or database. While some of the KII/FGD beneficiaries were aware of the systematic review, very few had looked at it. The majority of LRCP implementing partners interviewed and four KII respondents familiar with the systematic review indicated that it was too technical to be useful for most targeted beneficiaries of the LRCP, and that the process of producing it could have been more inclusive, as well as have included some practical applications for using the information it produced. As one informant put it, people from the MOE could not put its results into practice and it contributed to a “gap between the academic community and the practitioners.” Similarly, several international and local NGO informants from Nicaragua pointed out that they attended the presentation of the systematic review, but the participants did not have access to the presentation itself—the results were not discussed with organizations to identify priorities and outline a strategy to develop capacity. Thus, even if it wasn’t meant for widespread use, apparently it generated an expectation in stakeholders beyond implementing partners and partner organizations regarding its use. According to five informants including implementers, an independent consultant, and a member of an international NGO, there are several aspects that could have helped make the systematic review be more useful: (1) stakeholder participation in its development; (2) making it more accessible to understand and use by LRCP beneficiaries; and (3) translation into Spanish. The review was carried out by personnel of AIR, except for the design phase, which had input from the LRCP core team. This decision was made because it needed to get done in order to inform implementation and needed a certain level of expertise and access to information such as journals. However, more involvement may have also resulted in more ownership and thus more usage of the review. Additionally, because it is a very particular kind of document, it is difficult to understand and use for beneficiaries such as MOE staff, teachers, and NGO project implementers. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 32 As shown in Figure 14, while more people were familiar with the database and stakeholder analysis, a relatively high percentage (36 percent and 28 percent, respectively) indicated that they were neutral about their usefulness or did not find them very useful. Finally, the emphasis on challenges to using evidence to inform practices varies across Guatemala, Honduras, Nicaragua, and Jamaica. Guatemalan respondents tend to identify the main challenge as not knowing how to apply research evidence to changes in practices, before converting it into policy change. Another commonly cited challenge was the need to locate and produce research evidence on the teaching of early literacy skills in bilingual contexts. In Guatemala and Honduras, more dissemination of evidence to promote changes in classroom practice was stressed, while in the case of Nicaragua, the need to generate local information through experimental research, or to implement international evidence-based practices in local contexts, was emphasized. In Jamaica, members of the partner organization and the MOE stressed the importance of developing capacity to carry out action research. Expected Result 2: Dissemination of Up-to-Date Knowledge about EGR Practices is Targeted to Diverse Audiences and Stakeholders LACK OF MASSIVE DIFFUSION. The lack of massive diffusion is a challenge that was identified by the various categories of actors interviewed. While the general perception was that the materials compiled and produced by the program have a lot of potential to illuminate EGL practices, the majority of stakeholders in Guatemala and Nicaragua suggested that greater diffusion is needed to achieve an impact at the classroom level.17 Most diffusion strategies tended to rely on events to share products, which limits dissemination to only those who attend. Beneficiaries from USAID, the MOEs, and NGOs perceived that there was a lack of follow-up to provide resources to participants, such as a copy of the presentation or outreach to stakeholders who could not attend. In this way, the strategy was not adequately comprehensive. As one LRCP team member put it, “the dissemination strategy should have been thought through strategically from the start.” LACK OF PRACTICAL APPLICATIONS. Informants from the MOE across the countries also pointed to the need for events to have practical applications and ways of applying what is shared in an activity or product. As stated by a member of the literacy team in the MOE in Jamaica, “There need to be more outputs… if there are more such conferences and they are accessible, that will extend the research and, if in these conferences people are taught more on how to do all this research and there is follow-up and other social 17 Achieving impact at the classroom level is not the mandate of LRCP, but rather the desired effect in the long term, per the LRCP’s theory of change. Thus, beneficiaries perceive LRCP potentially contributing to improvement in EGL outcomes. 0% 20% 40% 60% 80% 100% Systematic Review Stakeholder Analysis LAC Reads Database Partner Org. Database Very useful/useful Neutral Mod. useful/Not useful Figure 14. Usefulness of LRCP products 33 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV opportunities that can show people to do action research, then the outcome of the program will be a lot more efficient.” LACK OF A COHESIVE COMMUNICATION STRATEGY. Various members of implementing and partner organizations indicated that there was a lack of a cohesive communication strategy from the beginning to guide each partner organization. The common vision was that the communication strategy should have involved the communication specialists be integrators among the countries. This would have helped them to have a common message based on commonalities, such as the need for effective EGL teaching. Furthermore, the turnover and skillset of the communication specialists in the first years of the program has been a challenge. What was required was someone who specializes in communication and can manage a database. This was not in place for the first two years, resulting in a lack of guidelines for the local partners. Expected Result 3: Capacity is Strengthened at Institutions So They Can More Easily Implement Proven Approaches to Improve EGR Outcomes for Poor and Disadvantaged Children LACK OF CONTINUITY OF TRAINING EVENTS. One challenge of capacity strengthening has been the lack of continuity in events. While some events were useful to beneficiaries in order to acquire information (e.g., about the stakeholder analysis results, the database, or best EGL practices), events to develop new knowledge and skills tended be perceived as less useful. KII/FGD and survey data revealed that most training events lacked follow-on to allow for the process of knowledge/skill development. For example, beneficiaries who had participated in events in Guatemala and Jamaica indicated that events seemed to respond to particular identified needs, but that they did not build upon each other. Members of the LRCP core team and partner organizations also indicated that continuity had not been fully achieved. Online survey data shows lack of continuity of activities to be one of the reasons that the LRCP activities were viewed as being less useful (see Figure 15). Related to continuity is limited time—the most frequently cited reason for low-scoring usefulness by survey respondents across the countries. 0 4 8 12 16 20 It lacks relevance for the context of my country The technical level is too high The information has not been practical There was not enough time There was no continuity The activity did not provide the opportunity to learn through doing Dominican Republic Guatemala Honduras Jamaica Nicaragua Figure 15. Reasons an LRCP activity was not useful (online survey) USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 34 Informants from the MOEs in Guatemala and Jamaica and international and local NGOs in Nicaragua said that they appreciated the LRCP events where they could learn valuable information but that there is a need for a more comprehensive and systematic training strategy. This would lead to the development of professionals who specialize in different abilities to improve EGL. LACK OF PRACTICAL APPLICATION FROM WORKSHOPS. The third most frequently indicated response by survey respondents as to why LRCP activities were not useful was that they did not allow for learning through practical application or by putting things into practice. In Nicaragua, KII data reflect that several beneficiaries who had participated in training activities mentioned that the methodology did not allow for learning practical skills. These reasons were also mentioned by a few beneficiaries in Guatemala and Honduras. This is important to consider for future training activities. When there was a practical application and/or outcome from the workshop such as the systematization reports in Honduras, informants (from partner organizations and the MOE) mentioned that they were unsure how they would be used. As stated by a professor from the UPNFM, “One of the biggest strengths of the program is the systematization of experiences so then I wonder, what do we do with it? ...I think we need to elevate these findings to a category associated with decision making, probably by the Ministry of Education.” In other words, while it is valuable that research capacity is strengthened through the production of research and EGL resources, it is equally valuable that the outcomes of this are used to inform practice and decisions. Expected Result 4: Sustainability Platforms in Place to Continue and Strengthen EGR Interventions in Countries in the LAC Region Sustainability is a challenge for all projects and programs, whether they are implemented at a national or regional level. However, sustaining regional-level resource sharing and capacity strengthening without financial support presents an additional challenge. While the universities participating in RedLEI have collaborated thus far to define research agendas, establish RedLEI’s mission, and perform other initial tasks, they must have a motive to collaborate once funding ends. In the case of the CETT initiative, a regional network was not created, but rather participating LAC countries worked together to develop state-of-the-art teacher guides and a teacher training program. This regional collaboration ended with the funding; however, components of CETT have been adopted by several of the countries’ MOEs. In the case of the LRCP, there are also tangible outcomes and products that can continue to support EGL improvement, though there will need to be continual technical support, ideally through RedLEI. CONTINUOUS UPDATING OF THE LRCP PRODUCTS. The LRCP products are valuable because they take stock of the existing stakeholders working to improve EGL (stakeholder analyses) and existing research on EGL in the LAC region (systematic review) and offer up-to-date EGL resources from the LAC region (LRCP database). For these to remain useful, it will be necessary to update them on a regular basis. Because RedLEI was conceived as a way to sustain the LRCP’s efforts, it would be their role to lead that process. In the case of Guatemala, ASIES has been responsible for collecting and housing the resources. However, it is unclear if they will continue this role after the termination of LRCP because they will not have the LRCP staff members and they do not specialize in education. The website Guatemalalee.org will need to be hosted by another entity, such as RedLEI/UVG. TRAINING YOUNG RESEARCHERS. A strategy of LRCP, and specifically of RedLEI, is to train young researchers in the region in EGL research. However, this has been a challenge, according to members of 35 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV the implementing partners and partner organizations in Guatemala and staff from the UVG. They explained that the appropriate people for this role are already working, and that taking time from working to specialize in research is not possible for most people. Furthermore, they were unsure how many people want to specialize in EGL research and how to ensure that this would be lucrative for them. GENERATING RESOURCES. It is not apparent that RedLEI has in place a realistic plan for generating resources to sustain RedLEI. Some members of the UVG and RedLEI believe that the master’s program in EGL can generate revenue toward RedLEI’s sustainability. However, as the sustainability assessment points out, this will not be enough revenue.18 While the first cohort will receive funding from the LRCP, funding will not be there for future cohorts. Thus, RedLEI must generate income to support participants or other funding sources must be explored. A series of suggestions for doing this are provided in the sustainability assessment. MAINTAINING THE ALLIANCES ESTABLISHED UNDER LRCP. Important alliances at a national level have been established in all the LRCP countries. These include MOE personnel centrally and, in different regions of the countries, other organizations, networks, and private entities. Because activities to develop these alliances were funded by the LRCP, keeping them active without funding may be a challenge. The remaining months of the LRCP offer the opportunity for RedLEI to strengthen these alliances, establish new and strategic ones, and develop mechanisms and plans for joint work toward EGL improvement. MAINTAINING MOMENTUM AND CRITICAL MASS FOR SUPPORTING EGL. The interventions of the LRCP have begun to generate a new type of capacity in stakeholders: EGL knowledge and awareness of the importance of using evidence to inform practice. A momentum has been generated in a broad range of stakeholders. Across the LRCP participating countries, informants from different categories of actors expressed concern that this momentum may not continue if support stops too soon. Voluntary open responses to the survey question, “If the LRCP concludes its support now, has it achieved its goal of strengthening capacity to improve EGR based on proven best practices?” indicated a high level of consensus: of the 43 people who wrote a response, 56 percent believed that the project should continue because there is a lot left to accomplish. As one respondent wrote, “I believe there is more work that could be done with an extension of perhaps a year or two especially in the areas of training and public education to boost private sector support and family involvement.” The priority areas to continue strengthening, according to respondents, are “Develop teaching strategies and methodologies for teachers that are specialized and systematic” (82 percent), “Give higher priority to EGL in the national budget and education policies” (52 percent), and “Strengthen the methodological capacity to verify practices that improve EGL” (43 percent). These priorities are similar to those mentioned in most of the KIIs and FGDs and indicated in the structured questionnaire survey. Evaluation Question 1.D: What evidence is there that local partner institutions and key stakeholder institutions have increased capacity to improve early literacy in their home countries? This question was addressed under Evaluation Question 1.B: “What have been the most successful activities under each result, by country, to date, and for what reasons?” under Expected Result 3. 18 Sustainability Assessment: Central American Network of Early Grade Reading, RedLEI. November 2018, O’Brien and Associates International. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 36 Evaluation Question 1.E: What have been the most valuable aspects of the LRCP for USAID Missions? The demand-driven and flexible nature of the LRCP has been beneficial for Missions having needs that LRCP could respond to. For those with EGL bilateral projects, in most cases, the LRCP has had the intended effect of bolstering the projects’ effects and Missions’ goals. The LRCP has effectively complemented USAID bilateral project interventions that directly affect teachers in almost all of the LRCP countries. Additionally, the regional nature of LRCP was highlighted as being of value across Missions, due to the sharing of resources and experiences and the benefits of this for the EGL work supported by each Mission. Specific benefits to Missions are detailed as follows. USAID/ESC values the support that LRCP has provided to strengthen the implementation of the bilateral program that supports the ESC countries. The LRCP support is primarily to carry out EGL learning assessments at baseline, midline, and endline to evaluate the impact of the Early Learner’s Programme (ELP) implemented by the OECS. This has also helped to build capacity within the MOEs to assess EGL. The other components of the LRCP have been less visible to USAID/ESC staff but were highlighted by the OECS, such as: (1) training in in Bloom software in Guatemala for participants from the ESC region, which has allowed them to develop culturally and linguistically relevant EGL books and which they subsequently replicated in Jamaica with EGL stakeholders; (2) partnerships and sharing, especially with Haiti, which was supported by the LRCP initially but now by the ELP; and (3) technical guidance from the LRCP core team members to critically examine the ELP’s implementation and make the necessary adjustments. In the Dominican Republic, the LRCP complements the bilateral program primarily through capacity-strengthening activities. The LRCP holds EGL events and invites key people involved in Project Reads, better equipping them to implement the project. The LRCP invited 250 people to an EGL conference, then 40 technical staff from the Dominican Republic Ministry of Education (MINERD) were selected to participate in a workshop; they developed plans to work with teachers to better equip them for EGL instruction. A USAID/Dominican Republic staff member noted that this helps teachers develop a much-needed systematic approach to EGL. The stakeholder analysis was also helpful for this staff member, since it helped her better understand the education situation in the Dominican Republic, especially in regard to EGL. Capacity strengthening is also carried out through the online EGL course, in which two MINERD members participate. LRCP has been beneficial to USAID/Dominican Republic in that it facilitated overcoming tensions through methodological approaches, with a focus on EGL evidence and theory. This has shed light on the debate about approaches: the MINERD is implementing the communicative approach while USAID promotes a phonetic approach in its EGR projects. Bringing evidence to the table can help the discussion focus on evidence rather than ideology. Finally, EGL studies that were not disseminated were found in universities; the LRCP was instrumental in disseminating these studies. In Guatemala, the LRCP has complemented the bilateral program through capacity strengthening and the emphasis on EGL evidence. USAID/Guatemala staff members reportedly appreciate LRCP’s capacity building of MOE personnel, with whom they work closely, as does the implementing partner of the bilateral program, Juárez and Associates. According to USAID/Guatemala staff, LRCP has made “great strides” in strengthening MOE capacity and this momentum is important to keep up. The focus on evidence-production, such as funding researchers to study EGL issues in Guatemala, is important, as is the dissemination of evidence. The fact that this is the same implementer as LRCP, 37 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Juárez and Associates, has helped create synergy; for example, LRCP has put documents from the current USAID bilateral project, the Lifelong Learning Project (LLP), and the past one, Education Reform in the Classroom (REAULA, Spanish acronym), on the database. The stakeholder analysis is also deemed to be an important contribution for the MOE and USAID/Guatemala; it has allowed them to know where different key actors in EGL are working and ensure they do not “step on each other’s toes,” especially with the concentration of actors in the same regions who sometimes promote different approaches to bilingual literacy development. The work at the policy level is also considered to be important for reinforcing the policy work of the bilateral program. The LRCP has not been perceived as having provided a particular value-added for USAID/Honduras, even though there is recognition of the overall value of its interventions. There is not much familiarity with the LRCP activities and products in Honduras; however, respondents recognized that LRCP has produced important and useful information. According to USAID/Honduras education staff, there has not been ongoing communication with people involved in the LRCP, which could have in turn led to more communication between the bilateral EGL projects and the LRCP. USAID/Honduras staff members acknowledge that it is necessary to have integration between the bilateral projects and the work that the LRCP is doing nationally and regionally, yet it is less clear who should take that initiative. Because their large bilateral projects take priority, it is unclear who will take this initiative to integrate them with LRCP. However, though there is not coordination, the LRCP is strengthening the MOE’s capacity for EGL improvement, which can impact the effectiveness of the bilateral projects in improving EGL. For USAID/Jamaica, LRCP has helped to keep education as a national priority. The bilateral program in Jamaica ended in 2016; thus, the LRCP has provided welcome support for bolstering education. According to a staff member at USAID/Jamaica, the LRCP has helped to keep education as a priority for the government and has helped to “keep us moving forward” in education. EduConnect and MOE staff also perceive the LRCP as important for generating awareness of the importance of knowing EGL best practices and of EGL research for informing practice. There has been ongoing communication between EduConnect and USAID/Jamaica. USAID/Jamaica staff has also participated in some of the events, which were deemed to be creative and helpful for improving EGL. USAID/Nicaragua values LRCP because it has both reinforced the interventions of the bilateral programs and successfully carried out capacity-strengthening and policy-level work. As explained by a USAID/Nicaragua staff member, the LRCP responds to USAID/Nicaragua’s priorities in education, as well as the strategic guidelines for education in the government’s National Human Development Plan. For example, based on a review of the National Education Plan done by CIASES, USAID managed for the MOE to incorporate guidelines for learning assessments in early literacy. Additionally, the LRCP provided direct training to technical staff and teachers, including staff from the MOE, in the municipalities where the USAID EGL bilateral programs are being implemented. USAID/Nicaragua’s bilateral projects are working to improve bilingual and intercultural EGL, which is a value-added that LRCP has not yet provided but could, as pointed out by a USAID staff member. USAID/Peru values the education expertise LRCP provided for implementation of its education intervention. In the initial years of LRCP, USAID/Peru’s bilateral program was phasing out and they had a lack of education expertise. They successfully sought out the LRCP for assistance in establishing relationships with the regional governments that they had funded. The LRCP’s technical USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 38 expertise was also provided to the bilateral implementing partner for integrating gender in education materials. Thus, though the initially planned LRCP activities were not carried out, the LRCP responded to higher-priority needs of USAID/Peru. EVALUATION QUESTION 2 What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy￾level interventions? Evaluation Question 2.A: What are the strengths and weaknesses of the technical approach as structured (featuring a principal prime and principal sub-prime, as well as multiple national-level partners)? What alternative modalities might be considered for a program with similar or related goals? Overall, the arrangement between the prime organization, subcontractor, and the country-level partner organizations has worked very well, according to members of implementing partners and partner organizations, as well as Mission staff members who could speak to this issue. As pointed out by several members, the prime and sub-prime organizations had worked together on an education project previously (Basic Education Project in Nicaragua) and thus had a pre-established working relationship. Furthermore, oversight for countries was divided based on previous country-level implementation experience. Thus, implementers had already established relationships with the MOEs and other key education stakeholders in their designated countries. The arrangement of dividing oversight by country also allowed them to be efficient in achieving their products. Another strength cited by informants from implementing partners and partner organizations is that the LRCP core team worked together closely to carry out LRCP activities. They jointly discussed and designed products and interventions, which helped them enhance what was developed and make sure it was appropriate for the context(s) in which LRCP is operating. Partner organizations across the countries are very respected and institutionally strong, according to almost all the informants interviewed. They are highly regarded by the MOEs and other key education stakeholders, as well as having established partnerships with both. This has allowed for a wide geographical reach across the countries, as well as institutional strengthening in the MOEs at various levels and across technical departments. While the collaborative approach to making decisions and carrying out activities was deemed a strength by most of the LRCP core team members, members of partner organizations across the countries perceived bureaucracy in getting products approved, due to so many individuals having input. It was not always clear to them with whom to communicate or from whom to seek approval. Additionally, given the complexity of the organizational structure, USAID noted some initial challenges manifested in operational delays or communication gaps. The organizational structure of multiple partner organizations (one in each country) has allowed for resource sharing and mutual learning, as discussed above. However, it has also resulted in each country achieving different outcomes at different times. Each organization planned its activities on a different timeline, which affects the uniformity and synergy of sharing interventions and products across the countries. In cases where the partner organization was brought on board late (such as in Guatemala), or did not continue (such as in the Dominican Republic), the timing of activities varied even more. According 39 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV to several members of implementing partners, the late start in Guatemala has been an obstacle to achieving the expected results relative to other LRCP-participating countries. The lack of uniformity most remarked upon by informants was the communication strategy. Members of implementing partners and partner organizations mentioned that each created its own communication strategy, though the effectiveness would have benefited from applying dissemination strategies more uniformly. Evaluation Question 2.B: What are the strengths and weaknesses of having results with flexible indicators not tied to performance indicators (such as improved literacy levels)? It is difficult to track progress toward the objective and expected results through indicators for a program such as LRCP, because it does not have a fixed group of beneficiaries that it aims to benefit, through pre￾defined interventions such as an EGL program that trains a group of teachers to develop specific competencies. Furthermore, successfully achieving the goal of improving EGL rests on critical assumptions such as: (1) if evidence on EGL is made available, key stakeholders will use it; and (2) if they use it, EGL outcomes will improve. The national-level nature of the interventions in each country makes it challenging to clearly track results related to EGL outcomes attributable to the program. As such, most of the program’s M&E indicators are input and output indicators, but not outcome indicators. However, some of these are good proxies to indicate change toward the expected results beyond an input. For example, “number of hits on the database” is an indication of database use by country. As part of this indicator, the LRCP is also keeping track of the number of downloads and which resources are downloaded, which is also a good indication of evidence demand. However, it falls short of measuring whether and how it is used. The indicators to measure dissemination are also input indicators. The number of activities and number of participants are helpful for gauging the amount of activity taking place across the countries. Yet there is also an indicator aimed to capture the number of people who report increased EGR knowledge—which may provide helpful qualitative information if it is thorough and rigorous—such as members of partner organizations and other beneficiaries giving concrete examples of what they learned and how they are applying it. This is partially done through the pre- and post-surveys administered at the events. However, it does not reflect actual applications of what was learned. Similarly, indicators for the second expected result aim to capture the breadth of capacity-building activities as well as the depth or effect of them. Indicator 12 (“The number of organizations with strengthened capacity”) aims to do this via talking with members of the organization and following a protocol that asks about the benefits of the training. Measuring this indicator in such a way could be helpful to gauge the degree to which capacity strengthening is effective. Finally, for sustainability, reporting on the number of alliances formed is a helpful indication of conditions being put in place to continue strengthening EGL and is attributable to the LRCP. However, the indicator to report laws or policies created to improve EGR is not attributable to the LRCP. The LRCP’s policy work, such as influencing government education plans, is not captured by any indicator. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 40 Evaluation Question 2.C: What are the strengths and weaknesses of this regional literacy programming that focuses at the policy or systems levels? Given the stated objectives, the scope of activities, and investment levels, are there alternative approaches, models, or activities to supporting increased early literacy outcomes that USAID should explore? LRCP’s regional approach has allowed for cost-efficient sharing of resources and transferring capacity across countries, especially for partner organizations, to strengthen EGL knowledge and skills over time. It has created a network across and within countries that is a learning community, which motivates those involved to continue learning together. It is an ongoing process rather than just being a network to accomplish a single goal. A strength of the systems-level approach is that capacity is built with key stakeholders responsible for implementing EGL and/or creating the conditions for effective EGL, such as the MOEs, teacher training colleges, universities, and international and national NGOs. In other words, LRCP seeks to strengthen the capacity of key stakeholders who ultimately oversee developing guidelines, curricula, materials, and training for teachers. With this increased capacity, EGL improvement efforts can have sustainable outcomes, assuming knowledge and skills continue to be used to improve EGL. For example, two staff members of the MOE in Guatemala who participated in LRCP EGL workshops mentioned two decisions they made baesd to their newly acquired EGL knowledge. One is a revision of the teacher-preparation curriculum for pre-primary (kindergarten and pre-kindergarten) teachers to ensure it is based on EGL best practices; the other is to create a diploma program for departmental coordinators to develop EGL knowledge and skills. Regarding the policy-level approach, KII/FGD data reflect informants’ positive perception of the value of the approach of reinforcing EGL messages through media and creating a critical mass of stakeholders with a vested interest in improving EGL and strengthening EGL policy. For example, comments such as the value of “reinforcing key messages such as the significance of literacy,” “more awareness of the importance of reading and writing,” and “thinking together in search of solutions” to improve EGL were mentioned by a number of stakeholders from all countries and categories of informants. While these effects don’t automatically lead to policy change, they generate the conditions to facilitate it. One of the strengths of the LRCP has been having experts and researchers in Central America to share expertise and help generate contextualized research in the region; this was highlighted by most of the beneficiaries, members of partner organizations, and implementers interviewed. As a staff member of Juárez and Associates put it: “LRCP begins to put Central America as a producer of knowledge rather than a consumer. The foundation is being created for this. It must continue to be strengthened.” The approach of LRCP to empower local researchers was also cited as a strength of the program by members of implementing partners and partner organizations, as well as by several beneficiaries from the MOEs and NGOs in Guatemala, Nicaragua, and Honduras. The effects can be seen in RedLEI, where researchers from the partner universities are gaining expertise in research skills. Furthermore, instead of bringing a recipe that is ready to be circulated among teachers and stakeholders of the educational community, the LRCP fosters the need to develop a methodological decision-making culture, based on empirical evidence. This is seen in KII/FGD data across all LRCP countries and all categories of informants though many comments, such as on the need to test EGL methodologies in local 41 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV contexts, carry out experimental research, and conduct action research. Thus, there is a considerable advantage beyond the immediate effects. Partner organizations and other EGL stakeholders have benefited from information dissemination and training events; however, while the partner organizations have benefited from sustained engagement, this was not necessarily the case for other EGL stakeholders. One of the biggest challenges is the utilization of research results by the education community. The program encourages the use of evidence and for stakeholders to carry out research; several informants (from the MOE and NGOs), however, mentioned that they need such accompaniment. The exception to this is EduConnect’s work with the MOE in Jamaica and the participants in the online course that, in both cases, did research on classroom practices. Another example could be LRCP or RedLEI providing guidance in doing such research beyond individual researchers. Stakeholders from NGOs who support EGL efforts stressed the importance of carrying out experimental research to test methodological approaches in local contexts, which is research that LRCP or RedLEI could guide. In addition to carrying out research on EGL practices, alternative modalities could be designed with the aim of creating a critical mass of stakeholders with the capacity to translate research findings into practice. These may be in the form of more online courses or a series of workshops that guide participants in the practical application of implementing research findings, designing research, conducting action research, and carrying out research on EGL. Now that stakeholders have been identified and a critical mass has been formed, this could be the next phase of the LRCP. Additionally, platforms with key stakeholders from MOE, research organizations, universities, and the private sector could be formed with the objective of defining a research objective that responds to actual policy or program decisions (i.e., need-driven research). This would more likely lead to evidence use. Evaluation Question 2.D: What are the strengths and weaknesses vis-à-vis USAID Mission benefits and engagement? It is beneficial to Missions to be able to count on the LRCP to provide technical assistance for EGL-related tasks, as well as other Mission needs in education. It is also a benefit that their primary education counterparts, the MOEs, benefit from capacity strengthening. The database, stakeholder analyses, and other products of the LRCP offer valuable information to Missions that they can use to make decisions about their programs and design future programs. Missions can engage to the degree they are interested and/or want to engage with the LRCP. In some cases, this has resulted in high engagement. They do not have the task of managing the program, but they benefit from its actions and effects. While the LRCP was meant to bolster the effects of the education projects in participating countries, this has happened to varying degrees and in diverse ways. However, it has not happened in some countries, such as in Honduras, which may result in missed opportunities for creating synergies to improve EGL. One reason may be that the bilateral program had not been established before the onset of the LRCP. Also, when it did begin, there was not an effort to strategize about how the LRCP could complement the bilateral projects. There was no established mechanism to help ensure that coordination and/or joint planning takes place between the LRCP and the bilateral projects. According to members of the USAID/Honduras Education Office, coordination and planning meetings would help integrate the program and bilateral projects; since these are not required or planned, they do not take place. This articulation will likely have to come from someone who knows both the LRCP and the bilateral program, such as someone from LAC/RSD. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 42 CONCLUSIONS EVALUATION QUESTION 1 How effective has the LRCP been in achieving its goals by result and by country to date? 1.A: Where have the most resources and levels of effort focused, by result, by country, and to date? In the five countries currently participating in all components of LRCP, resources have gone primarily into capacity building and the LRCP products (stakeholder analyses, systematic review, and the LRCP database). In the other countries, resources were primarily used for Mission priorities related to supporting the uptake and continuity of investments in early literacy improvement, such as to successfully implement their education/EGL programs (as in Haiti, Peru, and the ESC). 1.B: What have been the most successful activities under each result, by country, to date, and for what reasons? RESULT 1. LRCP has consolidated and collected evidence in the LAC region through the systematic review, the LAC Reads database, and the stakeholder analysis; the stakeholder analysis is the most tangibly valuable, because it identifies key EGL actors, so interventions can catalyze and complement EGL￾improvement efforts. RESULT 2. The dissemination activities have reinforced other efforts to make EGL more central on the agendas of the MOEs and for civil society. Furthermore, the LRCP is seen as a context in which to help test different approaches to literacy development, use evidence to inform the discussion, and find common ground regarding the most effective approach. RESULT 3. Of the LRCP interventions, capacity strengthening has had the most visible effect on beneficiaries across the countries. Achievements in this regard have reached beyond the capital cities into different regions of the countries. Training events that built upon one another and had practical applications were the most useful for participants. RESULT 4. Foundational work for the sustainability of the LRCP interventions has taken place, such as the establishment of RedLEI, LRCP/RedLEI’s strategic alliances, EGL research by RedLEI and research fellows, the design of the EGL master’s course, and consolidated EGL resources. Across the countries, LRCP has strengthened capacity of a critical mass of EGL stakeholders and generated momentum for evidence use and EGL improvement. The master’s program in EGL is an opportunity for RedLEI to work together, generate funds, and strengthen EGL capacity in the LAC region. 1.C: What have been the greatest challenges or setbacks to achieving each result, by country, to date, and for what reasons? RESULT 1. There has been some demand for and use of evidence-based EGL information. However, this remains relatively low in the LAC region. The systematic review set a precedent in the region for a rigorous analysis on available EGL research and gaps and informed the LRCP research agenda, but it was not accessible for most beneficiaries due to its high technical level. 43 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV RESULT 2. There was a lack of a cohesive communication strategy from the beginning to help guide partner organizations with unified messages and strategies and more widely disseminate LRCP compilations and products. RESULT 3. Training participants highlighted the lack of training events that build upon one another to allow for developing knowledge and skills; they noted a need for more comprehensive and systematic training and learning opportunities. RESULT 4. The greatest challenges RedLEI has faced are in assuring it can generate its own funds while focusing exclusively on EGL research, to sustain efforts for improving EGL and training young researchers to conduct EGL in Central America. 1.E: What have been the most valuable aspects of the LRCP for USAID Missions? The demand-driven nature of the LRCP has allowed Missions to use it for their specific needs (even if not particular to the LRCP) and also benefit from specific LRCP activities. Capacity-strengthening activities, especially with MOEs, have helped to bolster the impact of the Missions’ bilateral projects and contributed to EGL interventions becoming more sustainable, although this has happened to different degrees and in different ways across the LRCP countries. EVALUATION QUESTION 2 What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy￾level interventions? 2.A: What are the strengths and weaknesses of the technical approach as structured? What alternative modalities might be considered for a program with similar or related goals? The LRCP’s organizational structure of a prime, sub-prime, and multiple partner organizations has worked well, which is likely due to the prime and sub-prime organizations having had previous working relationships and experience in the countries in which they are responsible for oversight for the LRCP. 2.B: What are the strengths and weaknesses of having results with flexible indicators not tied to performance indicators (such as improved literacy levels)? The flexible indicators for the LRCP, most of which are input and output indicators, are proxies for reporting planned activities and they capture the effects of these activities to some degree. However, some could be improved to better capture effects. With rigorous adherence to M&E, a few of them could be better proxies to indicate change toward the expected results. 2.C: What are the strengths and weaknesses of this regional literacy programming that focuses at the policy or systems levels? Given the stated objectives, the scope of activities, and investment levels, are there alternative approaches, models, or activities to supporting increased early literacy outcomes that USAID should explore? A strength of the LRCP’s regional policy- and systems-level approach is that it builds capacity of key stakeholders responsible for implementing EGL and/or creating the conditions for effective EGL, such as USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 44 the MOEs, teacher-training colleges, universities, and international and national NGOs. Additionally, it allows for sharing resources and knowledge across countries. However, the emphasis on outputs, such as disseminating information and offering training, makes it difficult to identify program outcomes/impact. 2.D: What are the strengths and weaknesses vis-à-vis USAID Mission benefits and engagement? Coordination between the Missions and the LRCP has varied across the countries. The close coordination in Guatemala has likely been facilitated by the implementer, Juárez and Associates, being the same for the bilateral project and the LRCP and the close working relationships among Juárez, the MOE, and USAID/Guatemala. In Nicaragua, there has also been close coordination among USAID/Nicaragua, the bilateral project, and the LRCP. In Honduras, there has not been coordination, likely due to the bilateral projects being a higher priority. RECOMMENDATIONS The following recommendations reflect both those provided by KII/FGD informants and those that the evaluation team identified based on its analysis of findings. EVALUATION QUESTION 1 How effective has the LRCP been in achieving its goals by result and by country to date? RESOURCES AND LEVELS OF EFFORT. For the remainder of the LRCP, focus resources on maximizing the impact from the investment in capacity strengthening by carrying out training activities targeted to those who have attended previous events that help them to develop concrete EGL skills, such as training and coaching teachers, elaborating EGL research designs, and conducting action research. RESULT 1 (A). Transfer the database to be housed by RedLEI in coordination with RedLEI universities and task the universities to do regular searches and reviews of new EGL research and resources. RESULT 1 (B). RedLEI should conduct annual or biannual updates of the systematic review to keep track of which evidence gaps are being filled and to what extent. The LRCP implementing partners should involve RedLEI in the first update, to ensure that there is capacity to do this after LRCP ends. Reviews should be made available in Spanish with a shortened version for a more widespread audience available in both English and Spanish. RESULT 2 (A). LRCP should hold events in each country with stakeholders who are researchers to discuss the findings of the updated systematic review in a way that is understandable and applicable, such as defining a research agenda and training needs. RESULT 2 (B). Continue dissemination activities, using the stakeholder analyses to ensure a targeted approach. RedLEI should work closely with partner organizations so they can assume this role after the end of the LRCP. RESULT 2 (C). The dissemination strategy should be made more interactive by sending out notifications about newly available resources in the database and for showcasing available resources according to the interests/specialization of the targeted audience. Provide summaries of studies and other resources that are relevant for different EGL stakeholders. 45 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV RESULT 2 (D). For future regional initiatives, ensure that there is a cohesive communication strategy across the countries and partner organizations that provides uniform guidance yet allows for country￾specific differences. Consider having the communication specialist located in the region to work closely in country with partner organizations. RESULT 3 (A). USAID and implementers should learn from the successful aspects of the capacity-building component of the LRCP to design future initiatives, such as targeting MOE and bilateral project personnel and conducting training events with continuity and practical applications, such as researching EGL practices in classrooms and analyzing them in light of evidence on effective EGL instruction. RESULT 4 (1). Before LRCP ends, hold an event in each country to present and discuss evidence on EGL methodological approaches for a discussion relevant to each country, e.g., bilingual EGL in Guatemala, Jamaica, and Nicaragua and phonics versus a communicative approach in the Dominican Republic and Honduras. Define a research agenda as a result of the events and a platform for sharing contextualized evidence. Give the MOE a leading role in this event to generate ownership. RESULT 4 (B). RedLEI should follow the recommendations provided in the sustainability assessment to ensure that it can generate its own resources and sustain the effects of the LRCP’s interventions. Examples of this are adopting a business development approach, carrying out evaluations and other services that generate income, and adopting a broader mission (such as “literacy for lifelong learning”) to expand beyond EGL in grades one through three. RESULT 4 (C). RedLEI should assess the demand for the EGL master’s course by consulting with and/or surveying stakeholders and base a marketing plan on demand. Additionally, RedLEI should initiate discussions with MOEs, starting with Guatemala, to seek their support in providing scholarships for interested technical staff who could most benefit from participating in the master’s program (as has been the case in El Salvador for UCA’s master’s in education policy). RESULT 4 (D). To improve EGL using evidence and seek ways to continue strengthening their capacity through training and networks, the LRCP implementers and partner organizations should prioritize identifying stakeholders that form part of the critical mass in each country and regionally that are working. This effort should be linked with RedLEI to the degree possible to ensure sustainability. EVALUATION QUESTION 2 What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy￾level interventions? LRCP TECHNICAL APPROACH. Future similar initiatives aimed at bolstering the effects of bilateral projects should be structured in a consistent way across countries in terms of mission-demand-driven flexibility, such as regional sharing of technical expertise, resources, and practices in specific content areas. ALTERNATIVE MODALITIES. For future similar initiatives, implementing partners and partner organizations could develop a regional capacity-building program with online courses and in-person workshops for selected MOE mid-level technical personnel and other key EGL stakeholders, such as NGOs implementing EGL-improvement initiatives; these could include applied course content such as USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 46 planning and conducting teacher-training workshops, designing an EGL study, and/or carrying out action research. FLEXIBLE INDICATORS. The LRCP implementing partners should come up with a rigorous and systematic way to measure capacity building and evidence utilization for the remainder of the LRCP. To assess capacity strengthening, LRCP should continue to use the pre- and post-surveys, but also develop tools to capture specific capacities acquired as well as their practical applications. To assess practical applications, follow-up is needed at least two months after the training, and the indicator “percent/number of people applying knowledge/skills from training” could be added. Additionally, consider adjusting/expanding the indicator to report out on EGL laws or policies that affect something directly attributable to LRCP actions, such influencing government education plans. FOCUS ON POLICY SYSTEMS. For future initiatives, USAID should consider a modality for translating evidence into practice by providing technical support to the MOE and other relevant stakeholders to design and conduct research to inform their most pertinent EGL policy and programming decisions. This could be designed as a regional program with a platform in each country to identify what studies should be done to address an EGL implementation issue (e.g., test different approaches to bilingual multi-grade EGL instruction to identify the most effective one). Technical assistance would then be provided to conduct the research. Research designs and studies would be shared through the regional platform. MISSION ENGAGEMENT. LAC/RSD/EDU should convene a virtual meeting with the USAID/Honduras education team to discuss how LRCP can best complement and bolster the bilateral projects’ impacts. The USAID/Honduras education team and implementers should hold monthly meetings with this aim. 47 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX 1. EVALUATION STATEMENT OF WORK 1. Background Since 2011, and in line with the USAID Global Education Strategy, a key aspect of LAC’s efforts to strengthen the regional policy dialogue has been focused on improved reading skills for children in the primary grades. To this end, LAC/RSD designed the regional LAC Reads Project, a suite of programs and mechanisms with partners that work collaboratively to increase the availability and understanding of evidence for improving early-grade reading (EGR) and boosting the capacity of key stakeholders to implement evidence-based, cost-effective practices in EGR. The LAC Reads Project was originally conceptualized into three components (See Table A-1, below). TABLE A-1. CONCEPTUAL MODEL OF OVERALL LAC READS PROJECT Component Description Mechanisms Implementer(s) Dates 1 LAC Reads: Promising Interventions Regional and Bilateral G1 Programs in Peru, Guatemala, Honduras, Nicaragua (Multiple) (Multiple) 2012 - present 2 LAC Reads: Evidence Infusion of rigorous cost and impact evaluation information for improved evidence-base for decision-making on key literacy issues in LAC LAC Reads Evaluation Contract Mathematica 2012 - 2019 3 LAC Reads: Capacity Assistance with adoption of evidence-based practices and strengthening of capacity of key institutions to continue to improve reading outcomes LAC Reads Capacity Program (LRCP) AIR, Juárez, and national partners 2014 - 2019 The LAC Reads Capacity Program (LRCP), the mechanism developed to facilitate the goals of the third component of the overall LAC Reads Project, was designed to assist both USAID bilateral efforts across the LAC region and to complement other regional efforts at improving the evidence base for decision making related to literacy improvement in the LAC region and to build the capacity of key stakeholders institutions to utilize evidence in their decision making. A relevant aspect to the background of this evaluation of the LRCP is USAID’s past investments in similar regional programs. For the past two decades, the overarching objective of the USAID LAC/RSD Education portfolio has been to improve the quality and equity of access to education and training in the region. One key component of this portfolio has focused on improving the quality and relevance of policy dialogue around education reform, both as a means of strengthening regional capacity to implement reforms, as well as a way to build political support for improving educational quality through the use of research and evidence. The Partnership for Educational Revitalization in the Americas (PREAL), funded from 1996 through 2013 by LAC/RSD, worked to create more informed policy dialogue by reaching out to both government and non-governmental actors across the region. The two external evaluations of PREAL (see Annex) showed its key strengths to be in its reputation as a trusted source of education analysis for decision-makers in the region, and its ability to convene key actors around important education reform USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 48 topics. Some of its weaknesses were described as its limited influence over how education reform is operationalized, and the lack of a regional sustainability plan. 2. Purpose The purpose of this performance evaluation is to twofold: first, this evaluation will inform midcourse adjustments in program implementation for the remainder of the LRCP, which currently runs through September 2019 and any possible extensions. Second, this performance evaluation will inform ongoing and future regional and bilateral USAID programming that addresses similar subject matter and key stakeholders. To this end, the primary audiences for this evaluation are USAID Education experts and agency implementing partners. The evaluation findings may also be of interest to regional education stakeholders. 3. LAC Reads Capacity Program (LRCP) Program Overview The LRCP itself (Component 3 in the above table) was designed with four (4) principal results to be achieved. These 4 results of the LRCP are laid out (Table A-2) below. TABLE A-2. FOUR RESULTS OF THE LRCP Result Description Key Activities 1 Evidence on EGR practices and outcomes is collected, consolidated, and systematized for practical use by stakeholders in the LAC region. Systematic review of evidence; creation of evidence database 2 Dissemination of up-to-date knowledge about EGR practices is targeted to diverse audiences and stakeholders. Stakeholder engagement (mapping and interviews to assess state of EGR in each country); website, marketing, conferences and events, publications 3 Capacity is strengthened at institutions so they can more widely implement proven approaches to improve EGR outcomes for poor and disadvantaged children Customized technical assistance provided to key stakeholders in- country based on demand and assessments 4 Sustainability platforms are in place to continue and strengthen EGR interventions in countries in the LAC region. Creation of regional research platform or network The level of effort devoted to these results was designed to be sequenced in such a way so that certain key aspects of successful completion of each result could be utilized for the following result. Successful completion of Result 1 would allow for the dissemination efforts identified under Result 2, while both the analyses and outreach efforts done under Results 1 and 2 would enable appropriate, targeted and customized technical assistance contemplated under Result 3. Finally, key aspects of all three results would be leveraged for and transferred to the regional research platform or network at the heart of Result Structure and Approach of the LRCP The American Institutes for Research (AIR) is the prime implementing organization and works in partnership with major subcontractor Juárez and Associates (J&A) and country level partners in the 49 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV implementation of the project. The project has a management and technical team at AIR and J&A, and a regional team with a Regional Coordinator based in Guatemala and a Regional Literacy Specialist based in Nicaragua. Implementation in the priority countries is shared by AIR and J&A, based on past history of work and experience in these locations. J&A has primary responsibility for implementation in Jamaica, Guatemala, Dominican Republic, and the Eastern Caribbean. AIR has primary responsibility for implementation in Haiti, Honduras, Nicaragua, and Peru. Country assignments were intentionally made based on each organization’s history in the region. Both partners provide technical assistance under all program results. Much of the LRCP work at the country level is done though the project’s national partners, in order to strengthen sustainability, ownership, and impact. Some national partners were defined in the original program proposal to USAID and included in the cooperative agreement once the project had been awarded. These organizations were chosen primarily because of their history and track record in education policy and practice in their respective countries – many of them having been that national partner organization of PREAL. Additional factors taken into consideration included EGL work, experience working with USAID projects, including regional education project experience, and a history of work with AIR and/or J&A. In two countries (Guatemala and Haiti) where no national partner was identified in the proposal, a competitive process to select a partner was defined and carried out post-award. In the case of the Eastern Caribbean, the LRCP works with the USAID/Early Learners (ELP) project. In Peru, there is no LRCP implementing partner; specific activities are carried out in coordination with the Mission. In Haiti, the scope of work was revised and expanded via a 2016 modification to the LRCP sub-award and via a pending subsequent modification. The LRCP work plan in Haiti is now focused entirely on Result 3 (assistance for capacity building) work and does not closely resemble the scope or scale of efforts in the other LRCP countries. In Haiti the project is managed through a dedicated project office in Port au Prince with a COP and technical and management staff. Work under LRCP Result 1, led by AIR, started shortly after award of the project. Work under the other results has been carried out in a phased approach that was informed by discussion and engagement with the USAID LAC Bureau AOR and team and with each of the USAID Mission education teams in the project’s priority countries, as well as with project staff, partners, and other stakeholders. Additionally, work under Results 2, 3, and 4 has been informed and shaped by the findings of the Result 1 systematic review as well as by the results of the stakeholder mapping and analysis process carried out by project partners under the technical leadership of the LRCP team. As of 2017, the bulk of effort planned for Result 4 was channeled into an MOU with the Universidad del Valle de Guatemala (UVG), which agreed to house what has since become named the Central American Network for Early Literacy (in Spanish, the Red de Lectoescritura Inicial, or RedLEI). A cooperative agreement was then signed with UVG through September 2019, which included program and personnel funds for launching RedLEI and plans for sustaining it past the period of performance of the LRCP. This effort to feed LRCP resources (both financial and technical) into RedLEI and help make it effective and sustainable became the central effort of Result 4 and a major priority for USAID/LAC. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 50 4. Evaluation Focus and Key Questions Bearing in mind the twofold purpose of this evaluation and its intended audience, per Section 2 of this document, the evaluation questions should drive the evaluation design and report. Evaluation Question 1: How effective has the LRCP been in achieving its goals, by Result, and by country, to date? In answering this question, the following probes (at a minimum) should be considered: ● Where have the most resources and levels of effort gone, by result, by country, and to date? ● What have been the most successful activities under each result, by country, to date, and for what reasons? ● What have been the greatest challenges or setbacks to achieving each result, by country, to date, and for what reasons? ● What evidence is there that the capacities of local partner institutions and key stakeholder institutions have increased capacity to improve early literacy in their home countries? ● What have been the most valuable aspects of the LRCP for USAID Missions? Evaluation Question 2: What are the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy-level interventions? In answering this question, the following probes (at a minimum) should be considered: ● What are the strengths and weaknesses of the technical approach as structured (featuring a principal prime and principal sub-prime, as well as multiple national-level partners)? What alternative modalities might be considered for a program with similar or related goals? ● What are the strengths and weaknesses of having results with flexible indicators not tied to performance indicators (such as improved literacy levels)? ● What are the strengths and weaknesses of this regional literacy programming that focuses at the policy or systems levels? Given the stated objectives, the scope of activities, and investment levels, are there alternative approaches, models, or activities to supporting increased early literacy outcomes that USAID should explore? ● What are the strengths and weaknesses of the program vis-à-vis USAID Mission benefits and engagement? 5. General Evaluation Parameters Planning and implementation of the evaluation study will be closely coordinated with LAC/RSD, USAID/field Missions, and the implementing partner and its partner institutions in the region. It is of particular importance that the right evaluation team be selected. Selection of the evaluation team should be done in close collaboration with USAID, and according to the following minimum standards (see Table 3, Illustrative Evaluation Team and LOE, below). 51 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ● The evaluation team should consist of a Team Leader and a small number of Research Assistants (up to, but no more than 4). The evaluation team may make use of short-term, task-related consultants as needed and appropriate, given the finalized scope of work and budget constraints. ● The evaluation team must have an appropriate mix of technical skills to conduct the evaluation. Recognized experts with appropriate academic credentials should have some combination of the following skill sets, in order of priority: (i) deep knowledge of the LAC region (particularly Central America), (ii) strong knowledge of qualitative research methodologies in the social sciences, (iii) strong knowledge of early-literacy work and research, and (iv) adequate knowledge of a combination of policy, systems, or institutional capacity interventions in education ● The Evaluation Team leader must have: ▪ Full English and Spanish language fluency. ▪ Outstanding written and verbal communication skills in English. ▪ Superior understanding of and prior experience with education policy work (education and/or other social sectors) in the LAC region ▪ Demonstrated success in program evaluation and report preparation ▪ Outstanding research skills and ability to synthesize large amounts of disparate information into clear, succinct, and readable prose. ● In rounding out the evaluation team, the wealth of local expertise that exists in Central America in the areas of education policy evaluation and research should be fully explored. The Evaluation team should make appropriate use of local expertise for analysis as well as data collection. 6. Services, Deliverables, Performance Requirements and Standards Objective: Evaluation/Assessment Addressing Key Questions in Manner of Utility to USAID Completed Requirement 1.1: Develop an evaluation plan, including data collection and analysis plan and instruments in English Standards: ● Draft evaluation plan provided within 4 weeks of start of period of performance ● Draft evaluation plan includes data collection methodology, an analysis plan, and instruments to address the key questions listed in section III. ● Draft evaluation plan identifies data sources that will be used for each question, including sources of data that are already available, such as monitoring reports and prior evaluations. ● Final evaluation plan and instruments are informed by discussion with USAID. Data collection instruments, including any survey, its questions and recipient list, should be designed in close collaboration with USAID. ● Final evaluation plan is informed by and in accordance with ADS 201 and USAID Evaluation Policy. Requirement 1.2: Collect and analyze relevant evaluation data Standards: ● Report complies with USAID Evaluation Policy and ADS 201 guidelines for evaluation reports (see Annex for both). USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 52 ● Site visits to at least two countries, with remote data collection as a possibility in countries or localities that are not easily accessible. Target countries for data collection may include Guatemala, Honduras, and Nicaragua (additions or changes to this list may be discussed in advance with USAID). ● Key informants identified based on knowledge of the sector and a USAID/ LRCP-provided list of key informants per country. ● Number of individuals surveyed or interviewed and/or focus groups held is sufficient (if not statistically significant) to provide methodologically sound and meaningful representation, findings, and conclusions. ● Secondary data (relevant statistics / indicators) should be collected as it relates the relevant LRCP work to be evaluated. ● All written documentation listed in key references section (VIII) should be carefully and systematically reviewed in advance of field work, and approved by USAID before field work is begun All relevant findings or data points from that literature should be reflected in the background section of the evaluation. ● Any “conclusions” developed are transparently and demonstrably based on analysis of findings. ● Any “recommendations” are based on conclusions from analysis of findings. Requirement 1.3: Produce evaluation report and related deliverables (in English) Standards: ● USAID input incorporated to report outline. ● Draft report submitted within 30 days of completion of analysis. ● Final report incorporates USAID input. ● Report includes executive summary, which summarizes significant points from the full report, including key findings and recommendations. Any information provided in the executive summary appears in the full report with further detail. ● Report includes a section providing key definitions and acronyms ● Report is compliant with USAID branding and marking standards (as laid out in ADS 320) ● Report includes a background section which clearly and succinctly explains at least the following: the purpose of the evaluation, the program and its components being evaluated, the background literature and other learning that is relevant to the evaluation. ● Report includes a methodology section which clearly and succinctly explains and justifies the research methods used for data collection and analysis, taking into account methodological limitations. ● Report clearly distinguishes findings (data and facts), conclusions (based on analysis of findings), and recommendations (based on analysis of conclusions). ● The logical connections between findings, conclusions, and recommendations are clear to the reader. Each conclusion is can be traced to one or more specific findings, and each recommendation clearly emanates from one or more conclusions. ● For any survey or interview data, the report includes an annex presenting any structured or semi￾structured interview protocols, a detailed and organized summary of findings from the survey, including summary statistics and an overview of respondents. Requirement 1.4: Present findings of analysis to USAID (in English) Standards: 53 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ● Presentation includes all key findings to LAC/RSD, relevant USAID Missions, and program implementers. ● Presentation includes a PowerPoint that summarizes findings. ● All relevant materials and presentations sent to USAID in advance of the day of presentation. 7. Timeline TABLE A-3. TIMELINE Date Action / Deliverable Comments September 15, 2018 Evaluation team selected Resonance will provide to USAID for confirmation September 17, 2018 Key references and reports delivered to evaluation team for review USAID will provide to evaluation team September 18, 2018 Kick-off planning meeting(s) with Resonance and Evaluation Team (at USAID) Meeting should be held at USAID with core evaluation team, USAID, and Resonance October 1, 2018 Kick-off planning meeting(s) with Evaluation Team and Implementing Partner (at USAID) Meeting should be held at USAID with core evaluation team, USAID, and Resonance and AIR/Juárez October 29, 2018 Draft Evaluation Plan for review by USAID, including travel plan, and data collection plan and instruments USAID will provide feedback within 2 weeks. February 8, 2018 Draft Evaluation Report for review by USAID USAID will review in anticipation of briefing December 15, 2018 Draft evaluation report for review by USAID USAID will review in anticipation of briefing February 20, 2018 Briefing on findings to USAID and AIR/Juárez USAID will provide feedback within 2 weeks March 15, 2018 Final report delivered to USAID USAID will provide feedback and/or final approval within 2 weeks USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 54 ANNEX ADDENDUM: KEY REFERENCES AND READING FOR EVALUATORS The following documents are mandatory background reading for the evaluation team and will be provided upon selection of the evaluation team and prior to initial planning meetings to discuss the evaluation design with USAID: 1. Education Reform Support Today (DeStefano and Crouch, 2006) 2. 2011 PREAL Evaluation Report (USAID) 3. LAC Regional Education Policy Evaluation Report (2013) 4. LRCP Cooperative Agreement, including RFA and Program Description (USAID) 5. LRCP Monitoring and Evaluation (M&E) Plan (AIR) 6. LRCP Quarterly Reports (AIR) 7. Country profiles and stakeholder analyses (unpublished - AIR) 8. Summary documentation on creation of RedLEI (AIR. Juárez , and UVG) 9. Systematic Review of Early Literacy Research in Latin America (AIR) 10. All content on www.lacreads.org 11. USAID Evaluation Policy (2016) 12. ADS 201: Program Cycle Operational Policy (related to USAID Evaluations) 13. USAID Evaluation Toolkit 14. USAID Global Education Strategy (2011) 55 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX 2. EVALUATION TEAM COMPOSITION DR. KRISTIN ROSEKRANS: EVALUATION TEAM LEADER Dr. Rosekrans has over 20 years working in education internationally. She works with DevTech Systems intermittently and as an independent consultant. She specializes in evaluation, research, and policy analysis specifically regarding early grade literacy, teacher education, and multilingual education. Her experience includes the design, implementation, monitoring, and evaluation of large-scale government programs, as well as pilot programs and non-governmental initiatives in Central America, West Africa, and the United States. She led the education team in USAID/El Salvador from 2003–2007 prior to directing a program to support the Ghanaian government in implementing a national multilingual education program. She has a master’s degree from Harvard in International Education Policy and a Ph.D. from the University of California, Berkeley, in Policy, Organization, Measurement, and Evaluation. Dr. Rosekrans was responsible for the management and coordination of the overall evaluation and its deliverables. She developed the evaluation design and data collection tools; provided direction to the evaluation team members throughout the evaluation; conducted virtual and in-country interviews and focus groups; administered the questionnaire and online survey; analyzed all data and developed overall and country-specific findings, conclusions, and recommendations; wrote the draft and final evaluation reports; and presented findings and recommendations to USAID and the implementing partners. DANIELA CARDOZO: RESEARCH ASSISTANT Ms. Cardozo has five years of experience working in international development. She currently works with Resonance contributing to the design, monitoring, implementation, and evaluation of a portfolio of international development programs from the home office and during short-term assignments in Latin America and the Caribbean (LAC). She has a Master’s degree in Political Economy of Late Development at the London School of Economics and Political Science, as well as a Bachelor’s degree in Political Science from the University of Florida. Ms. Cardozo provided research and logistical support for the LRCP evaluation. She assisted with the development of data collection tools; took notes for all interviews and focus groups; assisted in administering the questionnaire and online survey; helped to organize data during and after field work; contributed to the analysis and synthesis of the data obtained during field work and secondary analysis to develop preliminary findings; assisted in identifying key findings, conclusions, and recommendations to include in the final evaluation report; and contributed to the final evaluation report. DR. MARIO ALAS: EDUCATION SPECIALIST, HONDURAS Dr. Solis has more than 25 years of experience in the education sector. He is currently the Coordinator of the University Observatory of National and Regional Education of the Universidad Pedagógica Nacional Francisco Morazán. He has also participated in the following evaluations: “Standardized Assessment and Teaching Practices” (2015), “Factors Associated with Academic Performance” (2014), “Effective Educational Practices in Disadvantaged Social Contexts” (2012), “Factors and Strategies Associated with School Performance” (2011), and “Best Educational Practices: Effective Schools in Honduras”. He has a Ph.D. in Education from Flensburg University in Germany and a master’s degree in Sociology from the University of Costa Rica. He provided input on all data collection tools; helped to schedule meetings and interviews in Honduras; participated in the in-country interviews and focus groups; contributed to the analysis and synthesis of the data obtained during field work and secondary analysis to develop preliminary findings for evaluation; and assisted in identifying key findings, conclusions, and recommendations to include in the final performance evaluation report. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 56 DR. ANA ROXANDA RODRIGUEZ CABRERA: EDUCATION SPECIALIST, GUATEMALA Dr. Rodriguez has more than 25 years of experience in the education sector. She is currently the Executive Director of Foundation for Education and Social Development (FUDESA) and a professor at different universities in Guatemala. She has worked for more than 20 years as a consultant in different projects with the Ministry of Education General Directorate for Educational Quality Management (DIGECADE) and the World Bank. She has a master’s degree in Education from the University of Costa Rica and a Ph.D. in Political Science and Sociology from the University of Salamanca in Spain. Dr. Rodriguez provided input on all data collection tools; helped to schedule meetings and interviews in Guatemala; participated in the in-country interviews and focus groups; contributed to the analysis and synthesis of the data obtained during field work and secondary analysis to develop preliminary findings for evaluation; and assisted in identifying key findings, conclusions, and recommendations to include in the final performance evaluation report. LIC. CELSO LUIS ASENSIO FLÓREZ: EDUCATION SPECIALIST, NICARAGUA Mr. Asensio has more than 20 years in the development of professional skills in social programs. Of these, he has more than 15 years in education-related programs for the Ministry of Education (1980–1992 as chief or adviser, 2002–2005 as Educational Reform Consultant MINED/IDB, and 2007–2009 as Senior Technical Adviser for School Counseling UNFPA/MINED). He has been a Consultant or Officer in educational programs and projects supported by international organizations including UNICEF, UNFPA, IDB, the World Bank, USAID, and the European Union. He has a master’s degree in Public Economy and Development from the University of Barcelona in Spain. Mr. Asensio provided input on all data collection tools; scheduled meetings and interviews in Nicaragua; conducted all in-country interviews and focus groups; contributed to the analysis and synthesis of the data obtained during field work and secondary analysis to develop preliminary findings that will then be used to develop a final report; and assisted in identifying key findings, conclusions, and recommendations to include in the final performance evaluation report. 57 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX 3. DATA COLLECTION INSTRUMENTS Questionnaire for implementing partners and organizations Country of residence:_______________ Organization:_______________ Position:____________ The objective of this survey is to know your opinion and experience with the activities carried out through the LAC Reads Capacity Program. The information collected through this survey will be analyzed at a country and regional level and will be used only to contribute to the program evaluation analysis. 1. The project in your country was designed to be carried out at what level: (please mark all options that apply) a. __Central (authorities at the central level from the Ministry of Education) b. __Intermediate (mid-level specialist from the Ministry of Education) c. __Education community at the local level/schools (principals, teachers and students) d. __With the participation of private organizations e. __Another strategy was designed. Please specify:___________ 2. Which of the following points do you consider to be the most important aspects in order to improve literacy in your country? (Please order the following options from 1-7, where "1" is considered the highest priority option and "7" the least) a. ____ Teacher trainers with better skills and knowledge b. ____ Better teacher preparation programs c. ____ In-service teacher training with practical orientations (e.g. teaching strategies) d. ____ More access to evidence on effective strategies to improve early literacy (research, effective practices) e. ____ Changes in policies and laws focused on improving early grade literacy f. ____ Consensus on the most effective early literacy teaching methods g. ____ Other:________________________ 3. On a scale of 1-5, where 1 is considered a “low contribution” and 5 a “high contribution”, indicate for each of the following points to what extent you consider that the program has contributed to the following achievements to date: a. ___ Higher technical capacity of teacher trainers (more knowledge and skills) b. ___ Higher technical capacity of key stakeholders (more knowledge and skills) c. ___ Better teacher preparation programs d. ___ More in-service training for teachers to strengthen their knowledge and practices e. ___ Greater access to evidence on effective strategies to improve early literacy (research, best practices) f. ___ More use of evidence on effective strategies to improve early literacy (research, best practices) USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 58 g. ___ Changes in policies and laws focused on improving early literacy h. ___ Place early literacy on the public agenda i. ___ Create a more unified public agenda regarding early literacy j. ___ None – the program has not contributed to any of the above k. ___ Other:_____________________ 4. On a scale of 1-5, where 1 is considered a “low extent” and 5 a “high extent” to what extent has it been possible to establish strategic alliances with the following actors: a. ___ Ministry of Education b. ___ International NGOs c. ___ Local NGOs d. ___ Private Sector (Specify:__________________) e. ___ Other _________________________ 5. On a scale of 1-5, where 1 is considered a “low extent” and 5 a “high extent” to what extent has the following been achieved? a. ___ Disseminate information to key stakeholders b. ___ Encourage information to be used to implement best practices c. ___ Have continuity in the dissemination of information after workshops/events d. ___ Have continuity in capacity building e. ___ Establish sustainability strategies 6. In your opinion, what has been the most significant contribution that the LRCP has made to strengthen early literacy in your country? _____________ Why? ____________ 7. What LRCP intervention has been the least effective in strengthening the capacity to improve early literacy in your country? _________________ Why? ___________________ 8. In your opinion, what has been the most important achievement of the LRCP to date? ______ 9. Do you consider that the Ministry of Education has greater capacity to improve the teaching and learning of early literacy in your country, as a result of the program? Yes_______ No ______ If you answered “Yes”, explain how:________________ If you answered “No”, explain why:_________________ 10. Has your organization counted on the financial resources needed to achieve the objectives of the LRCP? Yes_______ No ______ 59 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Explain:___________________ 11. Has your organization counted on the necessary human resources to achieve the objectives of the LRCP? Yes_______ No ______ Explain:____________________ Cuestionario para organizaciones implementadores y socios País donde reside:______________ Organización:__________ Puesto/cargo de trabajo:___________ El objetivo de esta encuesta es conocer su opinión y experiencia con las actividades realizadas a través del Programa de Capacidades LAC Reads. La información recolectada a través de esta encuesta será compilada y analizada por país y a nivel regional. La información será utilizada únicamente para contribuir al análisis de la evaluación del programa. 1. El proyecto en su país fue diseñado para desarrollarse a nivel: (favor marcar todas las opciones que aplican) a. ___ Central (autoridades del nivel central del Ministerio de Educación) b. ___ Intermedio (personal técnico del nivel intermedio del Ministerio de Educación) c. ___ Local comunidad educativa / escuelas (directores, maestros y estudiantes) d. ___ Con la participación de organizaciones privadas e. ___ Se diseño otra estrategia. Por favor, explicar:__________________ 2. ¿Cuál de los siguientes puntos considera es la necesidad más imperiosa para mejorar el aprendizaje de la lectoescritura en su país? (Por favor, ordenar las siguientes opciones de 1- 7: en donde “1” se considera la opción más imperiosa y “7” la menos) h. ____ Formadores de docentes con mayor capacidad técnica i. ____ Mejores programas para la formación inicial de docentes j. ____ Processos de formación docente en servicio con aplicaciones prácticas k. ____ Mayor acceso a evidencias sobre estrategias efectivas para mejorar el aprendizaje de lectoescritura inicial (investigaciones, prácticas exitosas) l. ____ Cambios en política y leyes enfocadas en mejorar el aprendizaje de lectoescritura inicial m. ____ Consenso sobre los mejores métodos de enseñanza-aprendizaje de lecto-escritura inicial n. ____ Otro:_____________________________ USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 60 3. En una escala de 1-5, en donde el 1 se considera “baja contribución” y el 5 “alta contribución”, indique para cada una de las sigueintes puntos en qué medida considera que el Programa ha contribuido a los siguientes logros hasta la fecha: l. ___ Mayor capacidad técnica de formadores de docentes m. ___ Mayor capacidad técnica de actores clave n. ___ Mejores programas para formación inicial docentes o. ___ Más formación en servicio a docentes para fortalecer su conocimiento y prácticas p. ___ Mayor acceso a evidencia (resultados de investigaciones, prácticas exitosas) sobre estrategias efectivas para mejorar el aprendizaje de lectoescritura inicial q. ___ Mayor uso de evidencia (investigaciones, información, prácticas exitosas) sobre estrategias efectivas para mejorar el aprendizaje de lectoescritura inicial r. ___ Cambios en políticas y leyes enfocadas a mejorar el aprendizaje de lectoescritura inicial s. ___ Poner/elevar el tema de lectoescritura inicial en la agenda pública t. ___ Crear una agenda pública mas unificada en cuanto a la lectoescritura inicial u. ___ Ninguno – el Programa no ha contribuido a los logros arriba mencionados v. ___ Otro:_____________________ 4. En una escala de 1-5, en donde el 1 se considera “baja medida” y el 5 “alta medida”¿En qué medida se ha logrado establecer alianzas estratégicas con los siguientes actores: a. ___ Ministerio de Educación f. ___ Organizaciones no gubernamentales internacionales g. ___ Organizaciones no gubernamentales locales h. ___ Sector privado (Especifique:___________________) i. ___ Otro _________________________ 5. En una escala de 1-5, en donde el 1 se considera “baja medida” y el 5 “alta medida”¿En qué medida se ha logrado: a. ___ Difundir/diseminar la información a actores clave f. ___ Estimular que la información sea utilizada para implementar mejores prácticas g. ___ Tener continuidad en la difusión/diseminación de información despues de talleres/eventos h. ___ Tener continuidad en la formación de capacidades i. ___ Concretar estratégias de sostenibilidad 6. ¿En su opinión, cuál ha sido el aporte más significativo que el LRCP ha brindado para fortalecer la lectoescritura inicial en su país? _________ ¿Por qué?________ 12. ¿Qué intervención del LRCP ha sido la menos eficaz para fortalecer la capacidad de mejorar la enseñanza-aprendizaje de lectoescritura en su país? _________ 61 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ¿Por qué?_________ 13. ¿En su opinión, cuál ha sido el logro mas importante del LRCP hasta el momento?_______ 14. ¿Considera que el Ministerio de Educación en su país cuenta con mayor capacidad gracias al Programa, para mejorar la enseñanza-aprendizaje de lectoescritura inicial en su país? Si _______ No ______ Si respondió “Sí”, explique como:___________ Si respondió “no”, explique por qué:______________ 10. ¿Ha contado su organización con los recursos financieros necesarios para lograr los objetivos del LRCP? Si _______ No ______ Explique:__________ 11. ¿Ha contado su organización con los recursos humanos necesarios para lograr los objetivos del LRCP? Si _______ No ______ Explique:________________ Interview Guide for USAID, Implementers (prime and subs), and external consultants Name and position/title of Interviewee:_____________________________ Date:_______________ Place:__________________ Explain the goals of the evaluation and ask if she/he has any questions. Request permission to record the interview. Goals of the Evaluation: ● Determine how effective the LRCP has been in achieving its goals, by result, and by country, to date. ● Identify the strengths and weaknesses of the LRCP approach to catalyzing reading improvement through policy-level interventions. 1. Tell me about your role and involvement with LRCP 2. Please share your thoughts regarding some of the overall strengths about the way the LRCP was designed and implemented. Also, specifically in terms of its: (Probe: Ask for examples for each point. Discuss the strengths of the design and implementation) a. Regional focus with in-country variation b. Organizational structure (prime and main partners, multiple local partners) USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 62 c. Approach of capacity building, policy and systems (rather than more direct interventions to improve literacy at school level) 3. Please share your thoughts regarding some of the weakness about the way the LRCP was designed and implemented, in terms of its: (Probe for each point: Why? What alternative would be better and why? Discuss the weaknesses of the design and implementation) a. Regional focus with in-country variation b. Organizational structure (prime and main partners, multiple local partners) c. Approach of capacity building, policy and systems (rather than more direct interventions to improve literacy at school level) 4. Do you think that the efforts and resources by country have been distributed in the best way? If not, what would you like to be different about it? 5. What are some of the LCRP biggest achievements/successes to date? Please give specific examples. Probe: Talk about each of the eight countries. 6. Have you seen evidence of increased capacity of local partners and stakeholder institutions to improve literacy in each of the countries? Please give specific examples. a. If not, why do you think that is? Design? Implementation? Other reason? b. What evidence were you expecting to see and/or would like to see? c. Are the indicators good proxies for measuring the achievement of this goal? 7. What are some of the challenges that LCRP has faced? Please give specific examples. a. Have they been overcome? If so, how? If not, why not? 8. What was the expectation of the Mission(s) for the LCRP? Do you think that these have been met? a. What has been the most valuable/beneficial for the Mission? Please give examples. b. Have there been unexpected/unplanned activities? If so, what have they been? Have they been beneficial? c. What has been the least beneficial/the biggest shortcoming? Why? Please give examples. 9. What do you think the biggest challenges are in terms of the LRCP interventions becoming sustainable? Please talk about each country. 10. What recommendations do you have for strengthening the LRCP? Do you have any other recommendations for the future of LRCP? 63 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Ask if there are any additional comments or questions. If not, thank her/him for the time. Guía de Entrevista para Organizaciones e Instituciones Locales (ONGs, Universidades, Ministerios de Educación) Nombre y titulo del informante:___________________________________________________ Fecha:_______________ Lugar:______________________ Explicar los objetivos de la evaluación y preguntar si hay preguntas. Pedir permiso para grabar la entrevista/grupo focal. Objetivos de la Evaluación: ● Determinar la efectividad del proyecto en materia del logro de sus objetivos, por resultado y por país, hasta la fecha. ● Identificar las fortalezas y debilidades del enfoque del proyecto en fomentar una mejora en el aprendizaje de la lectoescritura estudiantil mediante intervenciones de política pública. 1. Cuénteme sobre el proyecto. Según su entendimiento, ¿Cuál es el propósito del proyecto? 2. ¿Qué opina sobre la forma que el proyecto busca lograr su propósito? (A través de la creación de una red regional, el fortalecimiento institucional, y la generación y difusión de productos de conocimiento y estudios). a. ¿Piensa que algunas de las formas eran más eficaces que otras? ¿Cuáles? b. ¿Piensa que hay otras formas más eficaces para lograr el propósito? 3. Cuénteme sobre su participación y/o relación con el proyecto. a. ¿ Nacido de alguna necesidad en particular? ¿Cuál? (¿Quién decidió? ¿Con qué motivo?) 4. ¿De qué manera se ha beneficiado del proyecto, y/o su organización/institución? ¿En que intervención/actividad participó para obtener este beneficio? a. ¿Cuáles han sido los aportes mas valiosos o útiles (actividades, productos, etc.) en términos entendimiento, uso, e incorporación de métodos de aprendizaje de lectoescritura en términos y/o otros objetivos del proyecto ? b. ¿Cuáles han sido los aportes menos valiosos o útiles (actividades, productos, etc.)? 5. ¿Tiene sugerencias de otros aportes o recursos que el proyecto podría ofrecer para fortalecer su capacidad de mejorar la lectoescritura? USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 64 6. ¿ Ha utilizado algo producido por el proyecto (e.g. información, estudios, conocimiento)? Por favor dar ejemplos. 7. ¿Piensa usted que el proyecto está logrando tener un efecto en cuanto al fortalecimiento institucional para mejorar el aprendizaje de la lectoescritura? ¿Por favor dar ejemplos? a. ¿Cómo podría ser más efectivo el proyecto para lograr este objetivo? 8. ¿Piensa usted que el proyecto está logrando tener un efecto en cuanto a las políticas públicas y leyes para mejorar el aprendizaje de la lectoescritura? Por favor dar ejemplos. a. ¿Cómo podría ser más efectivo para lograr este objetivo? 8. ¿Cuáles otros efectos – planeados o no planeados - ha visto en cuanto al mejoramiento de la lectoescritura? a. ¿Más conversaciones o diálogos en medios públicos? ¿Ejemplos? b. ¿Mayor capacidad de organizaciones o entidades públicas para mejorar la enseñanza-aprendizaje de lectoescritura? ¿Ejemplos. c. Mas generación o difusión de información/evidencia sobre enseñanza-aprendizaje de lectoescritura? ¿Ejemplos? d. Mas/mejor uso de información para mejorar la enseñanza-aprendizaje de lectoescritura? ¿Ejemplos? e. Nuevos mecanismos para mejorar las practicas y/o conocimiento en cuanto a la enseñanza-aprendizaje de lectoescritura? ¿Ejemplos? 9. ¿Cómo ha sido la relación y comunicación con las organizaciones socias del proyecto? 10. ¿Han habido desafíos o áreas de oportunidad en cuanto el apoyo provisto para el proyecto? a. ¿Con la implementación de las actividades? ¿Ejemplos? b. ¿Con el uso y/o diseminación de información? ¿Ejemplos? c. ¿Si han habido retos, como se los han superado? 11. ¿Desde su punto de vista, en qué medida se ha desarrollado el proyecto según lo planificado? ¿Cuáles ajustes ha visto que se han hecho? ¿Han sido efectivos en términos de lograr el propósito del proyecto? ¿Cómo si o como no? 12. ¿Su organización/institución ha recibido apoyo de forma continua por el proyecto? Por favor explicar. 65 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV 13. ¿Considera usted que se ha quedado una capacidad instalada dentro de su organización para poder seguir beneficiándose del apoyo recibido? ¿En que sentido? 14. ¿Tiene alguna recomendación para el futuro del proyecto? Preguntar si tiene otros comentarios y/o preguntas. Si no, dar gracias por su tiempo. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 66 The purpose of this survey is to get to know your opinion and experience of the activities performed through the LAC Reads Capacity Program. This program seeks to improve the quality of teaching of reading and writing, and to contribute to the strengthening of policies and practices of their teaching and learning, through the collection and dissemination of evidence on initial reading and writing and institutional reinforcement. The information collected through this survey is anonymous and will not be shared on any individual level, but will be analyzed as a whole and used only to make improvements to the Program. Answering the survey will take approximately 20 minutes. Please be sure to click “ok” before moving on to the next question when this option appears. Survey: LAC READS Capacity Program (LRCP) Page 1: Demographic Information 1. Country of residence Other (please specify) 2. Organization/Institution 3. Position 4. Number of years in this position 5. Sex 6. Age Female Male 67 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV 7. Academic qualification Postgraduate degree Bachelor’s degree Associate degree Technical degree Other (please specify) Survey: LAC READS Capacity Program (LRCP) Page 1: General Information 8. How long ago did you learn about the LAC Reads Capacity Program? Less than 1 year ago Between 1-2 years ago Between 2-3 years ago Over 3 years ago 9. How did you find out about the LAC Reads Capacity Program? (Mark all that apply) Public news Contacted directly by someone from the program I was invited to an event Other (please specify) By colleagues My organization is a member of the program Through the organization I work at Need for technical education Interest in the subject To be able to replicate it Other (please specify) 10. Why did you want to participate in the activities created by the program? (Mark all that apply) To make institutional alliances To develop teams that promote improvements in initial reading and writing To have new experiences USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 68 11. Please indicate which activities (conferences/events/workshops) supported by the Program you participated in. (Mark all that apply and the level of usefulness of each one) Very useful useful Neutral Minorly useful Not useful Not Applicable Presentation of LAC Reads Program Workshop on initial reading and writing Workshop on pre￾reading and pre-writing Meeting to share experiences International or domestic expert conferences Workshop on systematization Country profile presentation Key Actors Analysis Presentation Systematic research review presentation Database presentation RedLEI Program Presentation Online courses Other (write the name of the activity and its level of usefulness) 69 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV 12. For the activities that you found to be useful, indicate why they were useful. (Mark up to three options) I have been able to learn more about LAC Reads and their work I have gained new knowledge on teaching methodology I have learned more about the learning process I have broadened my understanding of how to measure learning I have gained new knowledge on education research Other (please specify) I have been able to replicate it/share it with other people I have been able to put what I learned into practice at my work I have learned about organizations that support reading and writing and how they do it I have made important connections for the improvement of initial reading and writing Not applicable It is not very relevant in the context of my country The technical level is very high There has not been applicable information There was not enough time Other (please specify) 13. For the activities that were not useful for you, indicate why they were not useful. (mark all that apply) There was no continuity The activity did not give the opportunity to learn while doing Not applicable 14. Are you familiar with the LAC Reads Program (LRCP) website at the regional level? Yes USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 70 15. How useful has the information on the LAC Reads Program (LRCP) website been for you? Very useful Useful Neutral Minorly useful Not useful I have not used it Program Documents (i.e. Systematic Review, Country Profile, Key Actors Analysis) Studies on initial reading and writing Studies on pre-reading and pre-writing Studies on other issues in education Research Methods Information on learning evaluation Teaching resources (textbooks, teaching guides, stories, etc.) Studies/information on teacher education Studies/information on bilingual education Studies/information on intercultural bilingual education Other (please specify and write its level of usefulness) 71 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV 16. How useful has the information on initial reading and writing from other websites been for you? Very useful Useful Neutral Minorly useful Not useful I have not used it Other (please specify and write its level of usefulness) 17. Which of the following items produced by the LAC Reads Program are you familiar with? Systematic review Key actors mapping Country profile Regional database National database (i.e. UCA, FEREMA, CIASES, ASIES, EduConnect). None of these 18. How useful have these LAC Reads (LRCP) Program products been? Very useful Useful Neutral Minorly useful Not useful I have not used it Key Actors Mapping/Analysis Systematic Review UCA Virtual Library Country Profile National database (i.e. UCA, FEREMA, CIASES, ASIES, EduConnect). Regional Database Other (Please include a comment if you wish) Survey: LAC READS Capacity Program (LRCP) Page 3: Program Content USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 72 19. Please mark the level of agreement for each statement that best reflects your views Strongly agree Agree Neutral Disagree Strongly disagree I don’t know/I don’t have enough information The program has helped develop research skills on initial reading and writing The program has gathered very useful information to improve the development of reading and writing in the country The participating organizations use this information in their work The program has contributed to developing teaching and methodological skills to improve initial reading and writing The program has highlighted the importance of using evidence of good practices for initial reading and writing The program has strengthened the ability to check which practices help improve initial reading and writing 73 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV 20. How useful have the following activities in which you participated been for you? (Evaluate them according to the following options) Very useful Useful Neutral Minorly useful Not useful I have not used it Training in municipalities/districts/departments National-level training Education center-level training Online training Exchange of experiences with national organizations Conferences or exchanges with other countries Applying and analysis of the reading and writing tests (EGRA or other test methods) Other (write the name of the activity and its level of usefulness) 21. Why do you consider the selected activities to be useful? (Mark up to three options) I have been able to learn more about LAC Reads and their work I have gained new knowledge on teaching methodology I have learned more about the learning process I have broadened my understanding of how to measure learning I have gained new knowledge on education research I have been able to replicate it/share it with other people I have been able to put what I learned into practice at my work I have learned about organizations that support reading and writing and how they do it I have made important connections for the improvement of initial reading and writing Not applicable Other (please specify) USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 74 22. Why do you consider the selected activities to not be useful? (mark all options that apply) It is not very relevant in the context of my country The technical level is very high There has not been applicable information There was not enough time There was no continuity The activity did not give the opportunity to learn while doing Not applicable Other (please specify) 75 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV 23. How necessary do you consider the following options for improving the learning of reading and writing in your country? (Evaluate them according to the following options). Very Necessary Necessary Neutral Minorly Necessary Unnecessary Teacher trainers with higher technical abilities Better programs for initial education of teachers Teacher educations processes in service with practical applications Making research on effective strategies to improve initial reading and writing more accessible Better use of evidence on effective strategies to improve initial learning of reading and writing Policy and law changes focused on improving initial learning of reading and writing Agreement on the best methods of teaching-learning and strategies to improve initial reading and writing Promoting participation and support of families That children have materials and books to read for pleasure Foster recreational/playful environments for learning Generating more research on effective strategies to improve initial reading and writing USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 76 Survey: LAC READS Capacity Program (LRCP) Page 4: Sustainability 24. Once the technical and financial cooperation of the Program is complete, how important are the following program results, in order to continue improving reading and writing? (Evaluate them according to the following options). Strongly agree Agree Neutral Disagree Strongly disagree Not Applicable The developed skills for applying tests for initial reading and writing The developed teaching and methodological skills for initial reading and writing The ability to carry out rigorous research on initial reading and writing The methodological ability to verify practices that improve initial reading and writing The work done as a network of organizations established at a regional/international level The work done as a network of organizations established at a national level The updating of materials and studies in the virtual library/database The use of materials and studies in the virtual library/database The support of the private sector Greater participation from universities and think tanks 77 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Strongly agree Agree Neutral Disagree Strongly disagree Not Applicable Higher priority on reading and writing in the budget and in educational policy Other (please evaluate this using the above options) 25. Please evaluate the work done by the LAC Reads coordinating institution in your country: Very good Good Average Below average Bad Comment on your response if you wish 26. If the LAC Reads program concludes it cooperation now, has it achieved its goal of strengthening the capacity to improve initial reading and writing based on proven good practices? Strongly agree Agree Neutral Disagee Strongly disagree I don’t know Comment on your response if you wish USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 78 27. From the following options, indicate the three items that you consider to be the most important to continue strengthening the capacity to improve initial reading and writing based on proven good practices (Please only mark up to four options) Develop more systematic and specialized teaching and methodological strategies at the teacher level Strengthen the capacity to carry out rigorous research on initial reading and writing Strengthen the methodological capacity to verify practices that improve initial reading and writing Develop and sustain work as a network of organizations at the regional/international level Develop and sustain work as a network of organizations at the national level Update materials and studies in the virtual library/database Increase use of materials and studies in the virtual library/database Achieve sustained support from the private sector Achieve support from international cooperation Achieve sustained participation of universities and think tanks Give higher priority to initial reading and writing in the budget and educational policy 28. Do you have any final recommendations for the future of the LAC Reads Program? Thank you very much for taking the time to respond to this survey. 79 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX 4. SOURCES OF INFORMATION Persons Interviewed USAID Organization Title USAID/LAC Bureau LRCP/Agreement Officer’s Representative USAID/Nicaragua Education Specialist USAID/Guatemala Education Development Officer USAID/Guatemala Learning Advisor Deputy Director of the Health and Education Office USAID/Jamaica LRCP/Activity Manager USAID/Barbados LRCP/Activity Manager USAID/Dominican Republic LRCP Activity Manager USAID/Peru LRCP Activity Manager USAID/Honduras Education Specialist USAID/Honduras Monitoring and Evaluation Specialist Implementers Organization Title Juárez and Associates Former Regional Director REDLEI Regional Director Juárez and Associates Regional Literacy Specialist LRCP Regional Director AIR Project Director Juárez and Associates Former Project Director Partner Organizations Organization Title ASIES, Guatemala Communication Specialist Literacy Specialist ASIES, Guatemala Chief Coordinator RedLEI Director of the Education Research Center, UVG USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 80 CIASES, Nicaragua Principal Investigator, Education CIASES, Nicaragua Literacy Specialist CIASES, Nicaragua National Communication Specialist RedLEI Vice Rector for Research, collaboration, and Partnership, UVG RedLEI Regional Director Administrative Assistant Project Officer Professional Development Coordinator, UVG Communication Specialist RedLEI Research Coordinator OECS, St. Lucia Curriculum Specialist FEREMA, Honduras Operations Manager Communications Specialist Literacy Specialist EduConnect, Jamaica Program Manager Communications Support Specialist Literacy Specialist LRCP, Dominican Republic Manager Other Key Informants Organization Title LRCP Consultant Vitruvian Consulting, LLC President Juárez and Associates Senior Project Director UPANA Director, Master’s in Literacy for bilingual and intercultural environments CAPRI, Nicaragua Education Coordinator CARS, Nicaragua Director CESESMA, Nicaragua Education Coordinator Fundacion Fabretto, Nicaragua Education Coordinator PCI, Nicaragua Subdirector Save the Children, Nicaragua Program Manager Fe y Alegría, Nicaragua Education Coordinator USAID Honduras Reading Activity Project Manager FEREMA/Intibucá, Honduras Coordinator/Education Center Director 81 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV FEREMA/Comayagua, Honduras Coordinator/Teacher Plan International, Honduras Education Specialist UPNFM, Honduras Former Coordinator at FDI UPNFM, Honduras Assistant to the Academic Vice Rector/Literacy Specialist Host Country Officials Organization Title Ministry of Education/Guatemala Vice Minister of Bilingual Education Ministry of Education/Guatemala Technical Vice Minister of Education Ministry of Education/Guatemala Minister of Education Ministry of Education/Honduras Minister of Education Ministry of Education/Honduras Ex Vice Technical-pedagogical Minister Ministry of Education, Honduras Technical Assistant Ministry of Education, Honduras General Subdirector of Basic Education Ministry of Education, Honduras Education Specialist Ministry of Education, Montego Bay, Jamaica National Literacy Coordinator Literacy Coordination Team Member DIGECADE, Guatemala Early Grade Manager Technical Specialist DIGECUR, Guatemala Director DOCUMENTS REVIEWED 1. American Institutes for Research, Early Grade Reading in Latin America and the Caribbean: A Systematic Review, December 2016. American Institutes for Research and Juárez & Associates, Monitoring and Evaluation Plan. 2. American Institutes for Research, RedLEI Sustainability Assessment Workplan, date unknown. American Institutes for Research and Juárez & Associates, Technical Application Lac Reads Capacity Program, March 2014. 3. American Institutes for Research, What works to improve Early Grade Literacy in Latin America and the Caribbean: 2016 Update of a Systematic Review, date unknown. 4. American Institutes for Research and Juárez & Associates, Monitoring and Evaluation Plan. 5. American Institutes for Research and Juárez & Associates, Technical Application Lac Reads Capacity Program, March 2014. 6. American Institutes for Research, RedLEI First Quarterly Report, April 2018. 7. American Institutes for Research, RedLEI Second Quarterly Report, July 2018. 8. American Institutes for Research and Juárez & Associates, Quarterly Performance Report, September 2018. USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 82 9. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, June 2018. 10. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, March 2018. 11. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, September 2017. 12. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, June 2017. 13. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, March 2017. 14. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, December 2017. 15. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, July 2016. 16. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, September 2016. 17. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, March 2016. 18. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, December 2016. 19. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, September 2015. 20. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, March 2015. 21. American Institutes for Research and Juárez & Associates, LAC Reads Capacity Program Quarterly Performance Report, December 2015. 22. American Institutes for Research and The Center for Research and Social Educational Action, Nicaragua Country Profile and Stakeholder Mapping in Early Grade Reading, August 2017. 23. CIASES, Nicaragua Country Profile, 2015. 24. CIASES, Stakeholder Analysis Early Grade Literacy, date unknown. 25. EDUCA, Country Profile, date unknown. 26. EDUCA, Stakeholder Analysis Early Grade Literacy, date unknown. 27. EduConnectJA, Jamaica Country Profile, date unknown. 28. EduConnectJA, Jamaica: Stakeholder Analysis Early Grade Literacy, date unknown. 29. FEREMA, Honduras Country Profile, 2016. 30. FEREMA, Stakeholder Analysis Early Grade Literacy, date unknown 31. International Business & Technical Consultants, Inc., Evaluation of the LAC/RSD Regional Education Program, March 2011. 32. JBS International, LAC Regional Education Policy Assessment, June 2013. 33. O’Brien & Associates International, RedLEI Sustainability Assessment: Technical and Cost Proposal, September 2017. 34. U.S. Agency for International Development, Request for Applications for the LAC Reads Capacity Program, Jan. 2014. 83 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX 5. DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Kristin Rosekrans Title Evaluation Team Leader Organization DevTech Systems, Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Evaluation conducted under the LAC Education Support Contract 7200AA18M00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Lac Reads Capacity Program, implemented by American Institutes for Research and Juárez and Associates I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 4/4/2019 Name Daniela Cardozo X USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 84 Title Research Assistant Organization Resonance Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Evaluation conducted under the LAC Education Support Contract 7200AA18M00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) LAC Reads Capacity Program, implemented by American Institutes for Research and Juárez and Associates I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Daniela Cardozo Date 3.28.2019 Name Ana Roxanda Rodríguez Cabrera Title Phd. Political Science and Sociology Organization DEVTECH SYSTEMS, INC. Evaluation Position? Team Leader Team member 85 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV Evaluation Award Number (contract or other instrument) Evaluation conducted under the LAC Education Support Contract 7200AA18M00024 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) LAC Reads Capacity Program implemented by American Institutes for Research and Juárez and Associates. I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. does not apply I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 27, 2019 Name Mario Alas Solís Title Ph. D. en Educación Organization DEVTECH SYSTEMS, INC. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Evaluation conducted under the LAC Education Support Contract 7200AA18M00024 USAID.GOV LRCP MIDTERM PERFORMANCE EVALUATION | 86 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) LAC Reads Capacity Program; Implemented by American Institutes for Research (AIR) and Juárez and Associates I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. does not apply I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 27, 2019 Name Celso Luis Asensio Flórez Title Consultant in Research, Evaluation and Education Organization DevTech Systems, Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Evaluation conducted under the LAC Education Support Contract 7200AA18M00024 87 | LRCP MIDTERM PERFORMANCE EVALUATION USAID.GOV USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) LAC Reads Capacity Program; Implemented by American Institutes for Research (AIR) and Juárez and Associates I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 03.27.2019