This publication was produced independently by Brenda Sinclair, Souhila Messaoud-Galusi, Salika Khoonbarthao, Nouandam Kommana, Amy Porter, Dominick Margiotta, and Jennifer Elkins of Social Impact, Inc., at the request of the United States Agency for International Development. Photo Credit: Amazon News EVALUATION REPORT MIDTERM PERFORMANCE EVALUATION OF THE USAID/LAOS LEARN TO READ ACTIVITY November 20, 2021 (THIS PAGE INTENTIONALLY LEFT BLANK) i | FINAL EVALUATION REPORT USAID.GOV TABLE OF CONTENTS TABLE OF CONTENTS....................................................................................................................I LIST OF FIGURES AND TABLES...................................................................................................II FIGURES .............................................................................................................................................................................. II TABLES ............................................................................................................................................................................... III ACRONYMS..................................................................................................................................... IV EXECUTIVE SUMMARY................................................................................................................VII INTRODUCTION ......................................................................................................................................................... VII EVALUATION OVERVIEW AND PURPOSE ......................................................................................................... VII EVALUATION QUESTIONS....................................................................................................................................... VII EVALUATION METHODS AND LIMITATIONS ................................................................................................. VIII FINDINGS AND CONCLUSIONS .......................................................................................................................... VIII RECOMMENDATIONS.................................................................................................................................................XI INTRODUCTION.............................................................................................................................1 EVALUATION OVERVIEW............................................................................................................................................ 1 PROGRAM CONTEXT AND BACKGROUND...................................................................................................... 1 LEARN TO READ ACTIVITY PARTNERS................................................................................................................. 2 EVALUATION QUESTIONS ..........................................................................................................3 EVALUATION QUESTIONS.......................................................................................................................................... 3 EVALUATION DESIGN AND METHODOLOGY....................................................................5 EVALUATION DESIGN .................................................................................................................................................. 5 DESK REVIEW ........................................................................................................................................................ 5 DATA COLLECTION........................................................................................................................................... 5 SCHOOL SELECTION CRITERIA..................................................................................................................... 5 KEY INFORMANT INTERVIEWS...................................................................................................................... 6 SAMPLING.......................................................................................................................................................................... 6 DATA ANALYSIS.............................................................................................................................................................. 8 LIMITATIONS .................................................................................................................................................................... 9 FINDINGS AND CONCLUSIONS ............................................................................................ 10 EQ1: ACHIEVEMENT OF RESULTS...........................................................................................................................10 EQ1 FINDINGS ....................................................................................................................................................10 EQ1 CONCLUSIONS.........................................................................................................................................17 EQ2: IMPLEMENTATION FACTORS........................................................................................................................17 EQ2 FINDINGS ....................................................................................................................................................17 SUSTAINABILITY.................................................................................................................................................26 Sustainability Findings from Desk Review............................................................................................................... 26 SUSTAINABILITY KII FINDINGS ....................................................................................................................27 EQ2 CONCLUSIONS: INTERNAL/EXTERNAL FACTORS ....................................................................33 EQ2 CONCLUSIONS: SUSTAINABILITY.....................................................................................................34 EQ3: LTR/MOES COLLABORATION.......................................................................................................................34 ii | FINAL EVALUATION REPORT USAID.GOV EQ3 FINDINGS ....................................................................................................................................................35 EQ3 CONCLUSIONS.........................................................................................................................................43 EQ4: DEVELOPMENT PARTNERS.............................................................................................................................44 EQ4 FINDINGS AND CONCLUSIONS........................................................................................................44 EQ4 CONCLUSIONS.........................................................................................................................................52 RECOMMENDATIONS ................................................................................................................ 54 ANNEX A: EVALUATION DESIGN MATRIX ........................................................................ 57 ANNEX B: FULL RECOMMENDATIONS................................................................................ 62 EQ1 RECOMMENDATIONS............................................................................................................................62 EQ2 RECOMMENDATIONS............................................................................................................................63 EQ3 RECOMMENDATIONS............................................................................................................................64 EQ4 RECOMMENDATIONS............................................................................................................................65 ANNEX C: BIBLIOGRAPHY........................................................................................................ 67 ANNEX D: DATA COLLECTION PROTOCOLS ................................................................. 68 INTERVIEW PROTOCOL: USAID STAFF ..................................................................................................68 INTERVIEW PROTOCOL: SC STAFF ...........................................................................................................73 INTERVIEW PROTOCOL: RTR AND HI STAFF .....................................................................................79 INTERVIEW PROTOCOL: MOES STAKEHOLDERS ............................................................................85 INTERVIEW PROTOCOL: PROVINCIAL AND DISTRICT STAFF...............................................93 INTERVIEW PROTOCOL: PEDAGOGICAL ADVISORS/SCHOOL PRINCIPALS................98 INTERVIEW PROTOCOL: TEACHERS (PP, G1, G2)..........................................................................103 FGD PROTOCOL: PARENTS/CAREGIVERS ...........................................................................................108 FGD PROTOCOL: VEDC ....................................................................................................................................111 ANNEX E: EVALUATION TEAM............................................................................................. 114 ANNEX F: TERMS OF REFERENCE ........................................................................................ 115 LIST OF FIGURES AND TABLES FIGURES Figure 1: EQ1 Most Effective Component.................................................................................................................13 Figure 2: EQ1 Least Effective Component ................................................................................................................15 Figure 3: EQ2 COVID Impacts.....................................................................................................................................18 Figure 4: EQ2 LTR Operational Areas for Improvement......................................................................................23 Figure 5: EQ2 Milestones...............................................................................................................................................28 Figure 6: EQ2 Project Sustainability at MoES Survey Response...........................................................................29 Figure 7: EQ2 Recommendations for Sustainability ................................................................................................30 Figure 8: EQ3 Satisfaction with Level of Engagement.............................................................................................37 Figure 9: EQ4 Areas Where Additional Support is Required...............................................................................41 iii | FINAL EVALUATION REPORT USAID.GOV TABLES Table 1: Evaluation Questions and Related Themes................................................................................................3 Table 2: Sample Size and Key Stakeholders................................................................................................................8 Table 3: Stakeholders’ Suggestions for Areas for Improvement.........................................................................23 Table 4: Percentage of Stakeholders Providing Recommendations.....................................................................30 Table 5: Sample Size for Increased Stakeholder Engagement Question.............................................................41 Table 6: Complementary Components of LTR and Development Partners’ Supported Programs............45 iv | FINAL EVALUATION REPORT USAID.GOV ACRONYMS AMELP BEQUAL CBSR CE CLA CPD CPS DESB DFAT DTE DQA ECE ECD EGRA EQ ESP ET EU FCR FGD GE GESI G1 G2 GE GIDAP GoL GPE GPE II GPE III HI Activity Monitoring, Evaluation, and Learning Basic Education Quality and Access Learning Community-based School Readiness Community Engagement Collaborative, Learning and Adapting Continuous Professional Development Champasack Province District Education and Sport Bureau Department of Foreign Affairs and Trade Department of Teacher Education Data Quality Assurance Early Childhood Education Early Childhood Development Early Grade Reading Assessment Evaluation Question Effective Schools Program Evaluation Team European Union Findings, Conclusions, and Recommendations Focus Group Discussion Department of General Education Gender Equality and Social Inclusion Grade 1 Grade 2 General Education Gender and Inclusive Development Action Plan Government of Lao PDR Global Partnership for Education Global Partnership for Education II Global Partnership for Education III Humanity & Inclusion v | FINAL EVALUATION REPORT USAID.GOV ICT IE IEPC IRB IPs KII Lao PDR LCRP LEAP LTR M&E MEL MEAL MoES MoU MTE PA PESS PPE PP PO PWDs QA RIES RF RRP RtR SC SI SLP TIP Information and Communications Technology Inclusive Education Inclusive Education Promotion Center (IEPC) Institutional Review Board Implementing Partners Key Informant Interview Lao People’s Democratic Republic Laos Country Representative Learning and Equity Acceleration Project Learn To Read Monitoring and Evaluation Monitoring, Evaluation and Learning Monitoring, Evaluation, Accountability, and Learning Ministry of Education and Sports Memorandum of Understanding Mid-Term Evaluation Pedagogical Advisor Provincial Education and Sports Service Personal Protective Equipment Pre-primary Project Officers People With Disabilities Quality Assurance Research Institute for Educational Services Results Framework Reading Readiness Package Room to Read Save the Children Social Impact Spoken Lao Language Pilot Trafficking in Persons vi | FINAL EVALUATION REPORT USAID.GOV TL TLCs TLMs TTC UNICEF US USAID VEDC VTE XHK YLCs Team Leader Teacher Learning Circles Teaching and Learning Material Teacher Training College United Nations Children’s Fund United States United States Agency for International Development Village Education Development Committee Vientiane Province Xiengkhouang Youth Literacy Champions vii | FINAL EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY INTRODUCTION This document outlines the results of a midterm performance evaluation (MTE) of the United States Agency for International Development (USAID) Laos Country Representative Office Activity: USAID Learn to Read (LTR). The evaluation was conducted by Social Impact (SI) over the period of May through November 2021. Data collection took place virtually in July 2021 across four districts: Phonghong and Vang Vieng districts in Vientiane province and Mounlapamok and Bachieng districts in Champasack province. LTR is implemented through a consortium led by the Save the Children (SC) with Room to Read (RtR) and Humanity & Inclusion (HI) as implementing partners. The five-year project, covering the period of August 15, 2018 to August 14, 2023, aims to improve the reading skills of all children from pre-primary level to second grade, while also specifically targeting non-Lao-Tai speaking students and children with disabilities. LTR supports the Ministry of Education and Sports (MoES) with implementation of the national curriculum through the development of evidence-based supplemental reading materials, improving classroom instruction, and strengthening community engagement for improved literacy. LTR is implemented across 16 districts in Vientiane, Xiengkhouang, Attapeu, and Champasack provinces, reaching an estimated minimum of 63,584 children over the life of the project. EVALUATION OVERVIEW AND PURPOSE The purpose of the LTR performance evaluation was to assess progress achieved towards activity objectives, identify challenges encountered, and determine key activities that could potentially accelerate the sustainability of the LTR model within the Laos educational system. The evaluation was conducted in Year 3 of the Activity in order to provide recommendations to improve the performance and sustainability of the Activity during its remaining period from 2021 to 2023. The evaluation also sought to explore other development partner supported programs that complement LTR activities to guide the design of further USAID education investments in Laos. The primary audience for this midterm evaluation is USAID, SC, and their sub-implementing partners, and the MoES. EVALUATION QUESTIONS The four key evaluation questions and overarching themes are: 1. Progress achieved towards activity objectives: How likely will the three key result areas, enumerated in the results framework (RF), be achieved within the set timeframe? 2. Activity implementation/operations factors: What (and how) are internal processes (e.g., funding, management/operations, changes of context, etc.) supporting and/or hindering implementation? 3. Stakeholder dynamics: What evidence exists and what does it tell us, about how effectively LTR has been engaging the MoES and other key stakeholders at the central, provincial and district levels to meet the Activity objectives? 4. LTR partner activities: What aspects of other development partners’ supported programs, such as the World Bank second Global Partnership for Education (GPEII) and the Basic Education Quality and Access in Lao People’s Democratic Republic (Lao PDR) project (BEQUAL), are most complementary to, and support the goals and objectives of LTR? viii | FINAL EVALUATION REPORT USAID.GOV EVALUATION METHODS AND LIMITATIONS Due to the COVID-19 context and lack of available baseline data, the evaluation team (ET) utilized qualitative methods of data collection. Primary sources of data were collected through semi-structured key informant interviews (KIIs) and secondary data was obtained through desk review of relevant documents provided by USAID and SC. Overall, 95 individuals participated in the study. The respondent sample was comprised of 22 stakeholders at the central level representing USAID, MoES, Implementing Partners (IPs) and development partners. Across Champasack (CPS) and Vientiane (VTE) provinces, a total of 20 provincial and district government officials from the Provincial Education and Sports Service (PESS) and District Education and Sports Bureau (DESB) participated in KIIs. At the school level, 10 principals, 21 teachers (from pre-primary, grade 1, grade 2 and multigrade classes), and 20 community members were interviewed from 10 schools with a predominantly high proportion of non-Lao-Tai speakers. Respondents were selected through purposeful and convenience sampling taking gender, ethnicity, location, and experience into account. Several limitations of the evaluation include: 1) remote fieldwork, which may have altered interview dynamics, 2) non-random sample selection with a small sample size, which could cause selection bias; 3) lack of baseline data, Early Grade Reading Assessment results, or targets; and 4) gender imbalance among some respondent categories, due to inherent gender structures within schools and communities (e.g., male-dominated leadership positions). The ET mitigated these limitations through a thorough review of LTR Activity reports, construction of rigorous data collection tools, selection criteria, and triangulation of secondary data sources and primary data from multiple stakeholder groups. A summary of the key findings and conclusions for each evaluation question (EQ) is presented below. FINDINGS AND CONCLUSIONS EQ1- How likely will the three key result areas, enumerated in the RF, be achieved within the set timeframe? LTR is likely to achieve results for most components if the COVID-19 situation improves. Teacher training (Result 2) was the most effective component due to a comprehensive teacher professional development system that combines quality teaching and learning materials (TLMs) with phased teacher training, classroom coaching, and continuous professional development. Result 1, the materials development component, is also on track to achieve results. The production of reading corners and TLMs were major accomplishments that significantly contributed to the improvement of teaching instruction and reading skills among Lao-Tai and non-Lao-Tai speaking students. The most common challenges described by the teachers and trainers interviewed under Results 1 and 2 were a shortage of TLMs and storybooks, especially in pre-primary classrooms, and difficulty applying materials and methodologies in multigrade classrooms although LTR delivered training on this topic in January 2021.1 The community engagement (CE) component (Result 3) was considered the least effective component to date, and the least likely to achieve results by Activity end. Critical constraints affecting this component include delayed implementation in the south due to COVID-19, lack of a clear and well-articulated 1 SC reported that although 99% of schools with a pre-primary class on record received books, some schools were excluded if the records for that school were not accurate. ix | FINAL EVALUATION REPORT USAID.GOV strategy, sole reliance on schools to engage Village Education Development Committees (VEDCs) and parents without direct LTR support, lack of training and orientation for VEDCs and parents, and parent time constraints. One key challenge in rural areas is that families stay overnight on farms, especially during the harvesting season, which does not allow them to participate in school or LTR activities. Ten of 56 stakeholders, including PESS, DESB, school principals, teachers, parents and VEDCs in CPS and VTE, asserted that parents’ lack of understanding about the importance of education, particularly among ethnic parents in rural communities, is another barrier affecting CE. However, this does not reflect all parents, but about 30 percent of parents who do not enroll their children in school or who do not send them regularly. Teachers, parents and DESB PAs reported that language barriers affect parents’ participation because when they attend meetings, teachers speak too fast and parents may not fully understand the topics presented. Insufficient notice of training from DESB or schools and lack of communication were other inhibiting factors. Unlike the other two results that are closely linked to the MoES system, the CE component was viewed as an external component. As DESB is not engaged in monitoring CE activities and VEDCs have not been oriented to their role, Result 3 lags behind the other two results. EQ2 - What (and how) are internal processes supporting and/or hindering implementation? Implementation Factors: COVID-19 is the most significant external barrier impeding implementation. The COVID-19 lockdown from March to June 2020 and again from April 2021 to the present has disrupted teaching and learning. Due to lack of warning, students did not have access to textbooks, storybooks, or other learning materials. Without these materials or online resources, students could not complete the curriculum. COVID-19 also affected teachers, who were recently trained by LTR and lacked sufficient classroom practice to retain knowledge. Principals reported that teachers are forgetting concepts and methods learned in LTR trainings. Teacher training for Group 3 was postponed while master training was held virtually. Teacher-parent conferences and Reading Festivals under Result 3 were delayed as well. Another factor restricting implementation internally include a shortage of staff, such as Monitoring, Evaluation, Accountability, and Learning (MEAL) staff, project officers and community facilitators. For instance, although implementers agree that the workload justifies the need for more LTR staff in Attapeu province, some districts are still challenged to manage the heavy workload with limited staff because PESS does not want to deviate from the original staffing structure in the MOU. Moreover, insufficient notice of school visits hampers MoES PESS, and DESB participation, who are all eager to join. PESS and DESB all requested that LTR provide at least one week’s notice for school monitoring visits in order to afford them the opportunity to coordinate and participate in LTR activities. Factors supporting the implementation of the Activity included: MoES support, SC Hong Kong cost share, collaboration with BEQUAL/GPE, and a strong LTR team. Although the shortage of staff is a key constraint, LTR’s strong teamwork, relationship with the MoES at all levels, and with development partners accelerated the project’s achievement of results. One major achievement of the project was gaining the trust, support, and cooperation of the MoES. Sustainability: According to LTR’s phased approach to capacity building as defined in the Cooperative Agreement, LTR was at the end of Phase 2 (joint implementation in Cohort 2 schools) and beginning of Phase 3 (implementation by provincial or district government in Cohort 3). However, the sustainability x | FINAL EVALUATION REPORT USAID.GOV plan was not developed and communicated to all government stakeholders at the inception of the Activity. Hence, none of the 20 stakeholders asked about the sustainability plan were aware of the capacity building strategy (gradual transfer of leadership to provincial and district governments), while half were aware of the phased approach to school implementation. The ET learned that no sustainability plan had been shared with MoES, nor benchmarks for tracking progress, as referenced in the cooperative agreement. Rather the sustainability plan was recently developed and submitted as part of Year 4 planning. Thus, stakeholders were not aware of this plan, which is reflected in the KII findings. Two-thirds of stakeholders believe the Activity will be sustained, specifically the LTR materials, training, and monitoring. However, most agreed that it depends on MoES budget allocations. One-third of stakeholders believe community engagement is more difficult to sustain because the CE strategy is not linked to the MoES development plan; and is hampered by limited budget, resources, and community buy￾in. Stakeholder recommendations for ensuring sustainability indicate that there is a strong need for further capacity building and engagement of MoES in monitoring and evaluation (M&E), strengthened community engagement, additional refresher training, TLMs and support for teachers, a sustainable mechanism for producing TLMs, and a COVID-19 response plan for continued teaching and learning. EQ3 - What evidence exists and what does it tell us, about how effectively LTR has been engaging the MoES and other key stakeholders at the central, provincial and district levels to meet the Activity objectives? Government stakeholders’ overwhelming satisfaction with their level of participation in the LTR Activity indicates LTR has effectively engaged relevant departments of the MoES. The most effective forms of engagement are alignment of the LTR activity with MoES priorities and joint implementation of the Activity with MoES and development partners. At the community level, VEDCs and parents in CPS were not satisfied with their level of engagement. Four out of five parents interviewed in CPS had not been engaged in any LTR activities. VEDCs also lacked participation due to insufficient notice provided for trainings from schools or DESB officers, and also because the LTR activity did not inform the village authority directly, failing to meet their expectations.2 The CE strategy of working through schools and teachers to reach community members was effective in VTE, where most parents were actively participating and satisfied, but it may not be sufficient for meeting communities’ expectations in CPS. Despite lack of parent engagement in CPS, parents in both provinces requested more storybooks and training on how to support their children’s reading. EQ4 - What aspects of other development partners’ supported programs, such as the World Bank second Global Partnership for Education (GPEII) and the Basic Education Quality and Access in Lao PDR project (BEQUAL), are most complementary to, and support the goals and objectives of LTR? Based on interviews and a desk review of the next phase of UNICEF, GPE and BEQUAL programs, there are six complementary components with LTR: 1) ECE and primary teacher professional development, including RRP, 2) multigrade teaching, 3) COVID-response, 4) online teaching and learning platform, 5) community engagement, and 6) read-at-home initiatives. Among the pilot-tested interventions to date, RRP is the most credible and compelling activity. It has been pilot-tested by LTR, shows promising results 2 Although it is not LTR’s responsibility to notify VEDCs or parents of school-based activities, this finding demonstrates the expectation of stakeholders and lack of clarity on the roles and responsibilities among LTR, VEDCs and DESB officers in implementing CE activities. VEDCs expect LTR to orient them as well as the village authority. xi | FINAL EVALUATION REPORT USAID.GOV through a preliminary World Bank impact evaluation, and has been adopted by the MoES for integration into the new pre-primary curriculum. There is also significant overlap between UNICEF and LTR’s activities and target populations. UNICEF’s COVID-response interventions3, multi-media activities, and CE strategies are well aligned with the needs and gaps identified in this LTR midterm evaluation. Although BEQUAL will be working in higher grades for Phase II, specific activities that complement LTR include the Spoken Lao Language Pilot (SLP) and training on the new curriculum for non-Lao-Tai speaking students and ethnic teachers. RECOMMENDATIONS The recommendations have been consolidated into eight critical recommendations. These recommendations are listed in order of priority and further elaborated by EQ in Annex B. 1. LTR should develop a COVID-Response Plan to increase the likelihood of achieving results and sustainability for all three components. The COVID-Response Plan should outline strategies to train teachers through virtual or blended trainings, and to engage parents in at-home learning activities with access to TLMs. The comprehensive plan should also include elements recommended in the MoES COVID-19 response plan and by LTR stakeholders interviewed, including: PPE, at-home learning kits, back-to-school campaign featuring COVID-19 prevention protocols, catch-up lessons, online learning for urban schools, and a mobile community library so that storybooks are accessible during school closures. LTR should work with the MoES and development partners to develop a comprehensive plan building on existing pilot initiatives (See EQ4). LTR should leverage inputs from SC, MoES, UNICEF and other development partner staff involved in developing the Lao PDR MoES Education COVID-19 Response plan[1]. 2. LTR should develop a more effective community engagement strategy that links to the MoES Development plan. The strategy should focus on engaging VEDCs and parents in rural and remote communities. LTR should engage village authorities in LTR orientations and CE promotion. The Village Chief can disseminate information to villagers and mobilize communities to support reading. Trainings and orientations for parents should be organized around their schedules and conducted in a familiar language. In addition, LTR should involve older siblings of targeted students in reading activities because many teachers mentioned that students receive support from them to read. LTR should provide additional storybooks and guidance for how families can support their children’s reading. LTR should collaborate with UNICEF to design CE activities, such as engaging communities in materials development and read at home initiatives, reading campaigns and revolving community libraries. 3. Aligned with the LTR Cooperative Agreement, LTR should develop and/or modify a sustainability plan for the remaining period along with benchmarks for tracking progress and milestones. The sustainability plan should focus on transferring leadership responsibility to MoES in Phase 3, and building the capacity of DESB officers for training, coaching, and monitoring. LTR should integrate recommended activities from stakeholders from this midterm evaluation into the sustainability plan, capacity building of MoES, PESS and DESB for monitoring and assessment, refresher training and additional support for teachers, strengthening community engagement, and creating an extensive library of TLMs leveraging Khang Panya Lao resources, public-private 3 Lao PDR Education COVID-19 Response Plan http://www.moes.edu.la/moes/images/Argreement/Lao_PDR_Education_COVID-19_Response_Plan.pdf xii | FINAL EVALUATION REPORT USAID.GOV partnerships for printing new materials, and engaging teachers, students and community members in developing new storybooks and teaching aids. 4. LTR should provide additional refresher training for teachers: Over half of evaluation respondents requested additional refresher training and support for teachers, particularly multi￾grade and pre-primary teachers. Thus, LTR should provide refresher trainings for each cohort and additional resources until they master the content and methodology, which would increase the likelihood of sustainability and scale-up to new teachers. Refresher training is critical because teachers are already forgetting what they have learned in initial trainings. A refresher training for schools involved in this evaluation was scheduled for Summer 2021, at the time of data collection for this study. (For full list of recommended training topics, see EQ2 Recommendations in Annex B). 5. USAID and MoES should develop a strategy for sustaining and printing more materials: To address the shortage of materials, the need for more variety or replenishment, and to increase the likelihood of sustainability for Result 1, USAID and MoES should seek to engage local book publishers to print and distribute LTR storybooks as well as other ECE storybooks through public-private partnerships. 6. LTR should review the staffing structure in the MoU with USAID and the MOES against the workload requirements in each province to ensure staffing is sufficient to achieve Activity goals for Phase 3. According to Project Coordinators and Project Officers (POs), there is a shortage of POs in Attapeu province. Additional qualified MEAL staff and Youth Literacy Champions in ethnic communities with low community engagement are also recommended for Year 4, which may already be included in LTR’s Work Plan. 7. In general, LTR should provide advance notice to stakeholders for LTR activities: To increase the engagement of all stakeholders in LTR activities, LTR should provide at least one week notice for training, meetings, and joint school monitoring visits. Due to their obligations to multiple donor projects, the MoES requested several months’ notice. VEDCs, school principals and parents would also like to be notified when LTR is conducting school visits so that they can also partake in any LTR activity where communities are involved. Furthermore, it would provide an opportunity for LTR to meet with parents and VEDC members and provide orientation or address any concerns. 8. LTR should continue collaborating with development partners on activities that overlap with LTR objectives and target populations: For instance, LTR should continue implementing, refining, and scaling up RRP with GPE. LTR should collaborate with GPE, BEQUAL and UNICEF on COVID-response activities. LTR should collaborate with UNICEF on complementary activities that are well aligned with the needs and gaps identified in this midterm evaluation, specifically ECE materials development, blended training, and community engagement activities. USAID and LTR should further explore opportunities to collaborate with the MoES and BEQUAL on the Spoken Lao Language pilot, continuous professional development for teachers in Grades 1 and 2, and on lesson plans for multigrade classes. (See full list of recommended activities under EQ4 Recommendations). 1 | FINAL EVALUATION REPORT USAID.GOV INTRODUCTION EVALUATION OVERVIEW In May 2021, the United States Agency for International Development’s (USAID) Laos Country Representative Office Activity contracted Social Impact, Inc. (SI) to conduct a midterm performance evaluation (MTE) of the USAID Learn to Read (LTR) Activity implemented by Save the Children (SC) and its sub-partners, Room to Read (RtR) and Humanity & Inclusion (HI). The midterm evaluation took place in Year 3 of the five-year Activity that runs from August 15, 2018 to August 14, 2023. Data collection was carried out remotely in July 2021. PURPOSE The purpose of the midterm evaluation is to identify main areas of progress achieved to date, key challenges that have impeded results, and major factors that could potentially influence or enhance the sustainability of the LTR model within the Lao educational system. The MTE will provide key recommendations for improving the performance and sustainability of the Activity during its remaining period from 2021 to 2023. TARGET AUDIENCE The primary audience for this midterm evaluation is USAID, SC and their sub-partners, and the Government of Lao PDR (GoL). These stakeholders will use the findings and recommendations from this evaluation to address gaps and build on the successes to improve the implementation and sustainability of the project. The secondary and tertiary audiences of this evaluation are other government and non￾government organizations working on similar issues in Lao People’s Democratic Republic (Lao PDR), and the interested general public. PROGRAM CONTEXT AND BACKGROUND Laos has made significant progress in primary school enrollment over the past five years, with a national enrollment rate of 97 percent.4 Despite these achievements, drop-out rates at the primary level remain high, particularly in more remote and rural areas. Children in households that do not speak Lao as a first language – approximately half of the total population – face additional barriers and must first develop basic communication skills in the Lao language before they can access the national education curriculum. Similarly, children with mild disabilities (e.g., common vision impairments or partial hearing loss) require additional support to successfully learn to read. Children in non-Lao-Tai speaking households also have low exposure to reading books in Lao outside of school and insufficient availability of reading materials in Lao at home. The goal of the USAID Learn to Read Activity is to improve the reading skills for all children in pre￾primary (PP) level through second grade, with a particular focus on non-Lao-Tai-speaking children and children with disabilities. LTR aims to achieve this goal through three key results: 1) providing supplemental 4 USAID. (2019). Learn to Read Fact Sheet. https://www.usaid.gov/sites/default/files/documents/1861/USAID_Learn_To_Read_JAN_2019.pdf. 2 | FINAL EVALUATION REPORT USAID.GOV materials, 2) teacher training and support, and 3) community engagement. Under Result 1, the objective is to increase access to inclusive supplementary teaching and learning materials for reading. LTR develops Lao literacy program packages for pre-primary and early primary classrooms that specifically address the learning needs of non-Lao-Tai speaking students and children with mild disabilities. For Result 2, the expected outcome is improved application of inclusive pedagogical approaches for reading by teachers, principals and Ministry of Education and Sports (MoES) officials. The expected outcome of Result 3 is improved home and community environment to support reading. LTR will reach an estimated 65,313 children by the end of the project. LTR is implemented through a phased approach across 16 districts in four provinces of Lao PDR, including Vientiane, Xiengkhouang, Attapeu, and Champasack Province. The phased approach begins with 30 percent of schools targeted in Year 1 (Group 1); 40 percent of schools beginning in Year 2 (Group 2) and 30 percent of schools beginning in Year 3 (Group 3). During Phase 1, the Activity is led by LTR and implemented in close coordination with target district and provincial governments. In Phase 2, the Activity is jointly implemented with district and provincial governments and project activities are expanded to new schools within the target districts. For Phase 3, LTR will transfer increased leadership responsibility for implementation to the district and provincial governments, who will also lead the expansion of the LTR interventions in new schools within the target districts with guidance from LTR. This approach allows for testing and adaptation prior to expansion as well as gradual transfer of responsibility to provincial and district officials as part of LTR’s sustainability plan, as described in the Cooperative Agreement. LEARN TO READ ACTIVITY PARTNERS The LTR Activity is working closely with the MoES and development partners to implement evidence￾based approaches to improve reading. The Activity works collaboratively through a co-design and co￾implementation methodology with specific MoES units, including the: Research Institute for Educational Sciences (RIES), Inclusive Education Promotion Center (IEPC), and Department of Teacher Education (DTE), Early Childhood Education (ECE), Department of Planning, Department of External Relations, and Department of General Education (GE) – all of whom participated in this evaluation. LTR also works closely with leading development partners to strengthen the Lao education system with evidence-based approaches. These partners include the second Global Partnership for Education (GPE II) to collaborate on pre-primary readiness and Basic Education Quality and Access in Lao PDR project (BEQUAL) to support reading improvement and the grade 1 (G1) and grade 2 (G2) national primary curricula rollout. A key LTR priority is to align and coordinate with these education initiatives to ensure that it 1) complements and augments ongoing literacy efforts and 2) maximizes the return on MoES and donor investments in the education sector. 3 | FINAL EVALUATION REPORT USAID.GOV EVALUATION QUESTIONS EVALUATION QUESTIONS The evaluation focused on four key evaluation questions (EQ) related to the following thematic areas: 1) Progress achieved towards Activity objectives; 2) Activity implementation and operations factors; 3) Stakeholder dynamics; and 4) LTR partner activities. Table 1 shows the key questions followed by sub￾questions aimed to explore the themes in more depth. Sustainability questions appear in EQ2d and EQ3c. Table 1: Evaluation Questions and Related Themes EQs Theme Questions EQ1 Progress towards Activity objectives 1. How likely will the three key result areas, enumerated in the results framework, be achieved within the set timeframe? a. Which components of the three key result areas are perceived to have been most effective/successful during the implementation to date? What is driving these successful components? b. Which components, of the three key result areas, are perceived to have been least successful to date? What is hindering these components? EQ2 Activity implementation/ operations factors 2. What (and how) are internal operational processes, - other than Memorandum of Understanding (MoU) process, (i.e., funding, management/operations, changes of context, etc.) supporting and/or hindering implementation? a. What external/internal factors, if any, have impeded the implementation of the project (e.g., COVID-19, funding, etc.)? b. Are there any external/internal factors that have supported the implementation of the project (e.g., management, operations)? If yes, what, and how? c. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed and/or improved to increase the likelihood of sustainability? EQ3 Stakeholder dynamics 3. What evidence exists, and what does it tell us, about how effectively Learn to Read has been engaging the Ministry of Education and Sports and other key stakeholders at central, provincial, and district levels to meet the Activity objectives? 4 | FINAL EVALUATION REPORT USAID.GOV a. MoES: Are Ministry of Education and Sports stakeholders at the central, provincial and district levels satisfied with their level of participation? Do they perceive that their expectations have met by LTR to date? b. Subcontractors/IPs: Are program partners (Room to Read, Humanity & Inclusion) satisfied with their level of participation? Do they perceive that their expectations have been met by LTR to date? c. School-community stakeholders: Are teachers, caregivers, principals, and the Village Education Development Committees (VEDCs) satisfied with their level of participation? Do they perceive that their expectations have been met by LTR to date? d. In terms of engagement of stakeholders, what additional/modified support is required, if any, to accelerate the potential for Activity sustainability? EQ4 LTR partner activities 4. What aspects of other development partners’ supported programs, such as the World Bank second Global Partnership for Education (GPE II) and the Basic Education Quality and Access in Lao PDR project (BEQUAL), are most complementary to, and support of the goals and objectives of LTR? a. Among existing LTR partners, if USAID were to support any additional program components, which are most compelling and why? That is, which are most able to provide credible evidence of: (1) effectiveness at supporting the development of reading skills for children at pre-primary, grade 1, and grade 2 levels; and (2) organizational/programmatic strengths that could be leveraged during the remaining period and/or beyond? The ET learned during the initial kick-off meeting with USAID and SC that the baseline assessment, including an Early Grade Reading Assessment, was delayed as a result of budget constraints and COVID￾19 lockdown. Due to lack of baseline data and targets, the midterm evaluation EQ1 was not designed to assess learning outcomes or performance indicators. Rather, the EQs were designed to be qualitative in nature and to assess perceived progress achieved to date through a formative assessment relying on the expertise of SC, implementing partners, the MoES and other stakeholders to gain a better understanding of the challenges and achievements experienced to date in order to inform future programming for the remainder of the Activity. Triangulation of data from USAID, LTR, MoES, PESS, DESB and school stakeholders and analysis by ET experts strengthened the rigor of findings, conclusions, and recommendations. 5 | FINAL EVALUATION REPORT USAID.GOV EVALUATION DESIGN AND METHODOLOGY EVALUATION DESIGN Due to the COVID-19 context and lack of available baseline data, the evaluation team (ET) utilized a qualitative evaluation design. The methods of data collection included semi-structured key informant interviews (KIIs) to gain primary data in response to the key EQs and desk review of relevant documents to obtain secondary data to either validate or identify responses to EQs. DESK REVIEW Prior to data collection, the ET conducted a desk review of relevant project documents provided by USAID to gain an understanding of LTR’s overarching results framework and theory of change, key accomplishments, challenges encountered, progress to date, and engagement with the MoES and other stakeholders. The key project documents included: the Activity Monitoring, Evaluation and Learning Plan (AMELP), Annual Workplans for Years 1 and 2, Annual and Quarterly Progress Reports, and a Data Quality Assurance (DQA) Report, which was conducted by SI. The following gender background documents were reviewed: the final version of the USAID LTR Gender Equity and Social Inclusion (GESI) analysis and the Gender Inclusive Development Action Plan (GIDAP). Additionally, for EQ4, the ET reviewed project documents from development partners in order to identify areas of overlap with LTR activities. DATA COLLECTION Data was collected over the period of July 5 through July 31 across two provinces, four districts and 10 schools. Several additional interviews related to EQ4 took place in August and September. The three sites targeted for data collection were Vientiane Capital, Vientiane Province (VTE) and Champasack Province (CPS). In the north, Vientiane was selected over Xiengkhouang (XHK) because it received the full dosage of activities, while implementation in Xiengkhouang was disrupted5 due to the National Sports Games in 2020, which could impact the results of the Activity to date. In the south, Champasack province was selected instead of Attapeu province since it covered the widest range of activities, including the Inclusive Education (IE) pilot. VTE and CPS provinces were further selected to assess the differences in implementation of SC in the north and RtR in the south. The evaluation took place across four districts with high proportions of non-Lao-Tai speaking students, including: Phonhong and Vangvieng in VTE and Bachiang and Mounlapamok in CPS. A purposeful sample of 10 schools drawn from all 854 LTR schools were selected to participate in the LTR midterm evaluation. The schools were distributed equally with five per province and two to three per district. SCHOOL SELECTION CRITERIA The ET in collaboration with USAID and SC identified 10 sampled schools according to the following selection criteria: Group 1 schools, high proportion of non-Lao-Tai speaking student population, and rollout of activities under all three results plus any pilot interventions (e.g., reading festivals in Vientiane 5 In Xiengkhouang, the Provincial Education office brought the school year forward to May to November and December to May (with a break in February-March), so LTR implementation was disrupted to some extent. 6 | FINAL EVALUATION REPORT USAID.GOV and inclusive education pilot6 in Bachieng, CPS). Group 1 schools were selected to avoid overlap with the baseline survey, which is taking place in Group 2 schools. The most important criteria were Group 1 schools and a high percentage of non-Lao-Tai speaking students. The majority of students in nine out of 10 schools selected were non-Lao-Tai speakers, ranging from 74 to 100 percent of all students enrolled. KEY INFORMANT INTERVIEWS The ET interviewed a total of 95 stakeholders from central, provincial, district, school, and community levels. The team developed 14 data collection tools tailored to the specific respondent type. See evaluation matrix in Annex A for the stakeholder groups targeted for each EQ. At the national level, the ET interviewed key stakeholders from USAID, SC, RtR and HI, the MoES, and development partners (UNICEF, BEQUAL and the World Bank). At the provincial level, the ET interviewed Provincial Education and Sports Service (PESS) and LTR staff. At the district level, the ET conducted interviews with the District of Education and Sports Bureau (DESB) technical officers and both primary and ECE pedagogical advisors (Pas) as well as LTR staff. Within 10 sampled LTR target schools, the ET interviewed school principals and teachers in pre-primary, Grade 1, Grade 2, and multigrade classes. At the community level, interviews were conducted with 10 parents or caregivers (6 mothers, 1 grandmother and 3 fathers) and 10 Village Education Development Committee (VEDC) members, who were the education focal points for the village. Prior to conducting the KIIs, the ET obtained consent for stakeholders’ willingness to participate in the study and for recording the interview. The interviews were recorded electronically to ensure accuracy during notetaking and then translated from Lao into English. SAMPLING The ET applied a purposeful and convenience sampling technique for selection of key informants. The KII sample was generated from a contact list provided by SC and approved by USAID. The list encompassed the key stakeholders from Implementing Partners (Ips), MoES, PESS, DESB and schools that are working directly with LTR in Activity implementation. The ET and SC purposively selected individuals who have been with the project since the inception and have a deep understanding of project activities related to the three key result areas. In terms of teacher selection, the ET selected teachers who were trained in Year 1 and have had an opportunity to implement the LTR reading instruction with coaching and support from the principal, pedagogical coaches and/or LTR staff. The selection criteria for recruitment of parents and caregivers included: 1) they have a child enrolled in Pre-Primary (PP)-G2 level; 2) they are involved in community engagement activities (e.g., reading festivals and CE poster parent orientation meetings); and 3) they represent an ethnic or vulnerable group (child with disabilities) in the village that LTR is targeting. Gender was another factor taken into consideration. The team attempted to select five mothers and five fathers equally distributed throughout both provinces. However, the ET learned during the recruitment process that in some villages only mothers participate 6 In the Bachiang district of Champasack province, LTR in collaboration with the MoES is implementing an Inclusive Education (IE) pilot, in which the teachers are receiving more intensive coaching by the Gender Equity and Social Inclusion (GESI) team. 7 | FINAL EVALUATION REPORT USAID.GOV and in others only fathers, so it was not feasible to achieve a gender balanced sample (See Table 2 for final sample). In order to encourage recruitment of non-Lao-Tai speakers, the ET had local interpreters on staff to assist with facilitating the interview in the parents’ native tongue. Three of the 10 parents interviewed were non-Lao-Tai speakers. In the end, it was very difficult to recruit parents who met all criteria and thus parents were mostly selected based on convenience and availability. The ET also interviewed one representative of the VEDC in each school community. The ET purposively selected the education focal point since they work with communities to conduct outreach. It is important to note that all education focal points were male. Table 2 (on next page) illustrates the sample size for data collection by respondent type and sex. In total, 95 individuals, including 22 at the national level, 20 from PESS/DESB, 10 principals, 21 teachers, and 20 community members participated in KIIs. In terms of gender equity, 54 percent were male and 46 percent female. There was a greater representation of women in the MoES, particularly at the provincial and district level. However, at the school and community levels, males dominated leadership positions (e.g., school principals and VEDC), while the majority of teachers were female (76 percent versus 24 percent for males). Where the ET thought it could work outside the existing gender structure and recruit a more gender balanced sample, it attempted to do so, for example, with recruitment of parents and caregivers. The final sample was seven males and three females, and it too was dependent upon the gendered participation of parents in schools visited and availability of stakeholders. 8 | FINAL EVALUATION REPORT USAID.GOV Table 2: Sample Size and Key Stakeholders Stakeholder Group Total Respondents Males Females USAID 2 0 2 MoES Central 8 4 4 Implementing Partners 12 6 6 Development Partners 2 0 2 PESS 4 1 3 DESB 16 7 9 School Principals 10 10 0 Pre-primary Teachers 5 0 5 Grade 1 Teachers 6 2 4 Grade 2 Teachers 6 3 3 Multigrade Teachers 4 1 3 Parents 10 7 3 VEDC 10 10 0 Total 95 51 44 DATA ANALYSIS During fieldwork, the ET analyzed emergent findings through holding two Findings, Conclusions and Recommendations (FCR) meetings. The first FCR meeting occurred at the mid-point of data collection and the second after all interviews were completed. Upon completion of fieldwork, the Team Leader (TL) reviewed the preliminary data to identify response themes and patterns, which were then used to generate a codebook. All primary qualitative data were entered and coded using the analysis software Dedoose. Qualitative data were disaggregated by respondent type, location, sex, and language spoken. The TL conducted an Out-brief Presentation on August 26 to USAID and LTR stakeholders, which was prepared based on preliminary findings from the FCR meetings and 25 percent of coded data. After all data was coded into Dedoose, the TL reviewed the coding for accuracy and completed the full analysis of findings. Triangulation was utilized to test the validity of findings through analysis of data from multiple sources. The excerpts of notes for each EQ were also reviewed to extract relevant and powerful quotes from credible and noteworthy sources. The final findings, conclusions and recommendations were derived 9 | FINAL EVALUATION REPORT USAID.GOV from analysis using the FCR matrix to ensure alignment and that recommendations are rooted in evidence from the findings and conclusions. A final presentation of findings was presented to USAID and LTR stakeholders on September 24. The FCR table, evidence of key findings, along with feedback from USAID/SC stakeholders during the final presentation were incorporated into this final midterm evaluation report. The report includes both the frequencies of responses for key questions as well as supporting evidence. Note that the most relevant stakeholders were asked each question (See Evaluation Matrix in Annex A); thus, the sample sizes vary per EQ. The sample size (N) is reported along with findings throughout the report and reflects the total number of stakeholders who accurately responded to the question. Questions that were skipped, misunderstood, or followed by irrelevant responses were excluded from the sample to provide a clear and evidence-based response to each EQ. LIMITATIONS The main limitations to the evaluation include: Remote fieldwork: KIIs in Lao PDR were conducted remotely, over the phone or using Microsoft Teams, which poses a risk with interpreting questions, responses, and non-verbal cues. Because we were unable to meet with stakeholders in person and assess their understanding of the questions and body language during the interview, the ET decided to send the questionnaires to stakeholders in advance so that they would have an opportunity to review and reflect on the questions. This approach significantly aided the interviews, particularly at the central level who were well prepared for the discussion. At the district and community level, evaluators thoroughly explained and rephrased questions as needed. Selection bias: Sample selection was based on consultation with SC. This can lead to selection bias, as stakeholders might be more likely to guide evaluators towards those who have had positive experiences with the Activity. The ET took steps to mitigate this bias, such as triangulation-- analyzing findings from multiple sources of data to arrive at balanced findings and conclusions. The ET also made the final selection of schools from a larger sample that met the selection criteria. The ET further provided criteria for community stakeholder selection, which reduced selection bias. Small sample size: Due to the COVID-19 travel restrictions, the evaluation team had to shift from planned school visits to remote-based interviews in effect reducing the sample size from 200 to 95. The ET also had to replace Focus Group Discussions planned with caregivers and VEDCs with KIIs, averaging one parent and one VEDC representative per school. This resulted in a much smaller sample size for community representatives (10 parents/caregivers and 10 VEDC members). However, it is important to bear in mind when reviewing the report that all stakeholders were asked the community engagement component questions and thus the CE findings and other key findings are credible reflecting the viewpoint from all respondents from the MoES down to the community level. The ET performed both quantitative and qualitative analysis of data to provide readers with key findings based on total respondents as well as excepts from parents and VEDCs to ensure their voices were well-represented. Additionally, selection criteria aided in ensuring a diverse and purposeful sample of VEDC and caregivers, taking ethnicity, gender and roles played in the community into account. Lack of quantitative data: As noted in the evaluation design section, the midterm evaluation does not include test results of learning outcomes or indicator monitoring data from which to assess progress from baseline to midline. At the time of inception of the midterm evaluation, there was no baseline data 10 | FINAL EVALUATION REPORT USAID.GOV available, including values for indicators or EGRA results, largely due to COVID-19 school closures and budget constraints in Years 1 and 2 of the Activity. Additionally, due to the planned upcoming baseline assessment and school closures, it was not possible to include any other testing measures. Rather the midterm evaluation relied on participant self-reporting of learning outcomes observed and perceived progress achieved to date. To address any biases this may create, the ET employed rigorous coding and triangulation of data between various questions and respondent groups. The evaluation tools also included questions to identify anecdotal evidence or school-based testing of students’ reading skills. We have included anecdotal data in the report based on teacher formative assessments or observations, but the data is not standardized given schools have been closed and exit exams did not take place in 2021. The upcoming baseline EGRA, if implemented according to plan, will help to fill this gap. FINDINGS AND CONCLUSIONS EQ1: ACHIEVEMENT OF RESULTS How likely will the three key result areas, enumerated in the results framework (Figure 1), be achieved within the set timeframe? a. Which components of the three key result areas are perceived to have been most effective/successful during the implementation to date? What is driving these successful components? b. Which components, of the three key result areas, are perceived to have been least successful to date? What is hindering these components? EQ1 FINDINGS Likelihood to achieve results Under ideal circumstances, the ET would have utilized baseline data, targets and Activity monitoring data to assess progress achieved to date against result indicators and early grade reading outcomes. As noted in the evaluation design and limitations sections, Monitoring, Evaluation, and Learning (MEL) data were not available at the time of data collection. The report of the LTR DQA7 conducted in January 2021 noted significant issues with LTR’s assessed indicator data. LtR has been working to address the issues identified in the DQA. For example, the MEAL team developed an Activity MEAL Basics training package with plans to roll out, beginning with RTR Activity staff, in April 2021.8 The AMELP9 is still in draft form and does not include targets for indicators requiring baseline data. The Activity faced many challenges with recruitment of Monitoring, Evaluation, Accountability, and Learning (MEAL) staff and collection of baseline data, including a restricted budget as a consequence of Lao PDR’s Tier 3 status in the United States (US) State Department Trafficking in Persons (TIPS) report and COVID-19 school closures and travel restrictions10. LTR has initiated the baseline assessment in the northern region, and then paused due to COVID-19 lockdowns. Thus, the evaluation questions were designed with these challenges in mind and the findings for EQ1 are derived principally from qualitative data, namely KIIs with IPs and government stakeholders, including MoES, PESS, DESB, and school staff, and with inputs from LTR on activities achieved to date. 7 LTR Data Quality Assessment Report Final. (January 2021) 8 USAID Learn to Read Year 3, Quarter 2 Report (January 1-March 31, 2021). 9 USAID Monitoring, Evaluation and Learning (MEL) Plan. February 2021 10 USAID Learn to Read Annual Work Plan October 1, 2020 to September 30, 2021. August 2020 11 | FINAL EVALUATION REPORT USAID.GOV LTR is perceived as likely to achieve results for most components if the COVID-19 situation improves Based on qualitative data gathered, the ET concluded that if schools are able to re-open, LTR will be able to make progress on most programmatic components. However, if schools remain closed, it will be more challenging to complete teacher-trainings or organize community engagement activities. Among 26 respondents, over half (62 percent) believe the Activity is "Likely" to achieve results across the three key result areas but noted that it is contingent on the COVID-19 situation. IPs strongly believed that it is “Very Likely” that they will achieve all activities enumerated in the results framework if schools re-open and the COVID-19 case count declines. Only one principal stated that the Activity was unlikely to achieve results because in his opinion the materials might not be replaced if they are lost or stolen. For instance, in some schools, storybooks have been stolen or not returned. Result 1 is on track to achieve results, but a shortage of materials remains According to IP and MoES stakeholders interviewed about the status of activities, Result 1 is on track to achieve results despite setbacks in Year 1 due to the TIPs budget freeze and COVID-lockdown in Year 2. In fact, the initial challenge of developing materials with limited funds and time constraints was recouped during the COVID-lockdown when the budget freeze was lifted, and the team was able to fully devote their time to completing the development of storybooks and supplementary learning materials. USAID and SC respondents reported that all materials in the original plan have been developed and distributed for PP and G1. G2 materials were in the process of being printed and distributed at the time of data collection. Materials reportedly developed to date include: 16 storybooks for PP, 20 storybooks for G1 and 15 for G2. LTR has also developed materials to support children with visual impairments and non-Lao-Tai speakers that align with the new curriculum. Teachers reported receiving the following teaching and learning materials: teaching guides/handbooks, supplementary reading books for G1 and G2, supplementary teaching materials, storybooks for the reading corner, a book bag, and a notebook to record book borrowing. Teachers also reported receiving various teaching and visual aids, including: a set of consonant and vowel cards, pictures, vocabulary words with pictures, visual impairment kits, number cards, colored pencils, caregiver engagement poster, and a speaker. The teaching and learning materials (TLMs) have had a positive impact on teaching, learning and parent engagement, particularly, the storybooks, reading corners and pictures with vocabulary words. Reading corners attract more students and teachers to use the resources and enthusiastically engage in reading activities. Storybooks and pictures facilitate learning because students can see the real objects and better comprehend the vocabulary word in Lao. Alphabet cards assist with word identification, comprehension, and writing; and consonant and vowel cards help students to construct and read words as well as write. Challenges The most common challenges reported with using the materials produced under Result 1 were: difficulty using the materials, particularly in multigrade classes; insufficient materials and storybooks for each student; difficulty arranging reading corners in multigrade classes; and no coaching or materials for pre￾primary classes for some of the teachers and trainers interviewed. 12 | FINAL EVALUATION REPORT USAID.GOV According to teachers and trainers interviewed, the teaching materials are challenging to use and teachers need support from principals, trainers, and coaches. Teachers also state that there are insufficient materials; and it is difficult for them to create materials on their own, such as when there are not enough flashcards or teaching aids for the whole class. According to PESS, there have been cases where DESB reported incorrect numbers of classrooms or students resulting in a deficit in some schools. In these cases, the project was notified, and materials requested. MoES and SC staff do concur that there is a limited budget to produce sufficient materials for all students and schools.11 All five pre-primary teachers interviewed reported challenges with Result 1. Pre-primary teachers who also teach G1 or multiple grades lamented that they had not received any support from the project with their classes or any storybooks. They are photocopying and creating materials on their own replicating G1materials for use in PP classes. One PP teacher reported stolen materials, while two PP teachers reported insufficient materials and storybooks. One PP teacher in Bachieng CPS stated that they have not received any materials or support for pre-primary and so she is borrowing storybooks from the G1 class. Considering the ET met with teachers in Group 1 and that all materials for pre-primary and G1 have been distributed, it is likely that there are some PP classes were not included in the original distribution, possibly due to records not being updated. Most Effective Component Teacher training was considered the most effective component due to a comprehensive teacher professional development system The ET found that teacher training was the most effective component, for a number of reasons Firstly, the teacher professional development system established by LTR has been very effective. Training master provincial trainers, district trainers and principals has created teams of trainers within provinces equipped with the knowledge and skills to support teachers. Classroom observations and coaching have helped teachers to overcome challenges and gain confidence in applying the new teaching methods. Monitoring and coaching visits are leading to improved collaboration and communication between teachers and DESB officers/PAs. To date, all teachers in pre-primary, Grade 1 and Grade 2 have been trained from Group 1 and Group 2 schools. LTR is in the midst of planning the rollout of training for Group 3 teachers and master trainers have been trained. All teachers in PP, G1 and G2 have received coaching and are participating in Teacher Learning Circles (TLCs) led by the principal. DESB officers, PAs and school principals who serve as coaches reported significant improvements in teachers’ instruction and effective use of TLMs. Prior to the LTR activity, teachers used outdated teaching methods with no materials or teaching aids. They applied rote learning techniques (e.g., lecturing, copying from the blackboard, not interactive, long sentences, etc.). Now, due to LTR trainings, they are well informed of appropriate teaching methods and know how to adapt lessons for the early grades, non-Lao-Tai speakers, and students with learning difficulties. According to nearly half of respondents interviewed (49 of 81), teacher training was the most effective component. This was followed by materials development (28 of 81). A significant percentage of 11 According to SC, at the time of the evaluation, 242 (Group1 and 2) pre-primary classes had received 8 copies of 16 storybooks and 3 copies of 3 Reading Readiness Program storybooks, totaling 31,205 storybooks. 13 | FINAL EVALUATION REPORT USAID.GOV respondents stating teacher training was the most effective were at the school level (43 percent; 17 of 81) and from CPS (65 percent; 17 of 26 provincial and district respondents). Figure 1: EQ1 Most Effective Component Many respondents stated the combination of training and materials is the most effective LTR teacher professional development strategy. LTR TLMs, storybooks and reading corners are creating a more interactive, child-centered, and enjoyable learning environment. Teachers use the TLMs (e.g., pictures, flash cards, storybooks) and active-learning techniques to engage students (e.g., storytelling, role-playing, peer learning and air-writing). The school principal is also a driving force since they provide coaching and organize TLCs. TLCs were noted as useful for teachers to share experiences and learn from one another. The training, coaching and peer learning is increasing teachers' confidence to teach non-Lao-Tai speakers. A little over one-third of respondents (35 percent; 28 of 81) stated materials development was the most effective component. Supplementary TLMs, storybooks and reading corners were all cited as very effective, particularly for supporting non-Lao-Tai speakers. The visual aids, picture books and flashcards are helping non-Lao-Tai speakers to improve oral comprehension and vocabulary skills. Result 1 was also considered most successful in terms of utilization, sustainability and scale up. For instance, the MoES has adopted materials and published storybooks online in the MoES Khang Panya Lao teaching and learning platform for continued access, printing, and dissemination. 35% 49% 20% 20% 5% 0% 10% 20% 30% 40% 50% 60% R1: Materials Development R2: Teacher Training R3: Community Engagement All Components Equally Effective Both Materials and Training Most Effective Component “Overall, I have a deeper understanding of the teaching method. I know how to communicate with a non-Lao speaker. I am confident that I can teach them although I don’t speak their language.” CPS Mounlapamok G2 Teacher 14 | FINAL EVALUATION REPORT USAID.GOV Mainstreaming inclusive education (IE) into materials and teacher training has improved teaching and learning of non-Lao-Tai speakers and children with mild disabilities Based on interviews with GESI experts and IE MoES staff, LTR has successfully integrated inclusive education into all three components. For Result 1, they developed an inclusive education checklist to ensure materials are sensitive to gender, ethnicity, and disability. One achievement is that schools now have storybooks where People with Disabilities (PWDs) are lead characters. For Result 2, the GESI team ensured all training sessions provide teachers with examples of how to make teaching more inclusive. Furthermore, LTR is implementing an IE pilot in 46 schools in Bachieng, CPS, where teachers receive training in IE approaches and more intensive coaching from the GESI team. As a result of the above IE initiatives, the MoES Inclusive Education and Community Promotion Department and teachers have reported improvements in teaching and learning. Teachers are now using peer learning techniques to pair non-Lao-Tai speaking students with bilingual students or high achieving students with low performers to support them in learning Lao. Girls and boys now sit together at tables and learn from each other, rather than being separated, as in the past. The pictures, flashcards and other visual aids have greatly supported non-Lao-Tai learners and students with visual impairment to improve their vocabulary and word reading skills. Some teachers also use the local language in the class or provide extra instruction to support non-Lao-Tai speakers with learning Lao. Although there is no reliable standardized assessment data to assess reading outcomes as a result of these new pedagogies, teachers and MoES report that they have observed improved oral language skills among ethnic students. A G1 teacher in Mounlapamok, CPS stated that after four months of pairing non-Lao-Tai speakers with Lao speakers, non-Lao-Tai speakers are able to speak Lao well. Another G1 teacher in VTE has seen improvements in her ethnic students’ ability to read letters, consonants, and words. According to a DESB officer in VTE, students from other ethnic backgrounds (Hmong, Khmer) who used to be silent are now more active even if they are using their local language; they are beginning to speak more, and their classmates help them translate words to Lao that they identify in the pictures. Multigrade teachers have also been better able to manage their classes; however, the MoES notes that they are still mastering the techniques and need more support from the project. Least Effective Component Community engagement is the least effective component to date The evaluation strongly pointed to community engagement as the least effective component of LTR to date. The main challenge reported by one-third of respondents was that parents, especially in rural and remote areas, do not have time to participate in school activities or teach their children at home because they are too busy working on the farm. One-third of respondents also stated that parents often stay overnight in the fields and do not send their children to school during the harvesting season. This key finding applies to both CPS and VTE provinces. 15 | FINAL EVALUATION REPORT USAID.GOV Figure 2: EQ1 Least Effective Component Among 57 stakeholders who provided a response to this question, over 70 percent (42 of 57) stated the CE component is the least effective. At the district level, over half of respondents who believed CE is the least effective component were from CPS (56 percent; 15 of 27). The challenge for parent participation is greatest in rural and remote districts. For example, in Don village, in the agricultural district of Bachieng (CPS), 98 percent of the villagers work from 3 a.m. to 9 p.m. in rubber plantations. According to a Bachieng multigrade teacher, for this reason, they have no time to read to their children when they come home. In CPS, there are also lots of cassava plantations in which parents work during the day, and by the time they come home, they are too tired to help their children with homework. In urban areas, work schedules also conflict with school activities. Some parents work in hotels, as traders, or as government officials. Thus, they are unable to attend events during the day. To address the urban/rural challenges, some teachers and principals have tried to organize meetings at different times around parents’ schedules, or they try to meet one-on-one with parents at their convenience when they are not working in the plantation. For instance, teacher parent meetings can take place during the day when there is a new moon or full moon (occurs about once a month). Since parents do not go to the fields on these days, they would likely avail themselves to visit the school and participate in reading festivals, reading corner activities or teacher-parent meetings. However, attendance data for parent teacher meetings shows that of those schools that organized parent teacher meetings, on average 9% 7% 74% 11% 0% 20% 40% 60% 80% Materials Development Teacher training Community engagement None Least Effective Component “The main constraint for engaging parents is their lack of time, not their lack of cooperation. I believe they would want to participate in reading more if they have time. I know they see the importance of education and it’s why they send their kids to school.” CPS Bachieng Multigrade Teacher 16 | FINAL EVALUATION REPORT USAID.GOV 80% of students had a caregiver attend. Specifically in Bachieng, where 35 of the 46 schools targeted in Year 3 held meetings with caregivers, 90% of students had a caregiver attend.12 The third most common challenge affecting community participation is lack of understanding about the importance of education reported by 18 percent of stakeholders (10 of 56), including PESS, DESB, school principals, parents and multiple VEDCs in VTE and CPS. According to a CPS Bachieng DESB officer, there has been progress, but they have not yet achieved 100 percent enrollment (estimating about 60 percent to 70 percent). While some parents understand the importance of education, it is quite common for rural parents to not be able to fully support their children’s education due to economic circumstances. Another key challenge reported by 11 percent of respondents (5 of 45) is COVID-19 school closures, which has disrupted teaching and learning and delayed reading festival activities. These setbacks have hindered parents' participation in school and LTR project activities (teacher-parent conferences, reading corners, and reading festivals). Reading festivals have only been implemented in Group 1 schools in the northern provinces of Vientiane and XHK. Furthermore, fear of COVID-19 has prevented some parents from sending their children back to school and so they remain with their parents in the field. This further exacerbates the gap in community participation in school activities and impacts LTR outcomes. Lack of training for parents and VEDC, and a clear community engagement strategy are critical factors affecting achievement of Result 3 Despite heavy workloads and time constraints, a major factor contributing to lack of community engagement is that there has been no training for parents about the LTR activity from the school or directly from LTR project staff. LTR has trained teachers and school principals on reading corner orientation and the CE poster providing tips on how to support reading at home using storybooks/reading corners even if parents cannot speak or read Lao. Whether schools conduct this orientation and training with parents and VEDCs is dependent upon the school since LTR and DESB officers do not train VEDCs or parents directly. In CPS, PESS and pre-primary schoolteachers stressed that some parents do not understand the importance of education and the objectives of the project because they have not been sensitized and trained by LTR. PESS as well as other LTR stakeholders, including VEDCs and POs, believe that because there is no training, they are unaware of the importance of education/literacy and prioritize taking their child to the farm over sending them to school. Respondents believe they need greater awareness and alternatives to taking their children to the farms but are uninformed. Perhaps, training has not taken place due to COVID-19 or not all parents have participated in the trainings because of their busy schedules. One school principal requested a follow-up training on CE to more clearly learn how to engage parents and VEDCs stating that the training received was not clear. No clear strategy for engaging VEDCs and community members was also noted as a limiting factor compared to other results. According to a LTR project officer, the challenge for Result 3 is that the VEDC doesn’t have a clear understanding of the objectives and goals of the project. LTR has never held an official 12 Quarter1 2021 LTR Report. Note: data considers a ratio of total parents attending to total number of students but does not account for the fact that multiple parents/caregivers may attend for one student, while another student may have no parent/guardian in attendance. 17 | FINAL EVALUATION REPORT USAID.GOV meeting with them since the inception of the project. Instead, the LTR CE strategy is to work with school principals, who then coordinate with VEDCs and parents. The 20 percent of respondents (16 of 81) who stated that Community Engagement was the most effective component based their response on a pilot involving Youth Literacy Champions (YLCs). There were 17 YLCs recruited (4 in VTE and 13 in XKH) through the SC Hong Kong cost-share. They were very effective in facilitating reading festivals, teacher-parent meetings, and reading corner activities. EQ1 CONCLUSIONS Based on the above findings regarding the effectiveness of the LTR Activity and likelihood to achieve results, the following conclusions can be drawn: ● Despite funding delays and COVID-19 school closures, LTR has reportedly achieved its expected deliverables for Materials Development Component and Teacher Training Component to date and are on track to realizing results by the end of the term. ● The most effective LTR strategy for achieving reading outcomes is a comprehensive teacher professional development approach that combines quality teaching and learning materials with phased teacher training, classroom coaching, and a school-based support system for continuous reinforcement, growth, and professional development. ● The community engagement component has been constrained by delays in implementation due to COVID-19, the lack of a clear and well-communicated CE strategy, and the indirect communication and outreach approach of working through schools to reach VEDCs and parents. Based on the findings, it appears schools have not conducted trainings and orientations with parents and VEDC members, or that if they have, sufficient notice and communication of the events were not provided to all stakeholders. Parents’ time constraints are another factor hindering this component. Without LTR or DESB monitoring CE activities and providing follow-up, it is difficult to ensure activities are being executed according to the plan. Therefore, Result 3 lags behind the other two key results. EQ2: IMPLEMENTATION FACTORS What (and how) are internal operational processes, - other than MoU process, (i.e., funding, management/operations, changes of context, etc.) supporting and/or hindering implementation? a. What external/internal factors, if any, have impeded the implementation of the project? b. Are there any external/internal factors that have supported the implementation of the project (e.g., management/operations)? If yes, what, and how? c. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed and/or improved to increase the likelihood of sustainability? EQ2 FINDINGS Impeding External Factors COVID-19 is the most significant external barrier impeding project implementation COVID-19 is widely acknowledged to be the most significant external barrier impeding LTR implementation. Over 90 percent of respondents (74 out of 81) reported the most significant external 18 | FINAL EVALUATION REPORT USAID.GOV barrier impeding LTR implementation is COVID-19. The respondent breakdown by gender is 49 percent (36) female and 52 percent (38) male. Figure 3: EQ2 COVID Impacts COVID-19 impacted teaching and learning, according to nearly half of respondents (47 percent; 35 of 74), among which two-thirds were female. Schools were closed due to the COVID-19 lockdown from mid￾March to June 2020; and then again, towards the end of April 2021 until present. School closures disrupted teaching and learning schedules in the second semester. PESS and DESB officers in CPS reported that teachers were unable to complete 30 percent of all lessons planned for the 2020-2021 academic school year. Teachers in VTE also reported that they could not complete all lessons as planned. Midterm and final exams were postponed for Grade 1, Grade 2, and Grade 5. Teachers had to use formative assessment based on learning progress achieved to date in order to provide final grades. There were no end-of-term exams in VTE or CPS because schools feared the children would fail and not advance to the next grade. Some schools were converted to quarantine facilities further impacting student return following the initial lockdown. When schools re-opened in September 2020, some students did not return for fear of COVID￾19 or because they did not want to wear masks and follow the new protocols. LTR provided personal protective equipment (PPE) to encourage students to return to school and trained communities on how to make face masks, but many schools ran out of masks and hand sanitizer gel very quickly. The home￾made face masks reportedly lasted no more than two weeks and additional training and materials were requested by respondents during the midterm evaluation. Another impact of COVID-19 on teaching and learning was presumed learning loss due to insufficient resources at home. There were no resources for online learning or at-home learning during the 2020 and 2021 COVID-19 lockdowns. Students did not have textbooks, storybooks or other learning materials since the schools closed without warning. Students could not learn online due to lack of equipment or funds to top-up mobile phones for digital learning via WhatsApp or other applications. According to parents, students forgot what they had learned because they missed class for long periods of time and could not study at home and acquire new knowledge. Instead, they wanted to play with friends and parents were unable to support them either because of work obligations, illiteracy, or lack of resources. For the 47% 22% 11% 7% 0% 10% 20% 30% 40% 50% Impact on teaching and learning Impact on project implementation Impact on parents and students Other impacts COVID Impacts 19 | FINAL EVALUATION REPORT USAID.GOV parents who did attempt to teach their children, they stated that it was very difficult to engage their child because they lacked resources and were not trained. About one-fifth of respondents (22 percent; 16 of 74) highlighted the impact of COVID-19 on project implementation. According to respondents, the Activity cancelled school coaching and monitoring visits due to travel restrictions and social distancing guidelines. Recently trained teachers in Group 2 were not able to apply what they had learned for more than a few months and thus will likely require refresher trainings and intensive coaching. Due to COVID-19, the project was unable to complete activities as scheduled and was forced to postpone teacher trainings, reading festivals and other community engagement activities in the South (e.g., CPS). In-person meetings were not possible, so the LTR team transitioned to virtual meetings using Microsoft Office and conducted a Master Training for Group 3 virtually. Overall, 11 percent of respondents 8 of 74) stressed the impact of COVID-19 on parents and students. Because of school closures, parents had to take their children to the fields. An unintended outcome is that children became accustomed to going to the farm and did not want to return to school. Parents were also hesitant to send their children back to school for fear of COVID transmission. Other COVID-19 impacts reported by seven percent of stakeholders (5 of 74) included financial constraints due to loss of income. For example, volunteer teachers who are non-salaried employees were not paid during school closures. Impeding Internal Factors Shortage of staff and coordination of monitoring visits are internal challenges that have affected implementation According to 50 percent of stakeholders responding to this question13 (9 of 18), shortage of staff and coordination of school monitoring visits are internal factors that have impeded implementation of the Activity. Due to budget constraints, the project has experienced a lack of staff on the MEAL team, in provincial and district offices, and at the community level. Stakeholders reported that monitoring staff is limited to one per 15 schools, which means they are able to visit schools once every two months as 13 Although the question on what internal factors have impeded the implementation of the project was asked to all stakeholders at the national, provincial and district level, it was a very difficult question for most respondents to answer. The final sample size was 18, including: IPs, USAID, MoES, DESB, PESS and a few school principals. IPs, who were the most knowledgeable of the internal challenges, accounted for almost half of respondents (44 percent; 8 of 18). Innovative Teaching and Learning Approaches during COVID-lockdowns In collaboration with DESB, one primary school in Bachieng, CPS was able to continue teaching and learning during COVID-19 school closures using two effective approaches. First, teachers gave homework packets and at-home learning materials to students in Pre-K through G2 with a deadline to return it to school. Students were asked to read stories and write their vowels and consonants. As a result, they were able to complete the curriculum on schedule. Secondly, older siblings who could read Lao helped pre-primary and G1 students with reading and learning. The benefit of engaging older siblings and peer support is a recommended strategy in the GIDAP for addressing the needs of non-Lao speakers as well as an effective strategy implemented by UNICEF and other development partners. 20 | FINAL EVALUATION REPORT USAID.GOV opposed to on a monthly basis. SC requested additional project officers in the districts and were able to recruit 10 additional POs with funding from cost-share. Since then, activities have been running more smoothly in these districts. However, there are still some provinces that have had difficulty recruiting additional staff because of Memorandum of Understanding (MoU) restrictions. In Attapeu province, implementers agree the workload justifies the need for more staff, but PESS is not confident in approving the request because it differs from the original staffing structure in the MoU. Other provinces interpret the MoU differently and approve the much-needed additional staff persons. As a result, some provinces are still challenged to manage the heavy workload with limited staff. At the community level, LTR has experienced challenges recruiting volunteers from the villages, which was the original CE strategy to promote sustainability. Therefore, SC had to develop paid positions with cost-share funding for YLCs who served as assistants to help monitor activities. YLCs receive a monthly stipend depending on their full or part-time role in the project. This approach has been very effective in boosting community engagement in VTE and was recommended by a number of SC stakeholders. The remaining 40 percent of respondents (7 out of 18) stated the coordination and planning of school monitoring visits could be improved to increase participation of MoES stakeholders. PESS and DESB stakeholders noted that plans for school monitoring visits are often submitted to the education office at the last minute without sufficient notice to ensure DESB officers have freed up their schedules or identified replacements. PESS must approve DESB work schedules in advance and need more lag time. Also related to monitoring, stakeholders mentioned that it is difficult to coach and monitor multigrade teachers because they do not know which grade to focus on or which classroom observation form to use. Supporting External Factors MoES support, cost-share, and collaboration with BEQUAL/GPE are external factors supporting the implementation of the Activity Three external factors supporting the implementation of the Activity according to 20 respondents14 are: MoES support and cooperation (75 percent; 15 of 20); cost share (30 percent; 6 of 20) and collaboration with BEQUAL/GPE (10 percent; 2 of 20). Among these, government collaboration was considered a major accomplishment. 14 Although nearly all stakeholders were asked what external factors are supporting the implementation of the Activity, similar to the previous questions, only 20 stakeholders were able to respond. The main respondents included: IPs, USAID, MoES, PESS, DESB and principals. “I think that their [MoES] enthusiasm and their [MoES] participation has been a real highlight that has been supporting the implementation of this [LTR] activity” SC Senior Management 21 | FINAL EVALUATION REPORT USAID.GOV One of the key achievements of the project is gaining the trust, support, and cooperation of the MoES. The MoES has been closely collaborating with LTR since the inception of the Activity and at all levels. A significant contributing factor is that education is a central focus of the Lao government. The project is well aligned with the GoL education sector plan that prioritizes literacy, early childhood education and inclusion. This alignment aids in the MoES motivation to participate in activities, especially teacher training for pre-primary, G1 and G2. The MoES is very supportive in collaborating and facilitating project activities. They participate in trainings with eagerness and enthusiasm, according to LTR and SC senior management. Donors also highlighted that the trust of the MoES with LTR staff and IPs is a key factor contributing to the success of the LTR Activity. Cost-share from SC Hong Kong is another supporting factor that helped the project immensely during the time of the budget freeze by enriching the existing budget with an additional $1 million. The funds were well spent on investments related to the IE pilot, YLCs, and hiring additional POs in 10 districts. These investments led to impressive results making IE central to the project and government; increasing community participation in districts that had youth literacy champions; and, the POs were instrumental in overseeing and implementing teacher training, coaching and other support for teachers and principals. Cost-share further covered costs of printing materials and translation for the IE pilot. Finally, a few people mentioned the harmonious relationships and outcomes of LTR’s collaboration with BEQUAL and GPE. The BEQUAL and Global Partnership for Education (GPE) collaboration has been very positive and supportive of LTR objectives. Through BEQUAL’s teacher training, it has helped enhance teachers’ skills providing a basis for LTR to build upon. BEQUAL also provided LTR with a pool of skilled trainers (See EQ4 for more details). LTR has collaborated seamlessly with GPE and MoES to produce the pre-primary reading readiness package. Supporting Internal Factors Strong LTR coordination, internal reviews and budget are positive internal factors supporting the implementation of the Activity The LTR team has demonstrated strong teamwork, strong staff capacity and solid relationships with the government. An MoES official stated that they have not received any negative feedback at all from PESS or DESB in reports or verbal communication. ET members who have conducted numerous performance evaluations in Lao PDR validated this statement, noting that they were surprised to hear no negative feedback from any of the MoES stakeholders compared to other project evaluations in which there are always a few complaints. “If we didn’t have SC Hong Kong cost-share, inclusive education wouldn’t have the profile it has now.” LTR Senior GESI Specialist 22 | FINAL EVALUATION REPORT USAID.GOV The MoES is extremely pleased with LTR staff and partners. They state that the POs are friendly, have strong capacity and are approachable. RtR is a good implementation sub-partner who works well with SC staff. One MoES ECE stakeholder stated that LTR staff working with the Ministry are well-equipped with the knowledge and skills in developing teaching and learning materials. They further stated, “the team spirit of the staff has led to a mutual understanding between the project and MoES.” Recognizing the challenges of budget constraints and limited staff, a SC senior staff member complimented the LTR team’s resilience stating the “sheer grit of senior members of the team certainly helps” to facilitate implementation. Retention of key technical staff was cited as another key factor contributing to the strength of the LTR team. Among 36 respondents who responded to the question on internal factors15, over half (56 percent) stated the main internal factor supporting the implementation of LTR is the strong LTR team. About a third of respondents (28 percent; 10 of 36) reported there is strong internal coordination between LTR staff, teachers, and principals. Examples included: good cooperation and coordination between SC in the north and RTR in the south; POs workstations in district offices, which facilitates implementation; and good cooperation between the project, teachers, and schools. According to 22 percent of respondents (8 of 36), internal monitoring and project reviews are critical supporting factors. For instance, annual and semi-annual project review meetings, school monitoring visits, performance monitoring and feedback loops all help to improve project implementation. Finally, 14 percent of respondents (5 of 36) believed the budget is positively contributing to the success of LTR. In particular, the funds for teacher per diem or training allowance have helped to facilitate teacher trainings. Travel allowances for PESS and DESB facilitate school monitoring visits. These funds are essential for implementation and sustainability of the Activity. LTR Operational Areas for improvement Three LTR operational areas suggested for improvement included: a COVID-19 response plan, improved teaching and learning, and improved communication of monitoring visits All stakeholders were asked the key question of what LTR operational areas, including: governance, management, M&E could be changed and/or improved to increase the likelihood of sustainability. According to results in Table 3, there was a total of 56 respondents who answered this question with 15 The question on what internal factors have supported the implementation of the project was asked to all national, provincial and district stakeholders. Yet, due to the familiarity of respondents with the Activity and the difficulty of the question, it yielded 36 responses. The respondent types included: IPs, USAID, MoES, DESB, PESS, school principals and teachers. “Regarding the internal factor that has been supporting the project implementation, project staff at each level are focused on their work and do a very good job!” VTE PESS 23 | FINAL EVALUATION REPORT USAID.GOV representation from all target groups. Over half of the respondents (56 percent; 31 of 56) were from the school and community levels. Thus, although some of the findings in this section parallel those in EQ3 and the sustainability section, the respondent type and sample size differ. Figure 4: EQ2 LTR Operational Areas for Improvement The most common LTR operational areas suggested for improvement were developing a COVID-19 response plan (63 percent; 35 of 56), improving teaching and learning (48 percent; 27 of 56), and improving communication of school monitoring visits. Table 3: Stakeholders’ Suggestions for Areas for Improvement Respondent Type Number of Respondent s Percentage Implementing Partners 5 9% MoES 7 13% PESS/DESB 13 23% Principals/Teachers 21 38% VEDC/Parents 10 18% Total 56 100% 63% 48% 18% 0% 20% 40% 60% 80% COVID-response Teaching and learning Communication LTR Operational Areas for Improvement 24 | FINAL EVALUATION REPORT USAID.GOV COVID-19 response plan. As illustrated above, over half of respondents recommended COVID￾response strategies. PPE is in high demand. Principals, DESB and VEDCs requested more PPE (e.g., face masks, sanitizer gel, thermometers). According to a school principal in Mounlapamok, CPS, they received one box of masks, handwashing gel and soap from district authorities. Even with using it sparingly, they ran out and only have soap left. Secondly, online learning was requested for urban schools and schools that have been used as quarantine facilities. Online learning could be introduced through platforms such as Facebook or Lines; however, this is only applicable to urban areas where they have internet connection. For rural and remote schools that do not have internet access, stakeholders suggested provision of at￾home learning materials and training parents in its usage. The greatest challenge for parents during COVID-19 has been accessing books from the reading corner and obtaining basic learning materials to teach their child at home because schools are closed. Six out of 10 parents requested stationary, more storybooks, and materials for teaching at home (e.g., flashcards, pencils, guides, textbooks, coloring books). The highest priority was to help students gain access to the storybooks and textbooks so parents can teach them. Both mothers and fathers requested that schools allow book borrowing from the reading corner during COVID school closures and that the LTR Activity provide training to guide parents on how to teach their child to read at home. Creation of a mobile library or book bank where books stay in the community and are passed along to neighbors was another recommended strategy for ensuring access to books during COVID-19 lockdowns. According to a VEDC member in Mounlapamok, CPS, schools need more storybooks, so that there are at least one per child. There is an insufficient number of books for students to take home and learn. During the COVID-19 lockdown, two households shared one book and it was difficult for families to gather for shared teaching or reading, especially for parents who go to the fields and are far from the village. These students need their own books to continue reading and learning. “Provide parents with resources for teaching and learning at home. Students need to be able to take books home and exchange them with their friends.” VTE Vanvieng Mother Training topics requested: • Assessment/scoring • Reading corners • How to organize extra reading hours • Refresher on teaching techniques • Multigrade teaching • Writing exercises 25 | FINAL EVALUATION REPORT USAID.GOV Teaching and learning. Nearly half of stakeholders (48 percent; 27 of 56) requested additional training and support for teachers. See textbox above for topics often requested. School principals have observed during TLCs that trained teachers are still not familiar with the techniques. They need a refresher training16. Stakeholders also requested more teaching tools and reference guides. For instance, one multigrade teacher proposed that LTR write a booklet/guide on the different teaching methods for reading instruction and inclusive education that they and new untrained teachers can reference in the future. Teachers, principals and MoES staff all requested refresher trainings because the trainings focused on too many topics with too little time. A MoES representative recommended to extend the time for teacher training and to focus on one particular topic over the course of a few days at each workshop. Past trainings have focused on multiple topics (e.g., assessment and multigrades teaching). For this reason, teachers have been requesting refresher training in these areas. Note that LTR has provided training on multigrade lesson planning in January 2021 and will provide a follow-up training in Year 4. PESS in VTE requested that LTR involve provincial master trainers in coaching activities. The current cascade training model is for PESS Master Trainers to train district trainers who then train and coach teachers. PESS would like the provincial trainers to visit schools for monitoring and coaching to observe the implementation of the training and assess how well teachers are performing. Improved communication and advance notice for monitoring visits. MoES, PESS/DESB, school principals, VEDC and parents all requested more advance notice for school monitoring visits and trainings in order to afford them the opportunity to participate in community-based LTR activities. At the central level, MoES requested two to three months advance notice for school monitoring visits, so that they could manage their time and participate. At the provincial level, PESS would like LTR to notify them at least one week in advance of the activity or agenda, so they can notify DESB officers and manage their schedules. PESS also suggested that field staff visit schools at least once per week and then have a debriefing meeting with the team to discuss the findings, challenges, and successes, drawing comparisons amongst the schools visited. Schools requested to be informed of monitoring visits in advance so they can ensure that students will be present. During harvesting season, there is high absenteeism since children accompany their parents to the farm and stay overnight there. It is important to inform the principal so he can invite the students to come to school for teacher monitoring visits. VEDCs would also like to be notified of the plan and any training. A VEDC member from Mounlapamok, CPS, for instance, would like to be involved in the LTR Activity and requested sufficient notice to ensure participation. The MoES recommends that LTR at the provincial, district and school level meet and develop a plan to deal with shifting responsibilities of trained teachers when they are transferred, on maternity leave or retired so that the responsibility does not transfer to the principal or another untrained teacher in Grade 3 or Grade 4. The project should make several contingency plans with the school principal and teachers 16 LTR conducted a program of refresher training during Summer 2021 and will conduct a further refresher for Group 3 teachers in Year 4. 26 | FINAL EVALUATION REPORT USAID.GOV so that when trained teachers leave, schools have a back-up plan. For example, the trained teacher and principal could train the new teacher and transfer the TLMs. SUSTAINABILITY The status of sustainability was assessed through desk review and KIIs. Through review of the Cooperative Agreement and Modification I, the ET sought to understand the originally planned activities and deliverables for sustainability. Through KIIs, the ET aimed to assess what the current status, milestones and probability of sustainability is through the eyes of IPs and government counterparts. The ET analyzed findings from both sources to identify gaps and recommendations for the remainder of the Activity. Sustainability Findings from Desk Review Sustainability of LTR largely depends on the management and implementation of the sustainability plan as laid out in the Cooperative Agreement Modification 1 Under the Cooperative Agreement Modification 1, the LTR strategy for strengthening capacity of MoES personnel at all levels to support scale up of the Activity in new schools and districts is through a phased approach to capacity development at the provincial and district levels. The cooperative agreement Modification 1 describes the phased approach as: USAID Learn to Read will be implemented in three distinct phases in each target district to develop resources (described under IR1) and approaches with MOES and non-government education partners and gradually transfer responsibility and ownership to existing systems. ● Phase 1: Testing materials and establishing proof of concept: Implementation through government systems, managed by USAID Learn to Read, in close coordination with provincial and district governments, in an initial cohort of 30-50 percent of schools in target districts ● Phase 2: Testing feasibility of government-led implementation: Co-implementation by USAID Learn to Read and provincial or district government in a second cohort of 25-40 percent of schools in the target districts Phase 3: Transfer of leadership: Implementation by provincial or district government, with guidance from USAID Learn to Read, in a third cohort of the remaining 25-40 percent of target￾district schools LTR is currently in Phase 2 of the three-phased approach to MoES capacity building, which is aligned with the implementation schedule; Phase 3 will depend on COVID-19 context and response plan Based on desk review of the February 28, 2021, USAID/LTR MEL Plan, the Overview of Learn to Read Activities at district level in 2019 to 2020, and consultations with SC, the final percentages of schools for phased implementation and their respective start dates are as follows: 27 | FINAL EVALUATION REPORT USAID.GOV At the time of data collection (July 2021), LTR should have been at the end of Phase 2 working in the second cohort of schools with joint implementation between LTR and provincial/district governments. If the sustainability plan is on schedule, LTR should be preparing for Phase 3 and the handover of implementation to provincial and district governments. Under Phase 3, PESS/DESB would provide teacher coaching for Cohorts 1 and 2 and should begin training teachers for all grades for Cohort 3 with support from LTR and Master Trainers (July-August 2021). LTR has conducted Master Training for Cohort 3, but has not yet conducted teacher training due to COVID-19 school closures. Teacher training and coaching will depend on the COVID-19 context and response plan; whether teachers may be trained virtually or through a blended approach and whether schools re-open so that teachers may continue applying what they have learned and receiving coaching during the remainder of the Activity. Meanwhile, LTR needs permission to travel and provide support to DESB and schools as they proceed with implementation. According to the sustainability plan in the USAID LTR cooperative agreement Modification 1, LTR will apply Collaborative, Learning, and Adapting (CLA) principles. These strategies include conducting a diagnostic assessment of relevant functional areas of MoES and developing a sustainability plan with benchmarks for tracking progress. During initial consultations with SC, the ET learned that there was no sustainability plan17 or diagnostic assessment. However, SC has been working in close collaboration with and under the advice of the MoES since the inception of the Activity to design, implement and monitor results, since this will need to be a key element to ensure sustainability of materials, training and approaches being used in future years. SUSTAINABILITY KII FINDINGS To further assess the status of LTR sustainability beyond the four key EQ questions and desk review, the ET in collaboration with USAID developed the following sub-questions, which were included in the data collection tools for all government stakeholders (e.g., MoES, PESS, DESB/PAs) and implementing partners. a. Are you aware of the phased approach for gradual transfer of leadership and sustainability? b. What milestones or progress has been achieved to date? c. Do you think the project will be sustained within all levels of the MoES by the end of the Activity? Why or why not? d. What is needed, if anything, to ensure sustainability of the Activity by the end of its term? 17 The sustainability plan was recently developed and submitted as part of Year 4 planning for the rollout of activities to the third cohort of schools. Thus, stakeholders were not aware of this plan as the time of data collection, which is reflected in the KII findings. Phased Approach to Implementation • Phase 1: Implementation managed by LTR in 30 percent of schools (beginning in September 2019) • Phase 2: Co-implementation by LTR and provincial or district government in a second cohort of schools (beginning in September 2020) • Phase 3: Implementation by provincial or district government with guidance from LTR in a third cohort of 30 percent of schools (beginning in September 2021) 28 | FINAL EVALUATION REPORT USAID.GOV Awareness of Phased Approach and Milestones Achieved None of the MoES, PESS and DESB stakeholders were aware of the LTR capacity building strategy, while half were aware of the phased approach to school implementation Among 20 stakeholders who responded to the first sustainability question at the central, provincial and district levels, none were familiar with the phased approach for gradual transfer of leadership from LTR to provincial and district authorities. Half were familiar with the phased approach to implementation-- rolling out the Activity to 30 percent of schools in the first year, 40 percent in the second year, and the remaining 30 percent of schools in the third year. They understood the importance of phasing teacher training for Quality Assurance (QA); and observed that training smaller cohorts was more effective than training larger groups. Figure 5: EQ2 Milestones Although only 10 were familiar with the phased approach, 15 stakeholders provided responses to the follow-up question of what milestones and progress has been achieved to date. The results in the graphic show that 40 percent of respondents (6 of 15 who responded to the question) stated the project was in Phase 3 while less than one-third (4 of 15) stated Phase 2. As the phased approach coincides with the academic school year, the project was at the end of Phase 2 at the time of data collection. While the project was ramping up for Phase 3, it technically begins in September 2021. MoES and DESB respondents were not clear on the phases and groupings of schools. Some DESB officers knew that Group 3 master trainers were being trained, but none were aware of the leadership changes in each phase. Several LTR/SC staff stated that they wished to see PESS and DESB taking ownership of activities and leading the trainings, considering this is the third group of teachers being trained on the same content. Sustainability Probability LTR materials, training and monitoring are most likely to be sustained than community involvement 27% 40% 13% 20% 0% 10% 20% 30% 40% 50% Phase 2 Phase 3 Phase 4 Other Milestones 29 | FINAL EVALUATION REPORT USAID.GOV Figure 6: EQ2 Project Sustainability at MoES Survey Response Stakeholders believe activities designed to support implementation of the new curriculum (e.g., supplementary TLMs and teacher training) will be sustained because they are endorsed by the MoES and linked to the MoES development plan. In fact,the majority of respondents (26 of 38) answered “yes” the project will be sustained at all levels of the MoES. The pre-primary reading readiness package and storybooks are also believed to be sustainable because the government has decided to integrate them as supplementary materials into the new pre-primary curriculum. LTR staff asserted that teacher training activities are potentially the most sustainable activity because the government has the budget. Other stakeholders noted that the trainings will be sustained because teachers have been trained and will continue applying the knowledge and skills they have gained. School principals assert that teachers will continue to meet and exchange concepts through Teacher Learning Circles, and it will be sustained because it is mandated as part of the new curriculum. A LTR project officer also believes that DESB monitoring of teachers will be sustained because BEQUAL has been assisting the government with budgeting for monitoring visits. DESB officers in VTE agree that monitoring of LTR￾trained teachers will likely continue, but they may not be able to travel to all target schools. Of respondents who stated that the project would likely not be sustained, one-thid pointed to MoES budget constraints and lack of community engagement as the reason. According to LTR project staff, community engagement is the most difficult component to sustain because there is insufficient budget to engage relevant stakeholders. The component relies solely on volunteers, so it’s challenging to garner their engagement in the project. There’s no allowance for participating or monitoring CE activities. Another reason based on a LTR Project Coordinator’s observation is that communities and VEDCs currently do not participate much in LTR activities; thus, the likelihood of sustainability is low. 68% 32% 0% 20% 40% 60% 80% Yes No Will the project be sustained at all levels of the MoES? “Without community involvement, it is difficult to sustain.” VTE Phonhong DESB 30 | FINAL EVALUATION REPORT USAID.GOV Figure 7: EQ2 Recommendations for Sustainability What is Needed to Ensure Sustainability Additional capacity building, training, materials, and stronger community engagement are needed for ensuring sustainability Table 4: Percentage of Stakeholders Providing Recommendations Respondent Type N Percentage Implementing Partners 9 26% MoES 7 20% PESS/DESB 10 29% Principals/Teachers 9 26% Total 35 100% A total of 38 respondents provided recommendations for ensuring sustainability of the LTR Activity. They provided smart recommendations with specific actions that could be implemented within the timeframe of the project. The percentage of respondents providing each recommendation ranged from 16 percent to 21 percent; thus, the recommendations are similarly weighted. As Table 4 illustrates, PESS/DESB represented nearly 30 percent of respondents while school stakeholders and IPs represented about a quarter of respondents. 18% 21% 21% 18% 18% 16% 0% 5% 10% 15% 20% 25% Capacity building Strengthen M&E More Materials Teacher training and PD Community engagement Budget Recommendations for Sustainability 31 | FINAL EVALUATION REPORT USAID.GOV Capacity building. Stakeholders provided several recommendations for increasing the capacity of government officials to sustain the LTR Activity. First, LTR should follow the phased approach to sustainability for the next two years to create a sense of ownership, where the government leads activities with LTR providing technical support. Specifically, IPs and MoES staff suggested LTR assign ownership of LTR activities to DESB so that they can carry out the work in the future (e.g., appoint government officials to become project officers). Secondly, the MoES recommended LTR increase capacity of government officials through study tours to other provinces (e.g., from CPS to VTE to learn about CE activities or TLCs). Thirdly, respondents suggested that LTR provide capacity building to DESB officers on monitoring and budgeting. For instance, a Project Coordinator recommended that LTR develop a monitoring manual that includes guidelines and forms for conducting monitoring visits that MoES, PESS, DESB can continue to implement once the project ends. Lastly, IP Project Managers recommended extending the RRP training to kindergarten teachers in Year 4 since most schools have kindergarten and not pre-primary. Training kindergarten teachers on RRP is included in LTR’s Year 4 work plan. M&E. Another recommendation was for LTR to strengthen the M&E team capacity and system to produce accurate data on activity indicators and outcomes. For instance, M&E staff could conduct an EGRA to assess reading outcomes, which could potentially be carried out with Group 3 pending school re-opening.18 RIES would like a detailed statistical report of progress achieved against indicators. They would like to know the achievement of each component against targets to identify gaps and the way forward once the project ends. RIES would also like an evaluation of past activities and lessons learned (e.g., final evaluation) along with recommendations for how to integrate LTR into the government’s plan. The MoES ECE Department of Kindergarten would like the project to send copies of LTR TLMs that could be used as reference during monitoring visits. The department also requested receiving quarterly and annual progress reports from Year 1. Monitoring. District level staff (DESBs and IP) requested more school monitoring visits. DESB would like LTR staff to visit schools more frequently so that they can understand the real issues and help solve problems. District level IP staff recommended increasing the number of POs and staff at the district level 18 The baseline was the original plan to gather robust evidence of reading ability through EGRA or reading assessments. However, this was halted due to ongoing COVID-19 restrictions. Capacity Building in Phase 3 VTE PESS plans to invite instructors from the University of Laos and Bankem Teachers College, who are currently BEQUAL master trainers, to become LTR master trainers. PESS is trying to scale up LTR activities and needs support with the training. This is an ideal opportunity for PESS to lead the trainings in Phase 3 while LTR provides technical assistance. 32 | FINAL EVALUATION REPORT USAID.GOV so that the project can monitor teachers at least once per month. “This will increase sustainability since teachers will have more capacity by the end of the project,” stated a LTR project manager. Materials. Regarding materials development, DESB in CPS province suggested LTR do an inventory of existing materials in LTR schools because schools are already running out of stock. Teachers interviewed would like enough materials for each student in their class (e.g., textbooks, storybooks, and teaching aids). In Vangvieng, VTE, a teacher and principal requested a refresher training on teaching and learning materials development. One principal recalled how back in 2015, RtR trained them on how to repair old books. They have applied that training and have been procuring additional books to date, but feel that they need to be re-trained. Many stakeholders (teachers, principals, DESB) requested more TLMs (e.g., storybooks, flashcards, notebooks, and pencils). VTE PESS would like the project to provide five sets of storybooks. Teachers recommended 20 storybooks per grade and developing a school library since children need a variety of storybooks to increase their motivation and reading skills. A widespread concern is that if children read the same books over and over again, they will become bored and less motivated to read. Another issue cited by a school principal is that the stories are too short, so children have quickly finished reading all the books. To address this issue, several stakeholders recommended developing a global library or building capacity of stakeholders to use Khang Panya Lao (MoES Online Teaching and Learning Platform) for downloading and printing books. One recommendation from a senior SC staff member is to encourage story writing competitions at reading festivals and publish the stories. Each year, book publishers have a gathering; LTR should attend the event and submit stories and criteria to book publishers for publishing children’s books for emergent and beginning readers. Note that USAID has successfully implemented story writing competitions and publishing of the winning stories in a children’s books that was distributed to all target schools under the USAID Dominican Republic Effective Schools Program.19 USAID has also implemented the Early Learners program engaging kindergarten through third grade students in writing their own stories in the Write to Read contest in Latin America.20 Teacher training. To further build capacity of teachers and trainers, it was recommended that LTR provide refresher training for teachers/PAs and that they train pre-service teachers from universities and teacher-training colleges to become master trainers. Stakeholders suggested LTR continue evaluating, 19 Final Performance Evaluation of USAID/Dominican Republic’s Effective Schools Program (ESP) https://pdf.usaid.gov/pdf_docs/PA00KM95.pdf 20 Writing their own stories. https://medium.com/usaid-2030/writing-their-own-stories-177fd494b9ed “I would like LTR to conduct refresher training on teaching and learning materials development for teachers in PP, G1 and G2. This is because the materials are getting old. When the project ends, teachers should be able to make their own teaching and learning materials.” VTE Vanvieng School Principal 33 | FINAL EVALUATION REPORT USAID.GOV coaching, and providing refresher training to teachers in all phases and grades until they master the content. It was requested for all groups and grades because new teachers have joined Group 1 or Group 2 schools and have not been trained. School principals requested LTR to develop a peer-to-peer training system so teachers in grades 1 and 2 can train new teachers in how to use the techniques and create materials. Teachers requested more training on book borrowing, teaching/identifying children with disabilities, and on teaching non-Lao-Tai speaking students. They recommended including parents who are literate in the trainings. Teachers would like training on how to assess students hearing/vision/cognitive skills to enable them and parents to identify children with learning difficulties or physical disabilities. Community engagement. In order to ensure sustainability, the CE strategy needs to be better integrated into MoES plans and activities so that they can take ownership. It is also critical to engage VEDC and village authorities in LTR activities so that they understand project objectives. LTR has worked with teachers and schools thus far, but now respondents feel they must focus on CE, and work with communities to mobilize funds for books, develop materials, maintain books, and increase parent engagement in reading activities. The MoES recommends LTR focus CE efforts in schools that have challenges in order to develop solutions before the project ends (e.g., schools where harvesting season causes parents to take their children to the farms; parents need alternatives and help with this decision). A school principal recommended LTR train parents on book borrowing and maintenance. For illiterate parents, he recommended training on elaborating stories with supplemental pictures. Budgeting. When planning for sustainability, LTR must consider the MoES budget. The government has many priorities and needs to weigh the costs and benefits. The MoES IE representative requested a budget planning session with LTR staff to learn more about the project and how to incorporate LTR activities into the MoES budget. This is important considering past projects have not been sustained. For instance, one stakeholder noted that Room to Read had a project to create a reading room in the past and someone to manage it; when the project ended, there was no budget to sustain the activity. Hence, there needs to be a budget to sustain the reading corners in LTR-supported schools. The schools need someone to manage the reading corners and have access to funds to buy more books. EQ2 CONCLUSIONS: INTERNAL/EXTERNAL FACTORS ● COVID-19 is the greatest ongoing challenge hindering the implementation of the Activity. It poses a threat to teaching and learning, and project implementation. ● Since students had not returned to school at the time of data collection, the true impact of school closures on learning has not been formally assessed in schools. Yet, it is presumed based on past experience that the gap will have an effect on students’ learning outcomes and the incomplete school year will need to be recouped through supplemental learning and remediation when schools re-open. “I want the activity to promote reading and providing more books. I want the books to be given to us. I want to own them. I want some books to belong to the school and some to belong to the parents.” CPS Mounlapamok School Mother 34 | FINAL EVALUATION REPORT USAID.GOV ● Although the shortage of staff is a key constraint, strong LTR teamwork and capacity help drive results. LTR’s robust relationship with the MoES at all levels and with development partners have accelerated the project’s achievement of results. ● Internal project reviews with key MoES stakeholders and strong coordination with school partners have strengthened project implementation. Meanwhile, external communication of school monitoring visits by LTR staff could be improved in order to increase participation of PESS and DESB officers. ● The USAID budget and SC Hong Kong cost-share were vital to achieving LTR results, especially following the budget freeze. EQ2 CONCLUSIONS: SUSTAINABILITY ● Awareness of the phased approach to gradual transfer of leadership to PESS/DESB is low. Findings indicate that there is confusion about the milestones and progress achieved towards capacity building to date. ● LTR has not implemented the sustainability plan, diagnostic capacity assessment or benchmarks for tracking progress, as outlined in the cooperative agreement. The sustainability plan was recently developed and submitted along with Year 4 plans. Thus, stakeholders are not clear on the status of sustainability and the ET does not have sufficient evidence to conclude that the Activity will be sustained. ● The likelihood of sustaining materials and training is high, while for CE it is low. Findings suggest the main reason is because the materials and training components are linked to the MoES curriculum and development plans, while the CE strategy is not. CE is further hampered by limited budget, resources, and buy-in from target communities. ● While the majority of stakeholders believe LTR will be sustained, findings indicate that there is a strong demand for further capacity building in M&E and data collection, strengthened community engagement, additional training and support for teachers, a sustainable mechanism for producing additional TLMs, and a comprehensive COVID-response plan. Thus, the Activity is not likely to be sustained without further investment. EQ3: LTR/MOES COLLABORATION What evidence exists, and what does it tell us, about how effectively Learn to Read has been engaging the Ministry of Education and Sports and other key stakeholders at central, provincial, and district levels to meet the Activity objectives? a. MoES: Are MoES stakeholders at the central, provincial and district levels satisfied with their level of participation? Do they perceive that their expectations have met by LTR to date? b. Subcontractors/IPs: Are program partners (Room to Read, Humanity & Inclusion) satisfied with their level of participation? Do they perceive that their expectations have been met by LTR to date? c. School-community stakeholders: Are teachers, caregivers and principals satisfied with their level of participation? Do they perceive that their expectations have been met by LTR to date? d. In terms of engagement of stakeholders, what additional/modified support is required, if any, to accelerate the potential for Activity sustainability? 35 | FINAL EVALUATION REPORT USAID.GOV EQ3 FINDINGS LTR/MoES Collaboration Evidence of MoES roles and responsibilities suggests LTR has been effectively engaging relevant departments of the MOES at the central, provincial and district levels in planning, implementation, and monitoring of the LTR Activity MoES staff have played central roles in LTR implementation since the inception of the project. LTR collaborates with the MoES at all levels in planning, implementation, and monitoring of the Activity. At the central level, MoES staff are engaged in materials development, teacher training, planning, coordination, monitoring and supervision. Specific MoES departments that LTR collaborates with for materials development and teacher training include: the Research Institute for Education Science (RIES), Department of General Education (GE), Department of Teacher Education (DTE), the Department of Early Childhood Education (ECE), and the Inclusive Education Promotion Center (IEPC). LTR works with the IEPC on implementing an inclusive education pilot in Bachieng, CPS, on multigrade teacher training, and on integrating IE into all teacher trainings. One of the biggest accomplishments of the Activity is getting IEPC staff involved in all trainings at the national, provincial and district levels to ensure inclusive teaching methodologies. In the past, each department worked in silos. Now, the IEPC is playing a central role at all levels. RIES is the primary partner for early primary education. LTR engages with RIES and GPE for developing the Reading Readiness Packages (RRP) that align with the pre-primary curriculum. LTR collaborates with RIES for facilitating trainings at the provincial and district levels; and for monitoring the teaching and learning at all levels to ensure they follow the project guidelines. LTR works with the department for Early Childhood Education for developing reading corners and other reading activities. The department of General Education is responsible for training G1 and G2 teachers. LTR collaborates closely with the GE department and BEQUAL in developing supplementary packages to support teachers with teaching non-Lao-Tai speakers and children with mild disabilities the G1 and G2 curriculum. At the provincial level, LTR works with the GE, DTE and ECE departments. LTR trains PESS officers from these departments to serve as master trainers, who are then responsible for training district trainers and teachers. Within LTR target districts, DESB officers serve as teacher trainers and coaches. DESB PAs train and coach teachers, and conduct monitoring activities. LTR project officers work closely with DESB. LTR trains school principals to serve as coaches and TLC facilitators. Teachers and principals are also trained in community engagement strategies. Teachers implement reading corners, inclusive education techniques, phonics-based reading instruction, active learning pedagogies and community engagement activities (e.g., reading corner orientation, teacher-parent conferences, caregiver engagement, etc.). It is the responsibility of the school staff to work with VEDCs and parents to implement CE activities. 36 | FINAL EVALUATION REPORT USAID.GOV LTR and MoES staff at central, provincial and district levels have coordinated and cooperated to implement activities and address gaps identified via activity implementation reports. They meet regularly to plan, review progress and address challenges in order to achieve project goals. The most effective forms of MoES engagement are: 1) alignment with MoES priorities and, 2) joint implementation of the Activity with MoES and development partners The ET found that the joint implementation strategy of LTR working in close collaboration with MoES and development partners is very effective. According to 18 stakeholders who responded to this question, the most effective approach for MoES engagement is ensuring that LTR activities are aligned with MoES priorities and respond to their needs. For instance, Result 1 and Result 2 activities are implemented in parallel with the education sector plan to improve the curriculum. Therefore, the MoES assumes great responsibility in coordinating and implementing these activities. The central government and development partners working together to develop materials and train local staff to implement and monitor activities in order to strengthen their capacity is one of the most effective of LTR’s approaches, according to stakeholders interviewed. Engaging relevant departments of the MoES in curriculum and teacher training at all three levels (RIES, department of early primary education, primary education, and other relevant sectors) has been very effective in planning and rolling out teacher trainings. For instance, the government was involved from the beginning in producing six titles for the RRP that will supplement the revised pre-primary curriculum; and, in producing teacher guides, TLMs, and storybooks for teaching the Lao subject. Coordinating with the MoES and development partners (BEQUAL/GPE) for implementation of the national curriculum has helped LTR and its partners to mutually reinforce policies, strategies, and evidence-based teaching methods. Joint monitoring visits and joint review of monitoring outcomes to determine which areas, schools or teachers need more assistance and support has been highly effective in building capacity and a sense of ownership within the MoES. LTR Stakeholder Satisfaction Nearly all MoES staff expectations were met by the LTR Activity 37 | FINAL EVALUATION REPORT USAID.GOV Figure 8: EQ3 Satisfaction with Level of Engagement Of 71 total respondents, inclusive of MoES, PESS/DESB, teachers, principals, VEDC, and parents, 93 percent (66 or 71) are satisfied with their level of engagement in the LTR activity. Overall, 100 percent of PESS, DESB, Teachers and IPs (43 of 43) are satisfied with their level of participation. As reported under findings for EQ2, an MoES official stated that they’ve received no negative feedback from PESS or DESB’s reports. DESB officers reported that the project has supported them tremendously to conduct school visits, by providing transportation allowance and classroom observation forms. At the MoES central level, all but one stakeholder was satisfied because this individual would like to be involved in provincial and district monitoring. All MoES are satisfied with their involvement in developing the new curriculum, participating in master/teacher trainings, and in observing promising results from school visits. Nine out of 10 school principals were satisfied. One principal in Bachieng, CPS was unsatisfied due to unmet expectations. He had expected G2 teachers to have been trained and monitored by now, but there were delays caused by COVID-19 lockdown. The majority of principals state they are very satisfied because all of their expectations have been met and LTR has been responsive to their requests. All 86% 100% 90% 100% 70% 50% 93% 0% 50% 100% 150% MoES PESS/DESB Principal Teachers VEDC Parents Total Satisfied with level of engagement (Yes) “I would give 100% to the project fulfilling my expectations. The content of the trainings was practical in coaching the teachers, which in turn, improved their teaching performance. The students attended more regularly. They had fun interacting with the pedagogical advisors during school visits.” CPS DESB 38 | FINAL EVALUATION REPORT USAID.GOV principals are satisfied with the training outcomes and improved reading results they have observed in the students, as a result of the LTR teacher trainings. Teachers also stated that the trainings have improved their knowledge and skills, which in turn, has improved children’s reading skills. According to a G2 teacher in VTE, they are satisfied with the frequency and quality of coaching visits. The project team comes to monitor the class every month and they provide useful suggestions on reading approaches. Note that this contradicts other stakeholders (LTR district staff) assertions that the project conducts semesterly coaching visits, and conflicts with recommendations for more frequent coaching to ensure mastery of the training methods. There was a much higher degree of VEDC satisfaction in the north versus the south Out of 10 VEDCs interviewed (5 in VTE and 5 in CPS), all five VEDCs in Vientiane were satisfied compared to one VEDC satisfied in CPS. Among the four VEDCs not satisfied in CPS, two responded “no” they were not satisfied and two were coded as N/A because they were not familiar enough with the Activity to state if they were satisfied. The reasons for not being satisfied with the LTR Activity included: 1) because the LTR Activity did not inform the village authority directly, so their expectations were not met; 2) because they had not participated in any LTR activities or could not recall if they had; and 3) because there was insufficient “I think it would be better if the village authorities in each village attend the training organized by the Activity so they can come back and disseminate information to the villagers. This would make it easier for bringing parents into the process. This is my request. Do you think it is possible?” CPS Bachieng VEDC “I am satisfied with the level of outreach and communication provided by the VEDC because they always inform the parents/villagers one day ahead. However, it would be better if they could inform 3-4 days ahead because there are many parents working in different towns, so they need time to arrange their schedules.” VTE Vanvieng father 39 | FINAL EVALUATION REPORT USAID.GOV advance notice21 provided for the trainings (only one day notice and the VEDC member was at the farm, so they could not attend). The two VEDCs in CPS who were unfamiliar with LTR were in fact familiar with a project providing masks and hand sanitizer gel to the schools but weren't sure if it was from LTR or another project. On the other hand, the VEDCs in VTE who have been engaged in the LTR project were much more active. VEDCs work and assist teachers and schools on home education issues. According to a satisfied mother in Vang Vieng, VTE, VEDCs announce activities through the loud-speaker or by calling villagers to meetings. A satisfied father in VTE also appreciates the VEDC support but requested more notice for future events (see textbox below). The ET observed that VEDCs in VTE are generally more active than in CPS. Parents are more satisfied in Vientiane Province than in Champasack Province All parents interviewed in VTE were satisfied while the majority of parents in CPS were not familiar enough with the project’s work to be satisfied or not. In VTE, four out of five parents interviewed had participated in readings festival and orientation to reading corner activities; and all five participated in teacher-parent meetings. These parents consisted of two fathers and three mothers. The mothers in VTE have been supporting their children to learn how to read at home through telling stories, practicing pronunciation of letter sounds, and developing vocabulary through reading books from the reading corners. They are actively engaged in reading activities. Among the two fathers interviewed in VTE, one reads aloud to their child, and both support their G2 child with learning how to write in the Lao language. For example, one father bought a small whiteboard for his child to practice writing and drawing. This father, who also participated in many LTR activities, thought the parent-teacher conference was the most effective activity because he could meet other parents and confirm that his daughter actually went to school. Meanwhile, the caregiver engagement activity was the least effective (see textbox above). Reading festivals in VTE could also be improved since teachers were not well organized or actively engaging parents in the activities, according to this active father in Vang Vieng, VTE. 21 Note that even though it is not LTR’s responsibility to notify VEDCs or parents of school-based activities, this finding is demonstrating the expectation of stakeholders and the lack of clarity on the roles and responsibilities among LTR, VEDCs and DESB officers in implementing LTR CE activities. They expect LTR to orient them and the village authority and to provide sufficient notice for participation. “The teachers spoke very quickly during the meeting [Caregiver engagement session], so I didn’t know how to teach or support my child’s learning at home. I checked with my child whether she knows the teaching approaches; unfortunately, she did not know either.” VTE Vanvieng father 40 | FINAL EVALUATION REPORT USAID.GOV In CPS, two out of five parents said they had participated in teacher-parent meetings and orientation to RCs, while only one parent had attended the caregiver engagement session. The challenge is that they could not recall these activities well enough to state if they were satisfied. One mother in Mounlapamok, CPS did not state whether she was satisfied with the caregiver engagement activity in which she participated, but she did request more storybooks and TLMs for teaching at home because she sees her daughter enjoying the storybooks. Other parents who also weren’t familiar with LTR activities or had not participated were well aware that the children bring storybooks home to read by themselves and are noticing improvements in reading Lao. A father in CPS who participated in the reading corner orientation stated that he has no time to support his children’s reading at home because he works in rubber plantation. However, he did request support from LTR to provide at-home learning materials and storybooks so “we” could support their reading and to provide guidance for how parents can support children’s reading at home. This implies that the training would include the child’s mother and other family members. Another mother in Mounlapamok, CPS also works on a farm and teaches her child to read after work. She used to teach her children before the LTR training, but now uses storybooks with pictures so her child finds it more enjoyable. According to a VEDC member in Mounlapamok, CPS, the school in his village organized a one-day activity for groups of children and mothers to read together. It is not clear if this was part of a reading corner training. The VEDC member explained that it was very effective because the children were more comfortable asking their mother questions about everything in the book. Ways to Increase Stakeholder Engagement All key stakeholders requested additional support in the form of storybooks, training, exchanges, greater MoES staff involvement, and communication to increase engagement. All stakeholders were asked if there is anything that could be improved to build their capacity, engagement, and support for LTR. A total of 39 respondents from all target groups recommended areas for additional LTR support. As Table 5 shows, about half of all stakeholders were from schools and communities. Similar to responses in EQ2, stakeholders largely demanded more storybooks followed by refresher training, at-home learning materials, advance notice for monitoring visits, teacher exchanges, and additional support and training for multigrade teachers. Suggestions for at-home learning materials and advance notice mirrored those reported in EQ2 and thus are not detailed in this section. “The mothers opened the books and taught their children right away. It was more informal than studying with teachers.” CPS Mounlapamok VEDC 41 | FINAL EVALUATION REPORT USAID.GOV Figure 9: EQ4 Areas Where Additional Support is Required Table 5: Sample Size for Increased Stakeholder Engagement Question Respondent Type Number of respondent s Percentage Implementing Partners 4 10% MoES 5 12% PESS/DESB 13 31% Principals/Teachers 10 24% Parents/VEDCs 10 24% Total 39 100% More storybooks. Over half of respondents (15 of 39) requested more storybooks in response to children’s demand for more variety and teachers’ and parents’ requests for additional storybooks to expand children’s vocabulary and reading skills. All pre-primary teachers interviewed (5 out of 5) requested more teacher training, storybooks and TLMs (e.g., construction paper, flashcards, flipchart, posters, etc.). They would like to know where to go to obtain more storybooks in the future. 56% 18% 15% 5% 5% 5% 0% 20% 40% 60% Storybooks Refresher training At-home learning materials Advance notice Teacher exchanges Multigrade teacher… Additional LTR support required 42 | FINAL EVALUATION REPORT USAID.GOV Refresher training. Echoing requests in EQ2, the second most common request from all stakeholders was refresher training. District and school stakeholders (DESB officers, teachers, school principals, VEDCs) requested refresher training for all grades and cohorts (PP, G1 and G2). The justification was because the initial training was too short and because too much time has passed since the initial training, so teachers are beginning to forget the skills they learned (see textbox above). Teachers and principals specifically requested refresher training on “application of teaching techniques”. It is worth noting that at the time of data collection, refresher training was planned for the Summer of 2021. More training and support for multigrade teachers was requested by MoES and multigrade teachers. Multigrade teachers requested training on how to create lessons for multigrade classes, how to arrange reading corners, and how to assess learners. Although LTR provided training to multigrade teachers on lesson plan development and delivered a demonstration of a multigrade class in January 2021, MoES/PESS trainers and teachers have requested additional support because of the complexities of supporting learners in multigrade classes, especially since many teachers had never received training on this topic prior to LTR.22 Multigrade teachers would like a support system at the school level where they bring their questions and concerns related to multigrade teaching. There was a perception from multigrade teachers that they are not part of the TLCs, which were established for teachers to share ideas, challenges, and good teaching practices with implementing the new curriculum, including in multigrade classes. This was also exacerbated by the limited time afforded for TLCs due partly to COVID 19 restrictions. 22 USAID Learn to Read Year 3, Quarterly 2 Report (January 1 – March 31, 2021) “Refresher training for all stakeholders because teachers were trained, but only had 2-3 months to practice before schools closed. Now, many teachers are forgetting the skills they learned in the training.” CPS Bachieng School Principal “I want to know how to get parents involved in reading activities and how to encourage them. The Activity team has previously explained, but it is still not clear. I would like a review.” CPS Mounlapamok School principal 43 | FINAL EVALUATION REPORT USAID.GOV Government staff at all levels requested refresher training on parent engagement strategies. PESS/DESB would like to be trained along with teachers, VEDCs and village chiefs at DESB on how to increase parent engagement and their usage of RCs. Teachers and principals requested more training on CE strategies. According to respondents, parent engagement training was not clear for principals, VEDCs and parents (see textbox above). Although some of these topics may be outside the scope of the LTR Activity, government staff (MoES/PESS/DESB) would like additional training in the English language, online teaching, how to use social media for online trainings, ECE and Lao language teaching. Teacher and School Exchanges. PESS/DESB officials requested expanding TLCs across school clusters and districts for wider sharing of best practices. Government officials further suggested that the LTR team and DESB officers attend TLC meetings in order to help principals with facilitating and troubleshooting. This would help build capacity of teachers and principals and increase sustainability of TLCs. A school principal in Vang Vieng, VTE also requested school exchanges between districts for Grades 1-2 teachers to share experiences with implementing the new curriculum and teaching techniques. School exchanges were recommended to be held two to three times throughout the remainder of the project. Teachers across Group 1 and Group 3 can share lessons learned and best practices for scale-up. Those in Group 1, for instance, have more experience and can help teachers overcome challenges in Group 3. Engage more MoES staff. A LTR staff person recommended USAID provide more budget to engage relevant MoES staff in monitoring activities. The project currently does not have the funds to invite staff from the teacher development unit and the statistics unit to join school monitoring visits; yet they are key partners for sustainability. EQ3 CONCLUSIONS ● LTR has successfully engaged MoES, PESS and DESB in LTR activities. This is evident in the fact that they are satisfied or very satisfied with their level of participation in LTR activities. ● LTR effectively engaged parents and VEDCs in the north but fell short of expectations in the south. Parents interviewed in the north had higher levels of participation in LTR activities in caregiver engagement, reading corners and reading festivals. Parents were more aware of LTR and familiar with VEDCs compared to parents in the south. Meanwhile, the majority of parents interviewed in the south could not properly recall any LTR activities even if they attended. These findings indicate that LTR has not yet effectively engaged parents and community stakeholders in CPS. This is likely due to the delayed rollout of reading festivals and other CE activities as a result of COVID-19. The timing and communication of CE events has also affected participation. ● Parent and VEDC satisfaction were associated with their level of participation in the project, understanding of objectives, and engagement of village leaders. Due to the high participation of parents and VEDC members in the north, they understood the project objectives and were very satisfied with the project compared to the south where parents struggled to recall the activities in which they had participated. LTR’s failure to engage village authorities in CPS contributed to their dissatisfaction with the project and lack of engagement. The CE strategy of working through schools and teachers to reach community members may not be an effective strategy for meeting the communities’ expectations in CPS. ● Although parents lack time to read with their children at home in both VTE and CPS, the findings showed that when parents actively participate in LTR activities, they do become engaged in reading activities with their children. Moreover, 60 percent of parents (6 of 10) requested additional 44 | FINAL EVALUATION REPORT USAID.GOV storybooks and learning materials to support their children with reading at home, including three out of five parents in CPS. Thus, there is high demand for at-home learning materials (e.g., reading corner storybooks and TLMs) and guidance for parents in both provinces. Mothers tended to support their children with reading and studying at home; school activities targeting mothers in CPS have been successful. ● Additional support required for accelerating sustainability mirrored the EQ2 recommendations, including: provision of at-home learning materials and training parents (COVID-response), more storybooks/TLMs, refresher training, additional support and exchanges for multigrade teachers, and advance notice for school monitoring activities. However, a new recommendation was to engage other critical departments of the MoES in monitoring LTR activities to ensure sustainability (e.g., statistics department and teacher development unit). EQ4: DEVELOPMENT PARTNERS What aspects of other development partners’ supported programs, such as the World Bank second Global Partnership for Education (GPE II) and the Basic Education Quality and Access in Lao PDR project (BEQUAL), are most complementary to, and support of the goals and objectives of LTR? a. Among existing LTR partners, if USAID were to support any additional program components, which are most compelling and why? That is, which are most able to provide credible evidence of: (1) effectiveness at supporting the development of reading skills for children at pre-primary, grade 1, and grade 2 levels; and (2) organizational/programmatic strengths that could be leveraged during the remaining period and/or beyond? EQ4 FINDINGS AND CONCLUSIONS Findings for EQ4 are largely based on desk review of development partners’ public documents and interviews with key representatives from each donor or implementing partner. Complementary components of development partners’ supported programs: teacher professional development, COVID-19 response, and community engagement 45 | FINAL EVALUATION REPORT USAID.GOV Table 6: Complementary Components of LTR and Development Partners’ Supported Programs LTR UNICEF BEQUAL GPE Objectives Improved reading skills of all students PP-G2 (non-Lao-Tai speaking and children with mild disabilities) Improved quality and access to ECE in rural and remote areas (non-Lao-Tai speaking and children with disabilities) Improved teaching quality in Grades 1-5 Improved primary learning outcomes and strengthened teacher performance (PP-G3) Term 2018 - 2023 2022 – 2025 2022 - 2026 2021 - 2026 Teacher Professional Development √ √ √ √ Inclusive Education √ √ √ √ Multigrade teaching √ √ COVID-response √ √ √ Online Teaching and Learning Platform √ √ √ √ Community engagement √ √ Read at Home initiatives √ √ √ UNICEF Objectives. The objective of the next phase of the United Nations Children’s Fund’s (UNICEF) education programming is to improve the quality and access to ECE through more equitable policies, sustained investment, and improved capacity of teachers, ECE facilitators, village leaders, education officials and parents. Target population. Although the project will work across both public schools and community based ECE centers throughout the selected districts, the main target population is ECE children in rural and remote communities, with a particular emphasis on children who do not speak Lao and children with disabilities. The target population overlaps with LTR’s target population for pre-primary school children. Components. UNICEF’s three key result areas mirror LTR’s and include: 1. Capacity development support to ECE providers to provide quality ECE programs 2. Provision of quality ECE learning materials to promote school readiness (literacy, numeracy, socio￾emotional development); and, 3. Parent and community engagement. 46 | FINAL EVALUATION REPORT USAID.GOV Activities that overlap with LTR. The key UNICEF activities that overlap with LTR’s objectives as illustrated in Table 4, but may be implemented in different districts are: ECE professional development, coaching and peer learning; development of quality ECE learning materials and multi-media for Lao language acquisition and non-Lao-Tai speakers (includes reading corners, storybooks, TLMs, games); use of MoES online teaching and learning platform (Khang Panya Lao) for training, teaching, and learning at home; and community engagement in reading, learning and materials development. UNICEF also has COVID-response activities that could help address gaps identified in this midterm evaluation (e.g., take home learning materials, catch-up program, blended trainings, and digital offline resources). These activities are described in further detail below. Curriculum rollout. UNICEF is supporting the MoES with the national rollout of the pre-primary curriculum for the 2021 – 2022 academic school year. UNICEF supported the MoES with revising the curriculum and integrating objectives, strategies and materials developed under LTR. As a result, the curriculum now includes a Language Development Dimension that focuses on improving foundational skills in listening, speaking, and reading. Moreover, the pre-primary curriculum teacher’s guide now includes a separate reading readiness component that emphasizes the development of reading skills. As part of the new curriculum rollout, UNICEF and MoES plan to establish reading corners in every pre-primary classroom leveraging the 16 storybooks, postures and picture cards developed to date under LTR. UNICEF has been supporting the printing of LTR materials for other districts and provinces not targeted by the Activity. Capacity building. To build capacity of ECE providers to implement the new pre-primary curriculum, UNICEF will develop a blended learning course on Early Childhood Development (ECD) and the new primary curriculum that will include multi-age classes and supporting young children to read (using LTR materials). Similar to LTR’s teacher professional development approach, UNICEF will supplement teacher￾training with coaching, mentoring and peer-to-peer learning. UNICEF will also strengthen ECE knowledge and skills of PESS and DESB staff. Considering LTR’s experience with pilot-testing the RRP and providing teacher professional development and capacity building for DESB/PESS, they could potentially assist UNICEF with implementing similar approaches as well as providing coaching and feedback on implementation of the new pre-primary curriculum. ECE materials development. Under Result 2, quality ECE learning materials will be developed to promote school readiness for Lao and non-Lao-Tai speakers. UNICEF will develop multi-media (print, audio, videos) for Lao language acquisition in both Lao and other major languages. This includes development of storybooks, flashcards, picture books, and games that promote literacy and Lao language acquisition in accordance with the national Early Learning Development Standards. All materials will be uploaded onto Khang Panya Lao for other partners to reproduce and disseminate widely. UNICEF has already partnered with LTR to print and disseminate as well as upload LTR materials on Khang Panya Lao. A similar arrangement has been agreed to with Australia DFAT for decodable readers produced under BEQUAL. Thus, there is an opportunity for LTR to assist with both developing and disseminating UNICEF materials to expand access to quality ECE materials for LTR target schools. An innovative approach for language and literacy skills development is evelopp storybooks based on a local ECD TV program. UNICEF has produced 10 episodes for the first season of an ECD TV program entitled “My House”, which promotes physical, cognitive, social, and emotional development of children and engages family members to support young children’s learning and development in a fun, interactive way. The show also promotes language acquisition for young children and integrates COVID-19 47 | FINAL EVALUATION REPORT USAID.GOV prevention messages. UNICEF plans to develop storybooks based on “My House” that will be printed and disseminated widely. This approach seems highly effective since children can see characters in a storybook coming to life on the TV program and the books can reinforce or expand upon storylines in the TV episodes. Additional strategies for ECE materials development include translating global language resources into Lao (storybooks, games) for printing and wider dissemination, as well as global resources already in Lao language; and facilitating a child-to-child reading program (older children read to ECE school children). As noted under Result 1, reading corners will be set up in every pre-primary classroom initially drawing on LTR resources and then enriched with these additional resources developed under the UNICEF project. Online teaching and learning platform. MOES with support of UNICEF, GPE and EU, has developed the MoES Teaching and Learning Platform, Khang Panya Lao. It is an online, mobile, and offline tech platform full of digitalized curriculum content and additional learning resources for students and teachers. It can be accessed via the web and offline via an Application (App). UNICEF plans to use Khang Panya Lao for blended learning for professional development of teachers, principals, Pas and education staff, online learning during school closures or at-home supplementary learning, as well as a means to enhance digital skills of Lao children, teachers and MoES staff. UNICEF is exploring partnerships with internet providers (for internet access, provision of tablets) to increase access to Khang Panya Lao resources in rural and remote areas. The MoES and UNICEF not only plan to use the platform for online teaching and learning, but also as a shared database for printing and distributing high quality ECE materials. All LTR storybooks and curricular supplementary materials have been uploaded into Khang Panya Lao and it is expected that any new materials (e.g., audio files) will be uploaded in the future. Community engagement. UNICEF’s strategy for ECE is rooted in a community-based approach. UNICEF implements community-based school readiness (CBSR) programs where there are no existing ECE facilities. CBSR centers are established with communities and DESB, who commit to converting the CBSR center into a formal pre-primary class within two-years. This ensures ECE coverage in rural and remote communities and sustainability of the center. The centers are attached to the nearest primary school and run by two community volunteers who receive training from UNICEF. DESB and MoES provide monitoring and coaching support to the facilitators and VEDCs help run the program. Although this approach is quite different from LTR’s who work directly with schools who then engage community members, there are activities that complement LTR’s goals of improved reading in the school and community. For instance, UNICEF plans to work with communities and VEDCs to develop bilingual picture books. They will also conduct a reading campaign that encourages parents/families to read to their young children. This will include development of read aloud videos for teaching parents effective reading strategies, as well as encouraging parents to produce their own videos of their children or family reading together. Other relevant activities still under development include: a revolving library and a mechanism for at-home learning. COVID-19 response. UNICEF’s new program incorporates pilot activities tested during the COVID-19 era to address the ongoing challenges related to school closures and the pandemic. Specific strategies include blended training for capacity development, digital and multimedia resources for Lao and non-Lao-Tai speakers, catch-up programs, and take-home learning materials (e.g., coloring book, stationary, storybooks, picture books) as well as training for parents on how to support reading at home supplemented with read aloud videos. The TV program is another potential COVID-response activity 48 | FINAL EVALUATION REPORT USAID.GOV since children can learn while at home through an educational program. All of these activities respond to LTR stakeholders demands for COVID-19 response plans, materials, and ways to ensure teaching and learning at home during school closures. Although UNICEF and LTR have not collaborated in the past with the exception of sharing materials through Khang Panya Lao, UNICEF plans to scale up LTR materials and is interested in discussing areas for collaboration, such as the activities identified above that mutually reinforce each program’s objectives. BEQUAL Objectives. BEQUAL Phase II will focus solely on one critical factor: Teaching Quality. The end of program goal will be improved teaching to better facilitate inclusive learning and improved primary learning outcomes. The second phase of BEQUAL will cover the period of July 2022 to June 2026 and will build upon the achievements of the first phase, assisting the MoES to complete the national roll out of the curriculum for the full five grades of primary school. Target population. BEQUAL is targeting teachers in grades 1-5. Building on teacher professional development for teachers in grades 1-3 under Phase 1, BEQUAL 1I will focus on rolling out the curriculum, training, and professional development support to teachers in grades 4-5. LTR assisted BEQUAL with the rollout of the literacy curriculum in grades 1-2 and provided additional support in LTR target schools through supplementary materials, coaching and monitoring (community engagement, disability, teaching of non-Lao-Tai speakers). Grades 4-5 targeted under BEQUAL II are beyond the scope of the LTR project. However, BEQUAL will extend some activities under Phase I to teachers in grades 1-2 (e.g., Spoken Lao Language Program). Components. Though a system strengthening approach, BEQUAL phase II will encompass 3 components: 1. Primary curriculum and learning resources for grades 4-5, 2. Teacher professional development working through targeted teacher training colleges to provide pre-service and in-service training, and 3. Teacher management and support focusing on leadership, district support and teacher management policy. BEQUAL activities that overlap with LTR. There are three activities that overlap with LTR and would address gaps identified in this report: 1) Spoken Lao language pilot; (as part of their focus on gender and inclusive education); 2) teacher professional development (school-based continuous professional development and TTCs); and, lesson plans for multigrade classes. Note that the Spoken Lao language pilot aligns with the following concrete action listed in the GIDAP for inclusive teaching and learning materials: develop oral language bridge for children who do not speak Lao. BEQUAL will also develop and mainstream a COVID-19 strategy within six months of Phase 2 implementation. 49 | FINAL EVALUATION REPORT USAID.GOV Spoken Lao Language pilot. The Spoken Lao Language Pilot (SLP) is an activity that uses the mother tongue as a bridge to learning Lao. Teachers in the pilot program provided one hour a day of oral Lao language for non-Lao-Tai speaking students in G1. The Spoken Lao language pilot was implemented in 80 schools across 7 districts in 4 provinces. SLP targeted remote and rural schools in selected districts where a non￾Lao-Tai language is commonly spoken. COVID-19 disrupted implementation, which restarted in October 2020. Results of the pilot were expected to be available in mid-2021; however, they are still not public. USAID and SC are interested in collaborating with BEQUAL to scale up the pilot. BEQUAL is also keen to continue collaborating with LTR, but state that RIES is currently assessing the pilot and deciding on next steps for scale-up. They would like to review and revise the SLP materials prior to rolling out in 2022. The MoES is considering several options for scale-up of the SLP: 1) possibly extending it from G1 to G2 classes, 2) a scale-up designed specifically for non-Lao-Tai speaking populations, or 3) scaling up nationally through mainstreaming selected relevant approaches into the curriculum. The scope of the Spoken Lao Language program and possible collaboration with LTR and development partners is subject to the results of the pilot and further dialogue with the MoES. Teacher professional development. As BEQUAL has been supporting the rollout of the Grades 1-2 primary curriculum with LTR in phase I, further collaboration may be possible. For instance, BEQUAL supported continuous professional development through provision of Teacher Support Packs, classroom observation tools, peer learning approaches (co-teaching, WhatsApp group, TLC activities), and teacher demonstration videos to support them implementing the new curriculum. These materials may be useful for LTR future trainings, coaching, and TLCs. BEQUAL is also exploring collaboration with DTE and GE in developing virtual training for teachers in remote areas using the MoES online platform (Khang Panya Lao). That is another opportunity for LTR to collaborate with both BEQUAL and UNICEF on how to utilize the platform for online or blended trainings in the future, as part of their COVID-19 response plan. Lesson plans for multigrade classes. Many LTR stakeholders have been requesting additional training and resources for multigrade teachers, including how to develop lesson plans, manage classes, and assess teacher performance. BEQUAL has developed lesson plans for multigrade classes, which are prevalent in ethnic schools, pkus teacher guides with different learning needs. LTR has collaborated with BEQUAL on using multigrade teaching materials in the past, and may continue to collaborate with BEQUAL on additional refresher training and guides to support them with teaching in the multigrade classroom. COVID-19 strategy. BEQUAL’s COVID-19 strategy will outline measures that will be mainstreamed across all Phase 2 activities. The COVID-19 strategy will consider Gender Equity and Inclusive Education (GE&IE) elements such as access to information and communications technology (ICT), learning aids, and equipment for online learning. “I don’t know if there are any plans for collaboration, but we will continue to work with LTR on building capacity of teachers to implement the curriculum in specific areas (Lao language, multigrade, monitoring)” BEQUAL staff 50 | FINAL EVALUATION REPORT USAID.GOV BEQUAL staff suggested that LTR support parent engagement, particularly sensitizing them to the new curriculum and grading procedures (using Rubric). They suggested that parents become more involved in student learning, reading aloud and monitoring student academic performance. BEQUAL is also interested in learning more about LTR’s tools and trainings on inclusive education that they can incorporate into their trainings on the new curriculum. GPE Global Partnership for Education III (GPE III): Learning and Equity Acceleration Project (LEAP) Objectives and components. GPE III LEAP is a five-year program starting in 2021, with the goal of improving primary learning outcomes in primary grades and strengthening teacher and system performance. GPE III has 3 main components: 1. Prepared and motivated learners (Reading readiness program) 2. Prepared and motivated teachers (professional development for teachers in grades 1-3, including for ethnic teachers) 3. Effective school governance for improving teaching and learning Target population. GPE III will target teachers and students in pre-primary and primary grades 1-3. Activities that overlap with LTR. There are three GPE III activities that complement LTR: 1) expansion of the GPE Reading Readiness Program that was developed with LTR; 2) professional development of teachers to implement the new curriculum in grades 1-3; and 3) COVID-response initiatives, including a read at home initiative, a remedial education program and use of the MoES online teaching and learning platform. Component 1: Reading Readiness Program. The objective of Component 1 is to provide early learning opportunities for young learners ages 3-5 to be prepared and motivated for entering G1. In collaboration with the MoES, GPE and LTR co-developed and co-pilot-tested the Reading Readiness Program in selected pre-primary schools. According to GPE, the collaboration was very close and positive. A World Bank representative interviewed is very impressed with LTR’s professionalism and skills, exclaiming, “we are a great team.” Although not publicly available, the MoES and GPE stated during interviews that the impact evaluation results have been promising. As a result, the MoES and GPE have decided to scale these interventions. Under GPEIII, it will continue serving classrooms receiving interventions under Global “The MoES and development partners are working closely [on RRP]. I was impressed because it’s hard to get ownership from the government, so I was very impressed.” MoES representative 51 | FINAL EVALUATION REPORT USAID.GOV Partnership for Education II (GPEII) (33 districts) and will expand to 24 new districts for a total of 755 pre-primary classrooms. GPEIII will also scale up RRP to Teacher Training Colleges (TTCs) targeting pre￾service teachers and will continue to rollout to in-service teachers through teacher training, coaching and continued professional development. GPEIII foresees LTR playing a critical role in this scale up. Component 2: Improve the quality of teacher professional development. GPEIII will provide training on the new curriculum grades 1-3, develop a teacher monitoring and performance appraisal system, and establish a school-based professional development system. As part of component 2, GPEIII will also develop enhanced materials to train ethnic teachers on how to teach non-Lao-Tai speaking students the new curriculum for Grades 1 to 3. These activities overlap with LTR/BEQUAL activities for grades 1-2. LTR could collaborate with GPE through sharing materials already developed for non-Lao-Tai speaking students and/or improving upon existing materials. COVID-19 response Read at Home Initiative. Under Component 1, GPEIII is developing a read-at-home initiative to support parents with helping their children learn how to read. The project will provide books for households in hard-to-reach areas of target districts, support materials and train parents on techniques to support their children’s learning and reading skills at home to make reading part of their daily routine. The activity has not been fully designed yet and will be coordinated with the READ @Home Team from the World Bank and with UNICEF, who is the grant agent for the GPE response to COVID-19 project. UNICEF may also implement this activity as part of the short-term response to COVID-19 in locations not covered by LEAP. GPE is keen on working with LTR to implement this community-based activity. COVID-19 Response Remedial Education under Component 2: As part of GPEII’s COVID-response strategy, the project will equip teachers with the skills to assess and provide remedial education to students returning to school. COVID-19 Response Online Teaching and Learning Resources. GPEIII will collaborate with and build on the MoES online teaching and learning platform that UNICEF is leading the development of in collaboration with Microsoft, Cambridge University, and other development partners. The platform will make the pre￾school and primary curriculum available online, as well as other teaching and learning resources for students, parents, teachers, principals, and PAs. According to a GPE staff member interviewed, if LTR continues to cover teacher training and mentoring support, and the World Bank covers school-based management and learning outcomes, they can continue to complement each other. GPEIII foresees LTR playing a role in refining the RRP based on the IE results and feedback from coaches, scaling up the RRP, in pilot-testing the Read at Home initiative, in incorporating teacher assessments (World Bank’s TEACH classroom observation tool and COACH tool) into the RRP, and in parent engagement. Organizational/Programmatic Strengths It was difficult to gather evidence on organizational/programmatic strengths since evaluation reports of past program phases were unavailable. Thus, the findings below are limited to what was revealed during stakeholder interviews. Organizational strengths that could be leveraged during the remaining period and beyond include continuing to work with BEQUAL trainers and engaging VEDCs and DESB in community engagement activities 52 | FINAL EVALUATION REPORT USAID.GOV UNICEF. As described in the first findings section of EQ4, UNICEF has a very effective community￾engagement approach that could be leveraged by LTR. UNICEF has worked successfully at the community level relying on MoES staff to facilitate and monitor activities without a community facilitator or paid volunteer. The key strategy, according to UNICEF staff interviewed, is engaging the VEDC from the beginning of the project prior to implementation so they understand the purpose and their role; and, selecting communities where there is strong interest and high levels of participation. “If the community understands the benefit, they will lead the activities”, according to UNICEF staff. From the inception of the Activity or program, UNICEF engages with the VEDC, Village Chief, school principal, and DESB officers to create a system for implementing and monitoring CE activities. All such stakeholders are involved in initial dialogue and trainings to understand the project objectives and their roles and responsibilities. Once implementation begins, school principals work with VEDCs directly. When project staff and/or DESB officers visit the school, they consult with the VEDC to share findings from their observations and inquire about community needs. According to UNICEF, it is very important to work closely with the government authority and the community facilitator, which is typically the Village Chief. BEQUAL. BEQUAL developed a network of master trainers across provinces, schools and teacher training colleges that could be leveraged in the future. The secret to BEQUAL’s success was trying to identify the people with the right knowledge and skills who could actually deliver a training without having to significantly build their capacity. They recruited trainers from different levels of the MoES to select the most qualified and then built their capacity and developed a network of master trainers within Teacher Training Colleges, PESS, DESB and schools. LTR has been successfully collaborating with BEQUAL to recruit master trainers from the existing pool and has used many of the same skilled trainers. Another lesson learned from BEQUAL that was adopted by LTR was working across key departments of the MoES (e.g., RIES, DTE, GE). The Research Institute for Educational Studies develops the curriculum and conducts the master training on the new curriculum and teaching materials. The Department of Teacher Education is in charge of training teachers on the new curriculum. The Department of General Education is responsible for supporting and monitoring teachers at the school level. Thus, these three departments must work together. Prior to BEQUAL, these three departments were fully independent of each other. BEQUAL worked hard to bring them together so that they could collaborate. LTR has been successfully working with these and other departments of the MoES since the inception of the Activity. EQ4 CONCLUSIONS Based on the above findings regarding the most compelling components that provide credible evidence of effectively supporting the development of reading skills for children in PP, G1 and G2, the following conclusions can be drawn: “I know LTR has used a number of the same trainers, which has been great. And that’s exactly what they (the system) were designed to do.” BEQUAL staff person 53 | FINAL EVALUATION REPORT USAID.GOV ● Among the pilot-tested interventions and potential activities for scale-up, RRP is the most credible and compelling activity according to the Ministry stakeholders interviewed and analysis of EQ4 findings. There are multiple supporting arguments. First, the RRP and all supporting materials, including storybooks, have been developed by LTR and GPE and approved by the MoES for implementation. Second, there is preliminary evidence of its impact on students, according to stakeholders interviewed. Third, the MoES has decided to integrate the RRP into the national pre￾primary curriculum and roll out through pre-service and in-service training. Fourth, more support is needed at the district level for coaching and expansion into new pre-primary and kindergarten schools. The findings from previous EQs showed a significant need for more training, coaching and materials for pre-primary teachers. Therefore, to increase sustainability of the LTR materials, it is critical that LTR continues to support the refinement of the program based on results from the impact evaluation and the rollout into new kindergarten and pre-primary classrooms. ● Although BEQUAL will be working in higher grades for Phase II, there will be overlap with LTR activities in Grades 1-2. Specific activities that complement LTR are the Spoken Lao Language Pilot and training on the new curriculum for non-Lao Tai speaking students and ethnic teachers. ● There is significant overlap between UNICEF and LTR activities as both are targeting non-Lao-Tai language speaking students and those with disabilities in rural and remote communities. UNICEF’s COVID-response interventions, multi-media activities, and community engagement strategies are well aligned with the needs and gaps identified in this LTR midterm evaluation (need for additional training, TLMs, COVID-response, at-home learning materials and guidance for parents, and strengthened community engagement). ● UNICEF’s proposed activity to develop storybooks based on a popular ECD TV program is an innovative strategy for engaging families, including non-Lao-Tai speakers, in activities promoting language acquisition, reading skills and COVID-19 prevention. According to UNICEF staff interviewed, 85 percent of households have access to electricity. More research is necessary to explore how many children, particularly in remote and rural communities, have TVs or other streaming devices to determine how feasible it will be to implement this activity within LTR communities. If it possible to download and watch the TV program offline using phones or tablets, then it could be a worthwhile investment for collaboration. The below conclusions are based on findings regarding organizational strengths of development partners: ● LTR and GPE/BEQUAL have been effectively complementing each other’s work and mutually benefitting from their organizational strengths and implementation strategies. LTR and BEQUAL are working together to build the capacity of national trainers to implement the new curriculum using IE methodologies. ● GPE/BEQUAL/UNICEF acknowledge the value-add of LTR as the inclusion and reading experts; they are all keen to collaborate in the future and build on existing LTR initiatives. ● UNICEF has an effective community engagement strategy that could be leveraged in the remaining period of LTR and future USAID programs. 54 | FINAL EVALUATION REPORT USAID.GOV RECOMMENDATIONS The following eight critical recommendations were consolidated from the full list of detailed recommendations developed from analyzing each EQ’s findings and conclusions (See Annex B for the breakdown of recommendations by EQ). The recommendations are listed in order of priority and geared towards improving the performance and sustainability of the LTR Activity. They are primarily intended for USAID and LTR to consider as they design the next phase of LTR and other USAID-funded education programs in collaboration with the MoES and development partners. 1. LTR should develop a COVID-Response Plan to increase the likelihood of achieving results for Results 2 and 3, and sustainability for all three components. The COVID-Response Plan should outline strategies to train teachers through virtual23 or blended trainings, and to engage parents in at-home learning activities with access to teaching and learning materials. The comprehensive plan should also include elements recommended in the MoES COVID-19 response plan and by LTR stakeholders interviewed, including: PPE, at-home learning kits, back-to-school campaign featuring COVID-19 prevention protocols, catch-up lessons, online learning for urban schools/schools used as quarantine facilities, and a mobile community library so that storybooks are accessible during school closures. LTR should work with the MoES and development partners to develop a comprehensive plan building on existing pilot initiatives (See EQ4). LTR should leverage inputs from SC, MoES, UNICEF and other development partner staff involved in developing the Lao PDR MoES Education COVID-19 Response plan[1]. 2. LTR should develop a more effective community engagement strategy that links to the MoES Development plan for 2021-2025 for sustainability. The well-articulated strategy should focus on engaging VEDCs and parents in rural and remote communities. To address dissatisfaction in CPS, LTR should engage village authorities in LTR orientations and CE promotion. The Village Chief can disseminate information to villagers and mobilize communities to support reading. Trainings and orientations for parents should be organized around their schedules and conducted in a familiar language. Apart from targeting parents/caregivers for CE activities, LTR should also involve older siblings of PP, G1 and G2 students in at-home, community-based or after-school activities because many teachers mentioned that students receive support from their older brothers/sisters to read. LTR should provide additional storybooks and guidance for how families can support their children’s reading at home. LTR should collaborate with UNICEF to design CE activities, such as engaging communities in materials development and read at home initiatives, reading campaigns and revolving community libraries. 3. Aligned with the LTR Cooperative Agreement, LTR should develop and/or modify a sustainability plan for the remaining period along with benchmarks for tracking progress and milestones. The sustainability plan should focus on transferring leadership responsibility to MoES in Phase 3, and building the capacity of DESB officers for training, coaching, and monitoring. LTR should integrate recommended activities from stakeholders from this midterm evaluation into the sustainability plan, such as capacity building of MoES, PESS and DESB for monitoring and assessment, refresher training and additional support for teachers, strengthening community engagement, and creating an extensive library of TLMs leveraging Khang Panya Lao resources, public-private partnerships for printing new materials, and engaging teachers, students and community members in developing new storybooks and teaching aids. Drawing on lessons learned 23 Based on LTR experience, virtual training would require teachers (who do not have access to laptops or internet access) to travel to the districts and utilize resources at the DESB office in a safe environment following COVID-19 protocols. 55 | FINAL EVALUATION REPORT USAID.GOV in which RtR reading rooms were not sustained, the MoES should have a budget and dedicated person responsible for maintaining reading corners and procuring additional storybooks. 4. LTR should provide additional refresher training24 for teachers: Over half of evaluation respondents requested additional refresher training and support for teachers, particularly multi￾grade, and pre-primary teachers. Thus, LTR should continue to provide refresher training and additional resources until they master the content and methodology, which would increase the likelihood of sustainability and scale-up to new teachers. Refresher training is critical because teachers are already forgetting what they have learned in initial trainings. (For full list of recommended training topics, see EQ2 Recommendations in Annex B) 5. USAID and MoES should develop a strategy for sustaining and printing more materials: To address the shortage of materials, the need for more variety or replenishment, and to increase the likelihood of sustainability for Result 1, USAID and MoES should seek to engage local book publishers to print and distribute LTR storybooks as well as other ECE storybooks through public-private partnerships. 6. LTR should review the staffing structure in the MoU with USAID and the MoES against the workload requirements in each province to ensure staffing is sufficient to achieve Activity goals for Phase 3. According to Project Coordinators and POs, there is a shortage of POs in Attapeu province. Additional qualified MEAL staff and YLCs in ethnic communities with low community engagement are also recommended for Year 4, which may already be included in LTR’s Work Plan. 7. LTR should strengthen its M&E team capacity and system to produce accurate data on activity indicators and outcomes. The M&E staff could conduct an EGRA or other reading assessment to assess reading outcomes for Lao and non-Lao-Tai speakers, which could potentially be carried out with Group 3 pending school re-opening.25 It is critical for the credibility and sustainability of the Activity that they have accurate quantitative data for tracking progress achieved to date against indicators and targets. If not possible due to COVID-19 school closures, then LTR may need a proxy indicator, such as number of books read and other indicators to show improved reading habits at home. The final evaluation should include a statistical report of progress achieved for the MoES to understand the indicators, achievements and gaps that will need to be addressed in the future beyond LTR. 8. In general, LTR should provide advance notice to stakeholders for LTR activities: To increase the engagement of all stakeholders in LTR activities, LTR should provide at least one week notice for training, meetings, and joint school monitoring visits. Due to their obligations to multiple donors, the MoES requested several months’ notice. VEDCs, school principals and parents would like to be notified in advance when LTR is conducting school visits so that they can also partake. Furthermore, it would provide an opportunity for LTR to meet with parents and VEDC members and provide orientation or address any concerns. 24LTR has included refresher training for all teachers and cohorts in the original design of the Activity. The findings and recommendations affirm the need for maintaining this activity and potentially supplementing it with additional follow-up training as needed for pre-primary and multigrade teachers. 25 As noted previously, the continuing restrictions due to COVID-19 halted the baseline data collection for EGRA or reading assessments. 56 | FINAL EVALUATION REPORT USAID.GOV 9. LTR should continue collaborating with development partners on activities that overlap with LTR objectives and target populations: For instance, LTR should continue implementing, refining, and scaling up RRP with GPE. LTR should collaborate with GPE, BEQUAL and UNICEF on COVID-response activities. LTR should collaborate with UNICEF on complementary activities that are well aligned with the needs and gaps identified in this report, specifically ECE materials development, blended training, and CE activities. USAID and LTR should further explore opportunities to collaborate with the MoES and BEQUAL on the Spoken Lao Language pilot, continuous professional development for teachers in Grades 1 and 2, and on lesson plans for multigrade classes. 57 | FINAL EVALUATION REPORT USAID.GOV ANNEX A: EVALUATION DESIGN MATRIX 58 | FINAL EVALUATION REPORT USAID.GOV EVALUATION MATRIX – USAID LTR MTE # Evaluation Question Data Collection Tools Data Source and Quantity Location of Data Collection Data Disaggregation Data Analysis 1 Progress towards Activity objectives: How likely will the three key result areas, enumerated in the results framework, be achieved within the set timeframe? 1. Which components of the three key result areas are perceived to have been most effective/successful during the implementation to date? What is driving these successful components? b. Which components, of the three key result areas, are perceived to have been least successful to date? What is hindering these components? KIIs FGDs KIIs: USAID (1 KII) SC (6 KIIs national) SC (2 KII in district) RTR (1 KII national and 1 KII district) Humanity & Inclusion (HI): (1 KII:) Government Officials (6KIIs national) District Gov. Officials (8) Principals (10) Pre-primary Teachers (5) Grade 1 and 2 Teachers (10) World Bank (1 KII) BEQUAL (1 KII) FGDs: Parents/Caregivers (10) Village Education Development Committees (10) VTE CH Stakeholder type Sex Location Ethnic Group Findings, conclusions, and recommendations matrix (updated weekly) Data coding Qualitative thematic data analysis strategies 59 | FINAL EVALUATION REPORT USAID.GOV 2 Activity implementation/operations factors: What (and how) are internal operational processes, - other than MoU process, (i.e., funding, management/operations, changes of context, etc.) supporting and/or hindering implementation? 1. What external factors, if any, have impeded the implementation of the project (e.g., COVID-19, funding, etc.)? Are there any external factors that have supported the implementation of the project (e.g., additional funding sources, partnerships)? If yes, what, and how? 2. What internal factors, if any have impeded the implementation of the project (e.g., management, operations, human resources)? Are there any internal factors that have supported the implementation of the project (e.g., management/operations)? If yes, what, and how? 3. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed and/or improved to increase the likelihood of sustainability? Desk review KIIs FGDs KIIs: USAID (1 KII) SC (6 KIIs national) SC (2 KII in district) RTR (1 KII national and 1 KII district) HI: (1KII) Government Officials (6KIIs national) District Gov. Officials (8) Principals (10) Pre-primary Teachers (5) Grade 1 and 2 Teachers (10) GPE II (1 KII) BEQUAL (1 KII) FGDs: Parents/Caregivers (10) Village Education Development Committees (10) VTE CH Stakeholder type Sex Location Findings, conclusions, and recommendations matrix (updated weekly) Data coding Qualitative thematic data analysis strategies 60 | FINAL EVALUATION REPORT USAID.GOV 3 Stakeholder dynamics: What evidence exists, and what does it tell us, about how effectively Learn to Read has been engaging the Ministry of Education and Sports and other key stakeholders at central, provincial, and district levels to meet the Activity objectives? a. MoES: Are Ministry of Education and Sports stakeholders at the central, provincial and district levels satisfied with their level of participation? Do they perceive that their expectations have met by LTR to date? b. Sub-partners/IPs: Are program partners (Room to Read, Humanity & Inclusion) satisfied with their level of participation? Do they perceive that their expectations have been met by LTR to date? c. School-community stakeholders: Are teachers, caregivers and principals satisfied with their level of participation? Do they perceive that their expectations have been met by LTR to date? d. In terms of engagement of stakeholders, what additional/modified support is required, if any, to accelerate the potential for Activity sustainability? KIIs FGDs KIIs: USAID (1 KII) SC (6 KIIs national) SC (2 KII in district) RTR (1 KII national and 1 KII district) HI: (1KII) Government Officials (6KIIs national) District Gov. Officials (8) Principals (10) Pre-primary Teachers (5) Grade 1 and 2 Teachers (10) GPE II (1 KII) BEQUAL (1 KII) FGDs: Parents/Caregivers (10) Village Education Development Committees (10) VTE CH Stakeholder type Sex Location Ethnic Group Findings, conclusions, and recommendations matrix (updated weekly) Data coding Qualitative thematic data analysis strategies Quantitative data, if available, will be analyzed for basic statistics using SPSS 61 | FINAL EVALUATION REPORT USAID.GOV 4 Current LTR partners: What aspects of other development partners’ supported programs, such as, the second Global Partnership for Education (GPE-II) and the Basic Education Quality and Access in Lao PDR project (BEQUAL), are most immediately complementary to, and support of the goals and objectives of LTR? a. Among existing LTR partners, if USAID were to support any additional program components, which are most compelling and why? That is, which are most able to provide credible evidence of: (1) effectiveness at supporting the development of reading skills for children at pre￾primary, grade 1, and grade 2 levels; and (2) organizational/programmatic strengths that could be leveraged during the remaining period and/or beyond? KIIs KIIs: USAID (1 KII) SC (6 KIIs national) RTR (1 KII national) HI: (1KII) Government Officials (6KIIs national) GPE II (1 KII) BEQUAL (1 KII) VTE Stakeholder type Findings, conclusions, and recommendations matrix (updated weekly) Data coding Qualitative thematic data analysis strategies 62 | FINAL EVALUATION REPORT USAID.GOV ANNEX B: FULL RECOMMENDATIONS The following recommendations are organized by each evaluation question. The recommendations were developed based on evidence generated from analysis of findings and corresponding conclusions. EQ1 RECOMMENDATIONS ● USAID should continue investing in high-quality materials development and teacher professional development. Results to date indicate that this comprehensive teacher professional development approach is most effective for achieving the Activity goal of improved reading skills for PP, G1 and G2, including for non-Lao-Tai speaking learners. LTR should continue to roll out the trainings and materials in Cohort 3, provide refresher training for Cohorts 1 and 2 as planned, and consider scale up to Teacher Training Colleges for sustainability working in collaboration with the MoES and development partners. In the next phase, USAID should focus teacher training and materials development efforts on addressing challenges in pre-primary and multigrade classes (consistent with findings in EQ1, EQ3 and EQ4). LTR should develop COVID-19 response strategies to increase the likelihood of achieving results for Results 2 and 3. It is important that LTR considers other modalities for teacher training and CE, bearing in mind achievement of results for Components 1 and 2 is contingent on the COVID-19 situation improving (e.g., less social distancing, schools re-opening, travel restrictions lifted, and more people being vaccinated). LTR should develop a COVID￾response plan that outlines strategies to engage teachers in virtual26 or blended trainings, and to engage parents in at-home learning activities and other community-based reading activities (e.g., See EQ4 findings and recommendations from UNICEF/GPE that may be relevant). Appropriate reading assessment measures or proxies should be incorporated into the plan. USAID AND MOES SHOULD DEVELOP A STRATEGY FOR SUSTAINING AND PRINTING MORE MATERIALS. TO ADDRESS THE SHORTAGE OF MATERIALS, THE NEED FOR MORE VARIETY OR REPLENISHMENT, AND TO INCREASE THE LIKELIHOOD OF SUSTAINABILITY FOR RESULT 1, USAID AND MOES SHOULD SEEK TO ENGAGE LOCAL BOOK PUBLISHERS TO PRINT AND DISTRIBUTE LTR STORYBOOKS AS WELL AS OTHER ECE STORYBOOKS THROUGH PUBLIC-PRIVATE PARTNERSHIPS. LTR SHOULD COMMUNICATE MORE EFFECTIVELY ABOUT THE CE STRATEGY. CONSIDERING THE CE COMPONENT IS THE LEAST EFFECTIVE ACCORDING TO 70 PERCENT OF RESPONDENTS, USAID AND LTR SHOULD COMMUNICATE A CLEAR, WELL-ARTICULATED AND EFFECTIVE CE STRATEGY FOR ENGAGING VEDCS AND PARENTS IN RURAL AND REMOTE AREAS. FOR INSTANCE, LTR POS AND DESB OFFICERS SHOULD PROVIDE DIRECT TRAINING TO PARENTS AND VEDCS (RATHER THAN RELYING ON SCHOOLS) ABOUT THE PROJECT OBJECTIVES, ACTIVITIES, AND THEIR ROLES IN IMPROVING THEIR CHILDREN’S READING SKILLS. MEETINGS AND TRAININGS SHOULD BE ORGANIZED AROUND PARENTS’ SCHEDULES, AS OUTLINED IN THE GIDAP27. A TWO-PRONGED APPROACH, ONE FOR URBAN AND ANOTHER FOR RURAL/REMOTE AREAS, MAY BE NEEDED TO ADDRESS THE DISTINCT CHALLENGES IN THESE COMMUNITIES. FOLLOWING GIDAP STRATEGIES, LTR SHOULD WORK WITH THE VEDCS TO DEVELOP THE CE AND MOBILIZATION STRATEGY, CONTEXTUALIZE IT TO THEIR CONTEXT, AND 26 Based on LTR experience, virtual training would require teachers (who do not have access to laptops or internet access) to travel to the districts and utilize resources at the DESB office in a safe environment following COVID-19 protocols. 27 USAID/Save the Children. (2019). 63 | FINAL EVALUATION REPORT USAID.GOV ADDRESS PARENT ENGAGEMENT CHALLENGES IN RURAL AND REMOTE AREAS. EQ2 RECOMMENDATIONS ● LTR should develop a comprehensive COVID-19 response plan to support children’s continued engagement in learning at school or at home to achieve reading results. Apart from a COVID-19 strategy for continued teacher training and community engagement recommended under EQ1, LTR should develop a comprehensive COVID-19 response plan to continue engaging students in learning both at school and at home. This comprehensive plan should include elements recommended in the MoES COVID-19 Response Plan and by LTR stakeholders, including: provision of PPE, at-home learning kits, back-to-school campaign featuring COVID-19 prevention protocols, catch-up lessons, online learning for urban schools/schools used as quarantine facilities, and a mobile community library so that storybooks are accessible during school closures. LTR should work with the MoES and development partners to develop a comprehensive plan building on existing pilot initiatives (See EQ4). LTR should leverage inputs from SC, MoES, UNICEF and other development partner staff involved in developing the Lao PDR MoES Education COVID-19 Response plan28. ● LTR should continue providing refresher training and additional support for teachers. Because teachers are already forgetting the concepts learned in initial trainings, nearly half of respondents recommended providing refresher training and additional resources for teachers until they master the content and methodology, which in turn would increase the likelihood of sustainability. Recommended training topics included: materials development, multigrade teaching, assessment/scoring, reading corners/book borrowing, teaching non-Lao-Tai speakers, identifying students with disabilities, refresher on core teaching techniques, and incorporating more writing exercises into Lao language lessons. LTR should conduct intensive Continuous Professional Development (CPD) workshops allotting more time for each specific topic and for teaching practice. For sustainability and to support new teachers, LTR may consider developing additional reference guides (e.g., handbook of effective teaching methods for reading instruction and inclusive education), videos of best practices, and virtual coaching and assessment. ● LTR should provide more advance notice to stakeholders for LTR activities. All stakeholders requested at least a week’s notice for trainings, meetings, and joint school monitoring visits. While a week may suffice for schools, VEDCs, and parents, due to busy schedules and competing priorities, MoES requires several months’ advance notice. Where LtR relies on DESB to inform communities, it must stive to work with local partners to ensure it gives sufficient notice. ● Aligned with the LTR Cooperative Agreement, LTR should conduct a capacity assessment of MoES staff and develop or modify a sustainability plan for Phase 3 along with benchmarks for tracking progress and milestones. The sustainability plan should focus on transferring leadership responsibility to MoES in Phase 3, and on building the capacity of DESB officers for training, coaching, and monitoring LTR trained teachers. LTR Project Officers and other LTR district staff should be involved in developing the sustainability plan and conducting the capacity assessment, as POs had excellent recommendations for increasing sustainability. Since POs work closely with DESB officers, they can provide insight into what capacity needs to be built or strengthened. Once the sustainability plan is approved by USAID and MoES, LTR should roll out a communication strategy for ensuring all MoES stakeholders are aware of the planned activities and benchmarks for tracking progress and milestones. By the end of LTR, USAID, MoES and LTR should know where they are in terms of phases and milestones against planned results. 28 Lao PDR Education COVID-19 Response Plan http://www.moes.edu.la/moes/images/Argreement/Lao_PDR_Education_COVID-19_Response_Plan.pdf 64 | FINAL EVALUATION REPORT USAID.GOV ● LTR should integrate recommended activities from stakeholders into the sustainability plan. The most critical activities to consider are: building capacity of MoES, PESS, DESB for monitoring and assessment; strengthening community engagement in learning and materials development; developing a network of trainers at TTC, district and school level, and creating an extensive library of TLMs leveraging Khang Panya Lao resources, public-private partners for publishing or donating books, and materials developed through CE and story writing competitions. LTR should collaborate with other donors (UNICEF/GPE) to expand access to new storybooks and TLMs as well as provide training on Khang Panya Lao for sustained access and printing of new materials. To increase sustainability within the MoES, LTR should disseminate LTR TLMs and quarterly reports to the ECE Department of Kindergarten and report progress against result indicators to RIES. ● LTR should strengthen its M&E team capacity and system to produce accurate data on activity indicators and outcomes. The M&E staff could conduct an EGRA or other reading assessment to assess reading outcomes for Lao and non-Lao speakers, which could potentially be carried out with Group 3 pending school re-opening.29 It is critical for the credibility and sustainability of the Activity that they have accurate quantitative data for tracking progress achieved to date against indicators and targets. If not possible due to COVID-19 school closures, then LTR may need a proxy indicator, such as number of books read and other indicators to show improved reading habits at home. The final evaluation should include a statistical report of progress achieved for the MoES to understand the indicators, achievements and gaps that will need to be addressed in the future beyond LTR. ● For ensuring sustainability, LTR must link the CE implementation and monitoring strategy to the MoES development plan. In order for the CE component to be sustained, the CE strategy needs to be integrated into MoES education plans and activities, so that they can take ownership. LTR should link the CE strategy to the MoES development plan and other development partner strategies. In doing so, it may be easier to engage VEDCs in the implementation of activities and DESB officers in monitoring CE results. EQ3 RECOMMENDATIONS ● LTR must involve the village authority in LTR orientations and community engagement activities. In order to increase engagement of VEDCs, particularly in the south, it is imperative that LTR work with DESB and Principal training to engage the Village Authority more effectively who can disseminate information to the villagers and mobilize communities to support reading. VEDCs who were not satisfied with LTR stressed the need to start orientation and dialogue with the Village Chief, following a best practice implemented by UNICEF. ● LTR should organize meetings with VEDCs and parents in their own language and around their schedules with sufficient notice. Consistent with findings and recommendations in EQ1 and EQ2, LTR should conduct orientations and trainings (or refresher trainings) for parents and VEDCs. Parents and VEDCs need to better understand the LTR objectives, the reading corner activities (e.g., that they can take books home and won’t be held liable for wear and tear), and how to support their children’s reading at home. Despite their busy schedules, it is clear based on successes in VTE and parents’ requests in CPS that if parents were properly oriented and provided with resources for teaching reading at home (e.g., stationary, 29 As mentioned previously, the continuing COVID-19 restrictions have made baseline data collection impossible. 65 | FINAL EVALUATION REPORT USAID.GOV storybooks, flashcards, etc.) that they would be more engaged in reading activities. Evidence from VTE suggests that lack of training and orientation in CPS is the primary reason for the lower participation. In order to increase parent participation in LTR activities, LTR should organize parent engagement activities in CPS around their schedules and provide sufficient notice. ● Conduct exchanges with VTE and CPS to transfer lessons learned and best practices for CE from the north to the south. Considering the differences in parent and VEDC engagement in the north and south, LTR may consider an exchange with DESB officers, teachers, and principals to understand how to better engage parents. Some principals have also requested a refresher training to understand how to engage parents and VEDCs. ● LTR should involve siblings in LTR reading activities for non-Lao-Tai speaking students/parents. Apart from targeting parents/caregivers for CE activities, LTR should also involve older siblings because many teachers mentioned that PP, G1, and G2 students receive support from their older brothers/sisters to read, particularly if the parents do not read in Lao language. This recommendation is also aligned with the GIDAP strategy that states, “parents will be informed about the benefits of older siblings reading to young siblings, which improves reading proficiency of both siblings30.” ● LTR should provide additional support required for increasing participation and accelerating sustainability. Consistent with earlier recommendations to ensure the potential for sustainability, LTR should provide additional storybooks/TLMs, teacher training and exchanges, at-home learning materials and training for parents (COVID-response), and advance notice for monitoring visits. USAID and LTR should also consider including other critical departments of the MoES in monitoring activities (e.g., statistics and teacher development unit). EQ4 RECOMMENDATIONS ● LTR should continue implementing, refining, and scaling up RRP. In collaboration with the MoES and GPE, LTR should refine the RRP based on World Bank impact evaluation findings, feedback from coaches, and GPE suggestions to incorporate assessment and parent engagement strategies. LTR should work with UNICEF and GPE on the roll out of the RRP and the pre-primary curriculum in kindergarten, pre-primary classrooms and TTCs. ● LTR/USAID should collaborate with GPE, BEQUAL and UNICEF on COVID￾response. COVID-response activities recommended for LTR/GPE collaboration are: Read at Home initiative, remedial education, and use of the online teaching and learning platform for accessing the pre-primary curriculum and other TLMs. UNICEF COVID-response activities recommended for LTR/UNICEF collaboration include: take home learning materials, catch-up program, blended trainings for ECE pre-primary curriculum, and digital offline resources. BEQUAL’s COVID-response strategy is not yet established, but will include inclusive education strategies; thus, LTR should also dialogue with BEQUAL about the potential for collaboration. All development partners plan to use the Khang Panya Lao teaching and learning platform as a central database for storing MoES curricula and supplementary materials. This platform is another area to explore for collaboration among LTR and development partners. ● USAID/LTR should collaborate with UNICEF on ECE materials development, training, and community engagement. UNICEF’s proposed activities are well aligned with 30 USAID/Save the Children. (2019). 66 | FINAL EVALUATION REPORT USAID.GOV the needs and gaps identified in this LTR midterm evaluation. For instance, development of quality ECE learning materials and multi-media for Lao language acquisition and non-Lao-Tai speakers, including children with disabilities, is most relevant to LTR. The component includes reading corners, storybooks, games and TLMs. The next step, once the program is funded, is for LTR and UNICEF to work together on reviewing existing materials, identifying gaps in resources, and developing additional resources to implement in target districts (more storybooks, audio/visual resources, TLMs, etc.). As well, LTR and UNICEF can collaborate on strengthening community engagement approaches and developing materials for reading at home (e.g., bilingual picture books, storybooks aligned with the “My House” TV program or culturally relevant content), conducting reading campaigns, providing trainings for parents, and developing revolving community libraries. The bilingual picture books would assist non-Lao-Tai speaking parents with reading to their children while also learning the Lao language. ● USAID/LTR should further explore opportunities to collaborate with the MoES and BEQUAL on activities that complement LTR. Specific recommended BEQUAL activities that complement LTR are: the Spoken Lao Language Pilot, continuous professional development for the implementation of the new curriculum in G1 and G2, and continuing the enhancement of supporting materials for lesson plans for multigrade classes. Other areas for collaboration include: parent engagement, assessment, and incorporating LTR’s IE tools and trainings into BEQUAL trainings on the new curriculum. 67 | EVALUATION FINAL REPORT USAID.GOV ANNEX C: BIBLIOGRAPHY Centers for Disease Control and Prevention (CDC). (2021). How to Protect Yourself and Others https://www.cdc.gov/coronavirus/2019-ncov/community/large-events/considerations-for-events￾gatherings.html#anchor_1619540981216 LTR AMELP, February 28, 2021 LTR AMELP Annex 1: Indicator Tracking Table, April 12, 2019-2 Report on the data quality assessment of USAID/Learn to Read (LTR), Vientiane, Laos USAID. (2019). Learn to Read Fact Sheet https://www.usaid.gov/laos/fact-sheets/usaid-learn-to-read USAID. (2019). USAID Learn to Read Year 1, Quarter 1 Report (August 15 – December 31, 2018) USAID. (2019). USAID Learn to Read Year 1, Quarter 2 Report (January 1 – March 31, 2019) USAID. (2019). USAID Learn to Read Year 1, Quarter 3 Report (April 1 – June 30, 2019) USAID. (2019). USAID Learn to Read Year 2, Quarter 1 Report (October 1 – December 31, 2019) USAID. (2020). USAID Learn to Read Year 2, Quarter 2 Report (January 1 – March 31, 2020) USAID. (2020). USAID Learn to Read Year 2, Quarter 3 Report (April 1 – June 30, 2020) USAID. (2020). USAID Learn to Read Year 3, Quarter 1 Report (October 1 – December 30, 2020) USAID. (2021). USAID Learn to Read Year 3, Quarter 2 Report (January 1 – March 31, 2021) USAID/Laos. (2021). Terms of Reference: Mid-Term Evaluation of the USAID’s supported Learn to Read Activity USAID/Asia. (2018). Cooperative Agreement No. 72048618CA00002 USAID/Asia. (2018). Modification of Assistance No. P001 USAID/Save the Children. (2019). USAID Learn to Read Gender and Social Inclusion Analysis Report. USAID/Save the Children. (2019). USAID Learn to Read Gender Inclusive Development Action Plan USAID/Save the Children. (2019). USAID Learn to Read Monitoring, Evaluation, and Learning (MEL) Plan. USAID/Save the Children. (2020). USAID Learn to Read Year 1 Annual Report (August 15, 2018 – September 30, 2019) USAID/Save the Children. (2020). USAID Learn to Read Year 2 Annual Report (October 1, 2019 – September 30, 2020) 68 | FINAL EVALUATION REPORT USAID.GOV ANNEX D: DATA COLLECTION PROTOCOLS INTERVIEW PROTOCOL: USAID STAFF Stakeholder type: USAID Staff Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately one hour. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. [Focus Group Discussion participants only] In addition to the evaluation team, participants in focus group discussions will share their responses in a group setting. In order to respect all participants, we ask that all focus group participants agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Nonetheless, there is a risk that other discussion participants will repeat what is shared here today. Remember that you are free to refuse to answer any question. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefits to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately document your perspective. The audio recording will not be shared with anyone outside of our evaluation team. 69 | EVALUATION FINAL REPORT USAID.GOV Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 70 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Questions/Sub Questions Introduction To be recorded on note-taking template(s): Stakeholder type: ___________ Sex: __________ Location: __________ 1. To begin, please tell me about your involvement with the USAID LTR project. 2. What is your role with the project? 3. How long have you been involved with the project? 4. Which LTR activities and components have you participated in (master trainings, teacher trainings, coaching, monitoring and evaluation, materials development, community engagement, etc.) 5. Do you collaborate with other LTR development partners (BEQUAL or World Bank)? EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement of results – Gender and Inclusion Achievement to Date I would like to understand the status of the project and what have been the achievements or challenges to date. 6. What, in your opinion, what have been the major accomplishments of the LTR activity to date? 7. What have been the major challenges of the LTR activity to date? 8. How likely will the three key result areas, enumerated in the results framework, be achieved within the set timeframe? a. very likely b. likely c. unlikely d. very unlikely 9. What have been the successes and challenges in relation to integrating inclusive education into the LTR Activity? EQ1: Most and least successful components Activity Effectiveness Now, reflecting on the status of activities under each Result Area… Most/Least Successful Components 10. Which components of the three key result areas are perceived to have been the most effective/successful during implementation to date? a. What is driving the success of these components? 11. Which components, of the three key result areas, are perceived to have been least successful to date? a. What is hindering these components? 71 | EVALUATION FINAL REPORT USAID.GOV EQ2: Activity implementation/ operations factors hindering or supporting implementation Implementation Operations/Factors I would like to ask you about any factors, internal or external, that have impeded or supported the implementation of the Activity to date. Let’s begin with external factors. (Do not read examples. Only provide, if needed.) 12. What external factors, if any, have impeded the implementation of the project (e.g., COVID-19, budget constraints due to TIPS)? 13. Are there any external factors that have supported the implementation of the Activity? If yes, what, and how? 14. What internal factors, if any, have hindered the implementation of the project (e.g., management/operations/human resources)? 15. Are there any internal factors that have supported the implementation of the Activity? If yes, what, and how? 16. How has the project adapted to address these unforeseen challenges? a. Has it changed significantly from its original design? If so, how? b. In your opinion, how will these modifications affect the sustainability of the project? 17. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed or improved to increase the likelihood of sustainability? EQ3: Stakeholder Dynamics MOES Collaboration I am interested to understand how the Activity collaborates with the MoES at all levels and how effective the engagement has been in building capacity and sustainability of LTR. 18. Could you please provide examples of how the LTR Activity has engaged with the MoES at the central, provincial and district levels to meet the Activity objectives (e.g., reading taskforce, master trainers, coaches, monitoring and supervision, M&E, etc.)? 19. Which types of engagement have been the most effective? 20. Do you think that the LTR project will be sustained within all levels of the MoES by the end of the project? Why or why not? 21. What is needed to ensure the sustainability of the project by the end of its term? EQ4: Current LTR partners (WORLD BANK and BEQUAL) BEQUAL/World Bank I am interested to understand how the Activity collaborates with other development partners’ supported programs, such as the World Bank Global Education Partnership (GPE) and the Basic Education Quality and Access in Lao PDR project (BEQUAL). 22. How has LTR been collaborating with the World Bank and BEQUAL thus far in the project? 23. Which activities, if any, of the World Bank and BEQUAL are relevant and complementary to LTR’s activity and would be worthy of replication to accelerate the achievement of LTR’s objectives? 24. Are there any World Bank or BEQUAL organizational or programmatic strengths that could be leveraged during the remaining period of LTR? 72 | FINAL EVALUATION REPORT USAID.GOV Conclusion 25. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything that I did not cover that you would like to add? 26. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.) THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 73 | EVALUATION FINAL REPORT USAID.GOV INTERVIEW PROTOCOL: SC STAFF Stakeholder type: SC Staff Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 90 minutes. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 74 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Questions/ Sub Questions Introduction To be recorded by note-taker: Stakeholder type: ____________ Sex. ____ Location: ______________________ To begin, please tell me about your involvement with the USAID LTR project. 1. What is your role with the project? 2. How long have you been involved with the project? 3. Which LTR activities and components have you participated in (master trainings, coaching, monitoring and evaluation, materials development, community engagement, etc.)? 4. Do you or your organization belong to any steering committees with the MoES or Technical Working Groups (ECE, Basic Education Focal Group) under the Education Sector Working Group? If yes, which ones and how does this TWG interact with the project? How often do they meet? 5. Do you collaborate with other LTR development partners (BEQUAL or World Bank)? If yes, for which result area and in what capacity? 6. Do you travel to the provinces, districts, and schools? If yes, how often and for what purposes? 75 | EVALUATION FINAL REPORT USAID.GOV EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement to Date I would like to understand the status of the project and what have been the achievements or challenges to date. (Note: List of materials and activities to date will be generated, shared in advance, and then confirmed during interview to save time. ET will refer to the list of activities provided with the district recommendations.) 7. What, in your opinion, what have been the major accomplishments of the LTR activity to date? 8. What materials has the project developed under Result 1? a. What materials are remaining to be developed? b. Is there a plan to complete these materials by the end of the project? Is it feasible? Why or why not? 9. What challenges has the project experienced under Result 1? 10. What teacher-training workshops have been completed under Result 2? a. What training workshops are remaining? b. What other workshops have been completed (master-trainers, coaches, pedagogical advisors, school principals)? c. What other workshops are remaining? d. Is there a plan to complete these workshops by the end of the project? Is it feasible? Why or why not? 11. What are the challenges the project has faced under Result 2? 12. What activities has the project conducted under Result 3, community engagement? a. What activities are remaining? b. Is there a plan to complete these activities by the end of the project? Is it feasible? Why or why not? 13. What challenges has the project encountered under Result 3? 14. How likely will the three key result areas, enumerated in the results framework, be achieved within the set timeframe? a. very likely b. likely c. unlikely d. very unlikely 76 | FINAL EVALUATION REPORT USAID.GOV EQ1: Progress towards Activity Objectives Achievement of results – Gender and Inclusion Gender and Inclusion (GESI Specialist only) 15. Did your organization receive any training specific to gender and/or diversity (e.g., Gender 101 training)? a. If yes, was the training useful to you? b. Have you incorporated any of this learning into your work? If so, how? 16. What actions from the GIDAP have been completed to date to incorporate gender and inclusion into the design, implementation, staffing, MEL, partnerships, and other aspects of the project? a. How is LTR accommodating the specific learning needs of girls, children with mild disabilities, and ethnic groups? 17. What have been the successes and challenges in relation to integrating inclusive education into LTR programming? EQ1: Most and least successful components Activity Effectiveness Now, reflecting on the status of activities under each Result Area. Most/Least Successful Components 18. Which components of the three key result areas are perceived to have been the most effective/successful during implementation to date? a. What is driving the success of these components? 19. Which components, of the three key result areas, are perceived to have been least successful to date? 20. What is hindering these components? 77 | EVALUATION FINAL REPORT USAID.GOV EQ2: Activity implementation/ operations factors hindering or supporting implementation Implementation/Operations Factors I would like to ask you about any factors, internal or external, that have impeded or supported the implementation of the Activity to date. Let’s begin with external factors. 21. What external factors, if any, have impeded the implementation of the project (e.g., COVID-19, budget constraints due to TIPS)? 22. Are there any external factors that have supported the implementation of the Activity? If yes, what, and how? 23. What internal factors, if any, have hindered the implementation of the project (e.g., management/operations/human resources)? 24. Are there any internal factors that have supported the implementation of the Activity? If yes, what, and how? 25. How has the project adapted to address the unforeseen challenges? a. Has it changed significantly from its original design? If so, how? b. In your opinion, how will these modifications affect the sustainability of the project? 26. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed or improved to increase the likelihood of sustainability? EQ3: Stakeholder Dynamics MOES Collaboration I am interested to understand how the Activity collaborates with the MoES at all levels and how effective the engagement has been in building capacity and sustainability of LTR. 27. Could you please provide examples of how the MoES is engaged at each level of implementation and what roles they play in the project (e.g., reading taskforce, master trainers, coaches, monitoring and supervision, M&E, etc.) 28. Which types of engagement have been the most effective? 29. Could you please describe the phased approach for gradual transfer of leadership and what milestones, or progress has been achieved to date? a. In what phase are you in? What are the indicators for measuring progress? b. What have been the challenges and lessons learned from implementing this phased approach? c. Has the plan for sustainability changed since the design phase? d. Is there anything that could be improved to build their capacity for self-reliance and support for the project? 30. Do you think that the LTR project will be sustained within all levels of the MoES by the end of the project? Why or why not? 31. What is needed, if anything, to ensure sustainability of the Activity by the end of its term? 78 | FINAL EVALUATION REPORT USAID.GOV EQ4: Current LTR partners (World Bank and BEQUAL) BEQUAL/World Bank I am interested to understand how the Activity collaborates with other development partners’ supported programs, such as the World Bank second Global Education Partnership (World Bank) and the Basic Education Quality and Access in Lao PDR project (BEQUAL). 32. How has LTR been collaborating with World Bank and BEQUAL thus far in the project? 33. Which activities, if any, of the World Bank and BEQUAL are relevant and complementary to LTR’s activity and would be worthy of replication to accelerate the achievement of LTR’s objectives? 34. Are there any World Bank or BEQUAL organizational or programmatic strengths that could be leveraged during the remaining period of LTR? Conclusion 35. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything that we did not cover that you would like to add? 36. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.)? THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 79 | EVALUATION FINAL REPORT USAID.GOV INTERVIEW PROTOCOL: RTR AND HI STAFF Stakeholder type: RTR and HI Staff Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 90 minutes. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 80 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Question/ Sub Questions Introduction: To be collected from participant(s): Stakeholder type, Sex, Location To begin, please tell me about your involvement with the USAID LTR project. 1. What is your role with the project? 2. How long have you been involved with the project? 3. What is the role of RtR/HI in the project? How does it differ from SC’s role? Do you lead certain activities or implement in different areas? 4. Which LTR activities and components have you participated in (master trainings, coaching, monitoring and evaluation, materials development, community engagement, etc.)? 5. Do you or your organization belong to any steering committees with the MoES or Technical Working Groups under the Education Sector Working Group? If yes, which ones and how does this TWG interact with the project? 6. Do you collaborate with other LTR development partners (BEQUAL or World Bank)? 7. Do you travel to the provinces, districts, and schools? If yes, how often and for what purposes? 81 | EVALUATION FINAL REPORT USAID.GOV EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement to Date: I would like to understand the status of the project and what have been the achievements or challenges to date. (Note: List of materials and activities to date will be generated, shared in advance, and then confirmed during interview to save time. ET will refer to the list of activities provided with the district recommendations.) 8. What, in your opinion, what have been the major accomplishments of the LTR activity to date? 9. What materials has the project developed under Result 1? a. What materials are remaining to be developed? b. Is there a plan to complete these materials by the end of the project? Is it feasible? Why or why not? 10. What challenges has the project experienced under Result 1? 11. What teacher-training workshops have been completed under Result 2? 1. What training workshops are remaining? 2. What other workshops have been completed (master-trainers, coaches, pedagogical advisors, school principals)? 3. What other workshops are remaining? 4. Is there a plan to complete these workshops by the end of the project? Is it feasible? Why or why not? 12. What are the challenges the project has faced under Result 2? 13. What activities has the project conducted under Result 3, community engagement? 1. What activities are remaining? 2. Is there a plan to complete these activities by the end of the project? Is it feasible? Why or why not? 14. What challenges has the project encountered under Result 3? 15. How likely will the three key result areas, enumerated in the results framework, be achieved within the set timeframe? a. very likely b. likely c. unlikely d. very unlikely EQ1: Progress towards Activity Objectives Achievement of results – Gender and Inclusion Gender and Inclusion 16. Did your organization receive any training specific to gender and/or diversity (Gender 101 training)? 17. If yes, was the training useful to you? 18. Have you incorporated any of this learning into your work? If so, how? 19. What have been the successes and challenges in relation to integrating gender and inclusion into LTR programming? 82 | FINAL EVALUATION REPORT USAID.GOV EQ1: Most and least successful components Activity Effectiveness Now, reflecting on the status of activities under each Result Area… Most/Least Successful Components 20. Which components of the three key result areas are perceived to have been the most effective/successful during implementation to date? a. What is driving the success of these components? 21. Which components, of the three key result areas, are perceived to have been least successful to date? a. What is hindering these components? EQ2: Activity implementation/ operations factors hindering or supporting implementation Implementation/Operations Factors I would like to ask you about any factors, internal or external, that have impeded or supported the implementation of the Activity to date. Let’s begin with external factors. 22. What external factors, if any, have impeded the implementation of the project (e.g., COVID-19, budget constraints due to TIPS)? 23. Are there any external factors that have supported the implementation of the Activity? If yes, what, and how? 24. What internal factors, if any, have hindered the implementation of the project (e.g., management/operations/human resources)? 25. Are there any internal factors that have supported the implementation of the Activity? If yes, what, and how? 26. In your opinion, how has the project adapted to address the unforeseen challenges? a. Has it changed significantly from its original design? If so, how? b. How will these modifications affect the sustainability of the project? 27. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed or improved to increase the likelihood of sustainability? 83 | EVALUATION FINAL REPORT USAID.GOV EQ3: Stakeholder Dynamics MOES satisfaction MOES Collaboration I am interested to understand how the Activity collaborates with the MoES at all levels and how effective the engagement has been in building capacity and sustainability of LTR. 28. Could you please provide examples of how the MoES is engaged at each level of implementation and what roles they play in the project (e.g., reading taskforce, master trainers, coaches, monitoring and supervision, M&E, etc.) 29. Which types of engagement have been the most effective? 30. Could you please describe the phased approach for gradual transfer of leadership and what has been achieved to date? a. What have been the challenges and lessons learned from implementing the phased approach? b.Is there anything that could be improved to build their capacity for self-reliance and support for the project? 31. Do you think that the LTR project will be sustained within all levels of the MoES by the end of the project? Why or why not? 32. What is needed, if anything, to ensure sustainability of the Activity by the end of its term? EQ: Stakeholder Dynamics Sub-contractor satisfaction 33. Are you and your organization satisfied with your level of participation in the LTR activity? Do you/your organization perceive that your expectations have been met by LTR to date? 34. What is needed, if anything, to improve the level of engagement and ensure sustainability of the Activity by the end of the term? EQ4: Current LTR partners (World Bank and BEQUAL) BEQUAL/World Bank I am interested to understand how the Activity collaborates with other development partners’ supported programs, such as the World Bank second Global Education Partnership and the Basic Education Quality and Access in Lao PDR project (BEQUAL). 35. How has LTR been collaborating with World Bank and BEQUAL thus far in the project? 36. Which activities, if any, of the World Bank and BEQUAL are relevant and complementary to LTR’s activity and would be worthy of replication to accelerate the achievement of LTR’s objectives? 37. Are there any World Bank or BEQUAL organizational or programmatic strengths that could be leveraged during the remaining period of LTR? 84 | FINAL EVALUATION REPORT USAID.GOV CONCLUSION 38. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything we did not cover that you would like to add? 39. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.)? THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 85 | EVALUATION FINAL REPORT USAID.GOV INTERVIEW PROTOCOL: MOES STAKEHOLDERS Stakeholder type: MoES Stakeholders Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 60 minutes. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 86 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Questions/ Sub Questions Introduction To be recorded by note-taker(s): Stakeholder type: _____ Sex: _________ Location: _________ To begin, please tell me about your involvement with the USAID LTR project. 1. What is your role with the project? 2. How long have you been involved with the project? 3. Do you belong to any steering committees with the MoES or Technical Working Groups under the Education Sector Working Group? If yes, which ones and how does this TWG interact with the project? 4. Which LTR activities and components have you participated in (master trainings, coaching, monitoring and evaluation, materials development, community engagement, GESI trainings, etc.)? 5. Do you collaborate with other LTR development partners (BEQUAL or World Bank)? (If not, skip related questions) 6. Do you travel to USAID/LTR-supported provinces, districts, and schools? If yes, how often and for what purposes? EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement of results – Gender and Inclusion Achievement to Date I would like to understand the status of the project and what have been the achievements or challenges to date. (Note to interviewer: Only ask questions for results which the interviewer has been engaged.) 7. What, in your opinion, what have been the major accomplishments of the LTR activity to date? 8. What materials has the project developed under Result 1: supplemental materials developed that you are aware of or were involved in developing? 9. What challenges has the project experienced under Result 1? 10. What teacher training workshops have been completed under Result 2: Teacher Training that you are aware of or were involved in? 11. What are the challenges the project has faced under Result 2? 12. What activities has the project conducted under Result 3, community engagement that you are aware of or were involved in? 13. What challenges has the project encountered under Result 3? 14. How likely will the three key result areas, enumerated in the results framework, be achieved within the set timeframe? a. very likely b.likely c. unlikely d.very unlikely 15. If familiar with the IE pilot or any pilot: What successes or challenges has the project experienced in its pilot activities (Inclusive Education pilot, Caregiver engagement, reading corners) to date? 87 | EVALUATION FINAL REPORT USAID.GOV EQ1: Most and least successful components Activity Effectiveness Now, reflecting on the status of activities under each Result Area… Most/Least Successful Components 16. Which components of the three key result areas are perceived to have been the most effective/successful during implementation to date? . What is driving the success of these components? 17. Which components, of the three key result areas, are perceived to have been least successful to date? . What is hindering these components? EQ2: Activity implementation/ operations factors hindering or supporting implementation Implementation/Operations Factors I would like to ask you about any factors, internal or external, that have impeded or supported the implementation of the Activity. Let’s begin with external factors. 18. What external factors, if any, have impeded the implementation of the project (e.g., COVID-19, budget constraints due to TIPS)? 19. Are there any external factors that have supported the implementation of the Activity? If yes, what, and how? 20. What internal factors, if any, have hindered the implementation of the project (e.g., management/operations/human resources)? 21. Are there any internal factors that have supported the implementation of the Activity? If yes, what, and how? 22. How has the project adapted to address the unforeseen challenges? a. Has it changed significantly from its original design? If so, how? b.In your opinion, how will these modifications affect the sustainability of the project? 23. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed or improved to increase the likelihood of sustainability? 88 | FINAL EVALUATION REPORT USAID.GOV EQ2: Activity implementation/ operations factors hindering or supporting implementation MoES Priorities I am interested to understand how the MoES education priorities and how LTR aligns with the MoES goals and objectives? Can you please explain this to me? 24. Could you please explain MoES education priorities for reading, early primary education, and inclusion? 25. How does the LTR Activity support MoES policies and priorities given the new Education Sector Plan? 26. How has the Education Sector Plan and budget been impacted due to unforeseen challenges resulting from COVID-19? 27. Do you think that the disruption caused by COVID, and budget constraints will alter how much the USAID LTR project will be able to achieve during the lifespan of the Activity or whether it will be sustained? If yes, how so? EQ3: Stakeholder Dynamics MOES Collaboration I am interested to understand how the Activity collaborates with the MoES at all levels and how effective the engagement has been in building capacity and sustainability of LTR. 28. Could you please provide examples of how the MoES is engaged at each level of implementation and what roles they play in the project (e.g., reading taskforce, master trainers, coaches, monitoring and supervision, M&E, etc.)? 16. Which types of engagement have been the most effective? 29. Are you aware of the phased approach for gradual transfer of leadership and sustainability? a. If yes, what milestones have been achieved to date and what phase are you in? b.What have been the challenges and lessons learned from implementing the phased approach? 30. In your opinion, are you satisfied with your level of participation in the LTR project? Have your expectations been met? 31. Do you perceive that others’ expectations have been met by LTR to date? Why or why not? 32. Is there anything that could be improved to build your or others’ capacity to continue implementing LTR activities in the future? 33. Do you think that the LTR project will be sustained within all levels of the MoES by the end of the project? Why or why not? What is needed to ensure the sustainability of the project by the end of its term? 89 | EVALUATION FINAL REPORT USAID.GOV EQ4: Current LTR partners (World Bank and BEQUAL)EQ3: Stakeholder Dynamics BEQUAL/World Bank I am interested to understand how the Activity collaborates with other development partners’ supported programs, such as the second Global Education Partnership and the Basic Education Quality and Access in Lao PDR project (BEQUAL). (If the MoES respondent is not familiar, skip to #37). 34. How has LTR been collaborating with World Bank and BEQUAL thus far in the project? 35. What other types of activities do the World Bank and BEQUAL projects implement that support reading achievement of children in pre-primary and grades 1-2? 36. Which activities, if any, of World Bank and BEQUAL are relevant and complementary to LTR’s activity and would be worthy of replication to accelerate the achievement of LTR’s objectives? Are there any World Bank or BEQUAL organizational or programmatic strengths that could be leveraged during the remaining period of LTR? MOES Collaboration I am interested to understand how the Activity collaborates with the MoES at all levels and how effective the engagement has been in building capacity and sustainability of LTR. 37. Could you please provide examples of how the MoES is engaged at each level of implementation and what roles they play in the project (e.g., reading taskforce, master trainers, coaches, monitoring and supervision, M&E, etc.)? 17. Which types of engagement have been the most effective? 38. Are you aware of the phased approach for gradual transfer of leadership and sustainability? c. If yes, what milestones have been achieved to date and what phase are you in? d.What have been the challenges and lessons learned from implementing the phased approach? 39. In your opinion, are you satisfied with your level of participation in the LTR project? Have your expectations been met? 40. Do you perceive that others’ expectations have been met by LTR to date? Why or why not? 41. Is there anything that could be improved to build your or others’ capacity to continue implementing LTR activities in the future? 42. Do you think that the LTR project will be sustained within all levels of the MoES by the end of the project? Why or why not? What is needed to ensure the sustainability of the project by the end of its term? 90 | FINAL EVALUATION REPORT USAID.GOV EQ4: Current LTR partners (World Bank and BEQUAL)EQ3: Stakeholder Dynamics Conclusion 43. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover that you would like to add? 44. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.) BEQUAL/World Bank I am interested to understand how the Activity collaborates with other development partners’ supported programs, such as the second Global Education Partnership and the Basic Education Quality and Access in Lao PDR project (BEQUAL). (If the MoES respondent is not familiar, skip to #37). 45. How has LTR been collaborating with World Bank and BEQUAL thus far in the project? 46. What other types of activities do the World Bank and BEQUAL projects implement that support reading achievement of children in pre-primary and grades 1-2? 47. Which activities, if any, of World Bank and BEQUAL are relevant and complementary to LTR’s activity and would be worthy of replication to accelerate the achievement of LTR’s objectives? Are there any World Bank or BEQUAL organizational or programmatic strengths that could be leveraged during the remaining period of LTR? MOES Collaboration I am interested to understand how the Activity collaborates with the MoES at all levels and how effective the engagement has been in building capacity and sustainability of LTR. 48. Could you please provide examples of how the MoES is engaged at each level of implementation and what roles they play in the project (e.g., reading taskforce, master trainers, coaches, monitoring and supervision, M&E, etc.)? 18. Which types of engagement have been the most effective? 49. Are you aware of the phased approach for gradual transfer of leadership and sustainability? e.If yes, what milestones have been achieved to date and what phase are you in? f. What have been the challenges and lessons learned from implementing the phased approach? 50. In your opinion, are you satisfied with your level of participation in the LTR project? Have your expectations been met? 51. Do you perceive that others’ expectations have been met by LTR to date? Why or why not? 52. Is there anything that could be improved to build your or others’ capacity to continue implementing LTR activities in the future? 53. Do you think that the LTR project will be sustained within all levels of the MoES by the end of the project? Why or why not? What is needed to ensure the sustainability of the project by the end of its term? 91 | EVALUATION FINAL REPORT USAID.GOV EQ4: Current LTR partners (World Bank and BEQUAL)EQ3: Stakeholder Dynamics Conclusion 54. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover that you would like to add? 55. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.) BEQUAL/World Bank I am interested to understand how the Activity collaborates with other development partners’ supported programs, such as the second Global Education Partnership and the Basic Education Quality and Access in Lao PDR project (BEQUAL). (If the MoES respondent is not familiar, skip to #37). 56. How has LTR been collaborating with World Bank and BEQUAL thus far in the project? 57. What other types of activities do the World Bank and BEQUAL projects implement that support reading achievement of children in pre-primary and grades 1-2? 58. Which activities, if any, of World Bank and BEQUAL are relevant and complementary to LTR’s activity and would be worthy of replication to accelerate the achievement of LTR’s objectives? Are there any World Bank or BEQUAL organizational or programmatic strengths that could be leveraged during the remaining period of LTR? MOES Collaboration I am interested to understand how the Activity collaborates with the MoES at all levels and how effective the engagement has been in building capacity and sustainability of LTR. 59. Could you please provide examples of how the MoES is engaged at each level of implementation and what roles they play in the project (e.g., reading taskforce, master trainers, coaches, monitoring and supervision, M&E, etc.)? 19. Which types of engagement have been the most effective? 60. Are you aware of the phased approach for gradual transfer of leadership and sustainability? g. If yes, what milestones have been achieved to date and what phase are you in? h.What have been the challenges and lessons learned from implementing the phased approach? 61. In your opinion, are you satisfied with your level of participation in the LTR project? Have your expectations been met? 62. Do you perceive that others’ expectations have been met by LTR to date? Why or why not? 63. Is there anything that could be improved to build your or others’ capacity to continue implementing LTR activities in the future? 64. Do you think that the LTR project will be sustained within all levels of the MoES by the end of the project? Why or why not? 92 | FINAL EVALUATION REPORT USAID.GOV What is needed to ensure the sustainability of the project by the end of its term? EQ4: Current LTR partners (World Bank and BEQUAL) Conclusion 65. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover that you would like to add? 66. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.) BEQUAL/World Bank I am interested to understand how the Activity collaborates with other development partners’ supported programs, such as the second Global Education Partnership and the Basic Education Quality and Access in Lao PDR project (BEQUAL). (If the MoES respondent is not familiar, skip to #37). 67. How has LTR been collaborating with World Bank and BEQUAL thus far in the project? 68. What other types of activities do the World Bank and BEQUAL projects implement that support reading achievement of children in pre-primary and grades 1-2? 69. Which activities, if any, of World Bank and BEQUAL are relevant and complementary to LTR’s activity and would be worthy of replication to accelerate the achievement of LTR’s objectives? 70. Are there any World Bank or BEQUAL organizational or programmatic strengths that could be leveraged during the remaining period of LTR? Conclusion 71. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover that you would like to add? 72. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.) 73. THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 93 | EVALUATION FINAL REPORT USAID.GOV INTERVIEW PROTOCOL: PROVINCIAL AND DISTRICT STAFF Stakeholder type: Provincial and District Staff (PESS/DESB) Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 60 minutes. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 94 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Questions/ Sub Questions Introduction To be recorded by note-taker(s): Stakeholder type: _______ Sex: _________ Location: ______ To begin, please tell me about your involvement with the USAID LTR project. 1. What is your role with the project? 2. How long have you been involved with the project? 3. Which LTR activities and components have you participated in (master trainings, coaching, monitoring and evaluation, materials development, community engagement, GESI trainings, etc.)? If a master trainer, ask: a. What skills did you gain from the training? b. How have you been able to apply the skills you learned? What are you doing differently as a result of the training and support received by the USAID/LTR project? c. Have you faced any challenges in your role as a Master Trainer? If yes, what kinds of challenges? d. What recommendations do you have to address these challenges? 95 | EVALUATION FINAL REPORT USAID.GOV EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement of results – Gender and Inclusion Achievement to Date I would like to understand the status of the project and what have been the achievements or challenges to date. 4. What supplementary teaching and learning materials have the schools received from the LTR project (Result 1)? a. What impact has this had on teaching and learning? b. Have you encountered any challenges with receiving the LTR teaching and learning materials on schedule? If yes, what? 5. What teacher-training workshops have you been involved in, as a trainer, observer, or participant? a. How have teachers’ instruction changed since participating in the LTR training workshops? b. Have teachers faced any challenges in effectively implementing the LTR materials and methodologies in the classroom? 6. What types of community engagement activities have you participated in? a. Have you noticed any difference in parents’ support for their children’s education? If yes, what? b. Have you noticed any difference in parents’ involvement in literacy activities? If yes, in what types of activities are they more engaged c. Have you visited any homes and seen any changes as a result of community engagement activities (e.g., print rich environment, more children’s reading books)? d. Have there been any challenges or successes with schools or LTR engaging VEDCs, parents or caregivers in LTR reading activities (e.g., reading corners, teacher-parent conferences, reading festivals? 7. What, in your opinion, have been the major accomplishments of the LTR activity to date? a. Has LTR had an impact on students’ reading skills, particularly for non-Lao-Tai speakers and children with mild disabilities? b. If yes, how have students’ reading skills changed in pre￾primary, grades 1 and 2 as a result of project interventions? EQ1: Most and least successful components Activity Effectiveness Now, reflecting on the status of activities under each Result Area… 8. Among the three components of the LTR project (teaching and learning materials for non-Lao-Tai speakers and children with mild disabilities, teacher training and community engagement), which has been the most effective in improving children’s reading skills to date? a. Probe for specifics: What teaching methods, resources or community-based activities have had the most impact on students’ reading skills? b. What is driving the success of these components? 9. Which of the components have been the least effective to date? a. What is hindering these components? 96 | FINAL EVALUATION REPORT USAID.GOV EQ2: Activity implementation/ operations factors hindering or supporting implementation Implementation/ Operations Factors I would like to ask you about any factors, internal or external, that have impeded or supported the implementation of the Activity to date. Let’s begin with external factors. 10. What external factors, if any, have impeded or supported the implementation of the project (e.g., COVID-19, budget constraints due to TIPS)? 11. What internal factors, if any, have hindered or supported the implementation of the project (e.g., management/operations/human resources)? 12. Are there any program areas of the project that could be changed or improved, including: governance, management, monitoring and evaluation to increase the likelihood of sustainability? EQ2: Activity implementation/ operations factors hindering or supporting implementation Impact of COVID-19 I am interested to understand how COVID-19 impacted the USAID LTR Activity. 13. How has COVID-19 impacted learning at school, home and in the community in the LTR-supported schools? 14. Do you have any recommendations for how the LTR project could support teachers and students to continue learning during a COVID￾lockdown or brief interruption to schooling in the future? Or, for how to conduct trainings and coaching in the future in both urban and rural areas? 97 | EVALUATION FINAL REPORT USAID.GOV EQ3: Stakeholder Dynamics PESS/DESB Collaboration I am interested to understand how the Activity collaborates with provincial and district officers and how effective the engagement has been in building capacity and sustainability of LTR. 15. Could you please provide examples of how you have been involved in LTR activities and what roles you have played (e.g., trainer, coach, TLC facilitator, collecting data from classroom observations and assessments, supporting community engagement activities, etc.)? (Read examples and probe for others) 16. Are you aware of the phased approach for gradual transfer of leadership and sustainability? a. If yes, what milestones have been achieved to date and what phase are you in? b. What have been the challenges and lessons learned from implementing the phased approach? 17. Are you satisfied with your level of participation in the project? a. Have your expectations been met by LTR to date? Why or why not? b. Is there anything that could be improved to build your capacity and support for the project? 18. Do you think that the LTR project will be sustained within your PESS/DESB office? Why or why not? 19. What is needed to ensure the sustainability of the project by the end of its term? Conclusion 20. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover that you would like to add? 21. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.) THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 98 | FINAL EVALUATION REPORT USAID.GOV INTERVIEW PROTOCOL: PEDAGOGICAL ADVISORS/SCHOOL PRINCIPALS Stakeholder type: Pedagogical Advisors and School Principals Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 60 minutes. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 99 | EVALUATION FINAL REPORT USAID.GOV Evaluation Question Questions/ Sub Question INTRODUCTION To be by the note-taker: Stakeholder type: ______________ Sex ________________ Location: ___________________ To begin, please tell me about your involvement with the USAID LTR project. 1. What is your role with the project? 2. How long have you been involved with the project? 3. Which LTR activities and components have you participated in (master trainings, coaching, monitoring and evaluation, materials development, community engagement, GESI trainings, etc.)? If a coach, ask: a. What skills did you gain from the training? b. How have you been able to apply the skills you learned? What are you doing differently as a result of the training and support received by the USAID/LTR project? c. Do you conduct classroom observations and provide coaching to LTR trained teachers? If yes: d. For how many schools and how often? e. Could you please describe the coaching activities that take place when you visit the school? f. Have you participated in any teacher learning circles? If yes, what have been the results? g. Have you noticed any improvement in teaching approaches and use of materials over the course of the professional development program? If yes, what has changed? h. Have you faced any challenges your role as a Literacy Coach? If yes, what kinds of challenges? 100 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Questions/ Sub Question EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement of results – Gender and Inclusion Achievement to Date: I would like to understand the status of the project and what have been the achievements or challenges to date. 4. What supplementary teaching and learning materials have the schools/your school received from the LTR project (Result 1)? a. What impact has this had on teaching and learning? b. Have you encountered any challenges with receiving the LTR teaching and learning materials on schedule? If yes, what? 5. What LTR teacher-training workshops have you been involved in, as a trainer, observer, or participant? a. How have teachers’ instruction changed since participating in the LTR training workshops? b. Have teachers faced any challenges in effectively implementing the LTR materials and methodologies in the classroom? 6. What types of community engagement activities have you participated in? a. Have you noticed any difference in parents’ support for their children’s education? If yes, what? b. Have you noticed any difference in parents’ involvement in literacy activities? If yes, in what kinds of activities are they more engaged? c. Have you visited any homes and seen any changes as a result of community engagement activities (e.g., print rich environment, more children’s reading books)? d. Have there been any challenges or successes with schools and LTR engaging VEDCS, parents or caregivers in reading activities (e.g., borrowing books from the reading corners, teacher￾parent conferences, reading festivals)? 7. What, in your opinion, have been the major accomplishments of the LTR activity to date? a. Has LTR had an impact on students’ reading skills, particularly for non-Lao-Tai speakers and children with mild disabilities? b. If yes, how have students’ reading skills changed in pre-primary, grades 1 and 2 as a result of project interventions? 101 | EVALUATION FINAL REPORT USAID.GOV Evaluation Question Questions/ Sub Question EQ1: Most and least successful components Activity Effectiveness Now, reflecting on the status of activities under each Result Area: 8. Among the three components of the LTR project (teaching and learning materials for non-Lao-Tai speakers and children with mild disabilities, teacher training and community engagement), which has been the most effective in improving children’s reading skills to date? a. Probe for specifics: What teaching methods, resources or community-based activities have had the most impact on students’ reading skills? b. What is driving the success of these components? 9. Which of the components have been the least effective to date? a. What is hindering these components? EQ2: Activity implementation/ operations factors hindering or supporting implementation Implementation/ Operations Factors I would like to ask you about any factors, internal or external, that have impeded or supported the implementation of the Activity to date. Let’s begin with external factors. 10. What external factors, if any, have impeded or supported the implementation of the project (e.g., COVID-19, budget constraints due to TIPS)? 11. What internal factors, if any, have hindered or supported the implementation of the project (e.g., management/operations/human resources)? 12. Are there any program areas of the project that could be changed or improved, including: governance, management, monitoring and evaluation, to increase the likelihood of sustainability? EQ2: Activity implementation/ operations factors hindering or supporting implementation Impact of COVID-19 I am interested to understand how COVID-19 impacted the USAID LTR Activity. 13. How has COVID-19 impacted learning in your school and in the LTR￾supported schools and villages? 14. Do you have any recommendations for how the LTR project could support teachers and students to continue learning during a COVID￾lockdown or brief interruption to schooling in the future? Or, for how to conduct trainings and coaching in the future in both urban and rural areas? 102 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Questions/ Sub Question EQ3: Stakeholder Dynamics DESB/School Collaboration I am interested to understand how the Activity collaborates with provincial and district officers and how effective the engagement has been in building capacity and sustainability of LTR. 15. Could you please provide examples of how you have been involved in LTR activities and what roles you have played (e.g., trainer, coach, TLC facilitator, collecting data from classroom observations and assessments, supporting community engagement activities, etc.)? (Read examples and probe for others) 16. Are you satisfied with your level of participation in the project? a. Have your expectations been met by LTR to date? Why or why not? b. Is there anything that could be improved to build your capacity and support for the project? 17. Do you think that the LTR project will be sustained within your district/school? Why or why not? 18. What is needed to ensure the sustainability of the project by the end of its term? Conclusion 19. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there something that I did not cover, that you would like to add? 20. Are there any other recommendations you have to address the issues we discussed today that would improve the future implementation and sustainability of the Activity (e.g., COVID-19, budget constraints, etc.)? THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 103 | EVALUATION FINAL REPORT USAID.GOV INTERVIEW PROTOCOL: TEACHERS (PP, G1, G2) Stakeholder type: Teachers (Pre-primary, Grade 1, and Grade 2) Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 45 minutes. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 104 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Question/ Sub Questions Introduction To be recorded by the notetaker: Stakeholder type: _____________ Sex: _________ Location: ____________ To begin, please tell me about your involvement with the USAID LTR project. 1. How long have you been a teacher? 2. Are you from this village or community? 3. Do you speak the local language? (If multiple languages, ask which local languages do you speak?) 4. How long have you been teaching at this school? 5. Which LTR activities and components have you participated in (materials development, teacher training, community engagement, GESI trainings)? 105 | EVALUATION FINAL REPORT USAID.GOV EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement of results – Gender and Inclusion (7d) Achievement to Date I would like to understand the status of the project and what have been the achievements or challenges to date. 6. What supplementary teaching and learning materials have you received in your classroom that were developed by the USAID/LTR project (Result 1)? a. Could you describe what the situation was like before receiving these materials and how it has changed? b. Do you have a reading corner in your classroom that was established under the USAID/LTR activity? If yes, how has the availability of context-appropriate reading corners impacted students’ motivation to read and their reading fluency skills? c. Have you faced any challenges with using the LTR supplementary teaching and learning materials? If yes, what? Has the coach, principal or other teacher assisted with this? If yes, who? If not, why not? 7. What teacher-training workshops have you attended and when (Result 2)? a. Which training was the most effective? b. Was there anything that could have been improved? c. How has your instruction changed since participating in the LTR teacher-training workshops? d. What pedagogical approaches and methods are you using to address individual learning needs based on gender, language, and abilities? (GESI) e. What challenges have you faced to incorporate the new teaching and learning strategies into your classroom? f. Have you received any coaching support in the classroom from your principal or pedagogical advisor? g. If yes, could you describe how the coach supports you? Is it helpful? h. Do you participate in any teacher learning circles? If yes, is it helpful? i. If yes, how do you apply or build on what you’ve learned in LTR workshops during TLC meetings? j. Do you coach or train other teachers in the new methodology and materials received from the LTR project? (If yes, take note if it is voluntary or part of the peer-to-peer coaching model.) 8. What types of LTR community engagement activities have you participated in (Result 3)? (Skip, if not relevant) a. What is your role specifically in carrying out these activities? b. Have you noticed any difference in parents’ support for their children’s education? If yes, what? c. Have you noticed any difference in parents’ involvement in literacy activities? If yes, in what kinds of activities are they more engaged? d. Have you visited any homes and seen any changes as a result of community engagement activities (e.g., print rich environment, more children’s reading books)? 106 | FINAL EVALUATION REPORT USAID.GOV e. Have there been any challenges in conducting LTR activities with VEDCs, parents or caregivers (e.g., borrowing books from the reading corners, teacher-parent conferences, reading festivals)? 9. What, in your opinion, have been the major accomplishments of the LTR activity to date? a. Have you observed any improvement in students’ reading skills, particularly for non-Lao-Tai speakers and children with mild disabilities? b. If yes, how have students’ reading skills changed in pre￾primary, grades 1 and 2 as a result of project interventions? c. Do you have any reading test results or examples of what skills have improved as a result of LTR interventions (e.g., improved phonics, letter recognition, vocabulary, decoding, reading fluency and comprehension)? EQ1: Most and least successful components Activity Effectiveness Now, reflecting on the status of activities under each Result Area... 10. Among the three components of the LTR project (teaching and learning materials for non-Lao-Tai speakers and children with mild disabilities, teacher training and community engagement), which has been the most effective in improving children’s reading skills to date? a. Probe for specifics: What teaching methods, resources or community-based activities have had the most impact on students’ reading skills? b. What is driving the success of these components? 11. Which of the components have been the least effective to date? a. What is hindering these components? 107 | EVALUATION FINAL REPORT USAID.GOV EQ2: Activity implementation/ operations factors hindering or supporting implementation Implementation/ Operations Factors I would like to ask you about any factors, internal or external, that have impeded or supported the implementation of the Activity to date. Let’s begin with external factors. 12. What external factors, if any, have impeded or supported the implementation of the project (e.g., COVID-19, budget constraints due to TIPS)? 13. What internal factors, if any, have hindered or supported the implementation of the project (e.g., management/operations/human resources)? 14. Are there any program areas of the project that could be changed or improved, including: governance, management, monitoring and evaluation, to increase the likelihood of sustainability? EQ2: Activity implementation/ operations factors hindering or supporting implementation Impact of COVID-19 I am interested to understand how COVID-19 impacted the USAID LTR Activity. 16. How has COVID-19 impacted learning in your school, and village? 17. Do you have any recommendations for how the LTR project could support teachers and students to continue learning during a COVID￾lockdown or brief interruption to schooling in the future? EQ3: Stakeholder Dynamics School-community stakeholders 18. Are you satisfied with the level of training and coaching support you have received from the LTR project? Has the LTR project met your expectations for support and engagement? Why or why not? 19. Is there anything that the project should consider for sustaining these activities and continuing to improve reading skills of non-Lao-Tai speakers and children with mild disabilities? Conclusion 20. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover that you would like to add? 21. Are there any other recommendations you have to address - the issues we discussed today that would improve the future implementation and sustainability of the Activity? THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 108 | FINAL EVALUATION REPORT USAID.GOV FGD PROTOCOL: PARENTS/CAREGIVERS Stakeholder type: Parents/Caregivers Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 45 minutes in Lao or up to 90 minutes with translation. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 109 | EVALUATION FINAL REPORT USAID.GOV Evaluation Question Questions/ Sub Questions Introduction To be recorded by note-taker: Stakeholder type: ____________ Sex: _____________ Location: ________________ 1. Do you have a child in PP, G1 or G2? 2. What language do you speak at home? 3. Are any of you familiar with the USAID Learn to Read project and its goals and activities? a. The goal of LTR is to improve children’s reading skills in PP, G1 and G2, particularly for non-Lao-Tai speakers and children with mild disabilities. The project has three key components: 1) providing schools with supplemental materials (storybooks) for non-Lao-Tai speakers and children with mild disabilities and establishing reading corners, 2) training teachers in reading instruction and inclusive education, and 3) supporting parents of pre-primary students to help their children learn how to read in the local language and in Lao) EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement of results – Gender and Inclusion (7d) Achievement to Date 4. What LTR activities have you participated into date? a. Meeting to introduce reading corners and talk about the poster for supporting children’s reading (1 meeting in 2020-21 school year) b. Caregiver Engagement sessions, when you learned 12 activities to support your children’s reading (5 sessions in 2019-20 school year) c. Parent-teacher meeting d. Reading Festival e. Other 5. Have you done anything differently since participating in these activities? If yes, what? 6. Have you noticed any difference in your child’s ability to read in Lao? If yes, could you please provide an example? 7. How do you support your children’s reading at home? (e.g., borrow books from the reading corner at school, help with homework, read story books aloud, etc.) 110 | FINAL EVALUATION REPORT USAID.GOV EQ1: Most and least successful components Activity Effectiveness Most/Least Successful Components 8. Among the LTR activities that you have participated in, which has been the most effective in improving children’s reading skills to date? Why? 9. Which of the components or activities have been the least effective to date? Why? 10. Have you experienced any challenges with implementing any of the approaches learned through the project activities? 11. Are there any additional activities in the school or community needed to help improve your child’s reading skills? EQ2: Activity implementation/ operations factors hindering or supporting implementation Impact of COVID-19 I am interested to understand how COVID-19 impacted the USAID LTR Activity. 12. How has COVID-19 impacted learning in your school, home, and village? 13. What do you need to support your child to learn at home during a COVID lockdown when schools are closed? EQ3: Stakeholder Dynamics School-community stakeholders 14. Are you satisfied with your level of participation in LTR activities? Has the LTR project met your expectations for support and engagement? Why or why not? 15. Are you satisfied with the level of outreach and communication provided by the VEDC and other community volunteers? Is it effective? Why or why not? Conclusion 16. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover that you would like to add? 17. Are there any recommendations you have to address any of the issues we discussed today that would improve the future implementation and sustainability of the Activity? THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 111 | EVALUATION FINAL REPORT USAID.GOV FGD PROTOCOL: VEDC Stakeholder type: Village Education Development Committee (VEDC) Informed Consent Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the LTR Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 45 minutes in Lao and 90 minutes with translation. We will ask you questions related to the LTR Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 100 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the evaluation team. This means that only the evaluation team—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the LTR Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our evaluation team so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our evaluation team. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the Data Collection Coordinator, Manila Vongsouthi at “mvongsouthi@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No START THE AUDIO RECORDER [IF RESPONDENT(S) ANSWER YES] 112 | FINAL EVALUATION REPORT USAID.GOV Evaluation Question Questions/ Sub Questions Introduction To be recorded by note-taker: Stakeholder type: _______ Sex: _____ Location: ___________ To begin, please tell me about your involvement with the USAID LTR project. 1. What is your role on the VEDC? 2. Are any of you familiar with the USAID Learn to Read project and its goals and activities? a. The goal of LTR is to improve children’s reading skills in PP, G1 and G2, particularly for non-Lao-Tai speakers and children with mild disabilities. The project has three key components: 1) providing schools with supplemental materials (storybooks) for non-Lao-Tai speakers and children with mild disabilities and establishing reading corners, 2) training teachers in reading instruction and inclusive education, and 3) supporting parents of pre-primary students to help their children learn how to read in the local language and in Lao) 3. How has the VEDC been involved in the LTR project? EQ1: Progress towards Activity Objectives Successes and challenges to date per Result Areas Achievement of results – Gender and Inclusion (7d) Achievement to Date: 4. In which LTR training workshops have you participated? a. What did you learn during these trainings? b. Was there anything that you wish the training covered that it didn’t? 5. What LTR activities have you facilitated or participated in? a. Meeting to introduce reading corners and talk about the poster for supporting children’s reading (1 meeting in 2020-21 school year) b. Caregiver Engagement sessions, when you learned 12 activities to support your children’s reading (5 sessions in 2019-20 school year) c. Parent-teacher meeting d. Reading festival e. Other 6. What have been the results or successes of these activities? 7. What have been the unforeseen challenges you or the project has encountered when carrying out community engagement activities? 113 | EVALUATION FINAL REPORT USAID.GOV Evaluation Question Questions/ Sub Questions EQ1: Most and least successful components Activity Effectiveness Most/Least Successful Components 8. Among the LTR activities that you have participated in, which has been the most effective in improving children’s reading skills to date? Why? 9. Which of the components or activities have been the least effective to date? Why? 10. Are there any additional activities, trainings or interventions needed to strengthen the effectiveness of the LTR project? If yes, what? EQ2: Activity implementation/ operations factors hindering or supporting implementation Impact of COVID-19 I am interested to understand how COVID-19 impacted the USAID LTR Activity. 11. How has COVID-19 impacted learning in your school, home, or village? 12. Do you have any recommendations for how LTR can support teachers and students to continue learning during a COVID-lockdown or brief interruption to schooling? 13. What do parents need to help their children learn at home during a COVID lockdown when schools are closed? EQ3: Stakeholder Dynamics School-community stakeholders 14. Are you satisfied with your level of participation in the LTR project? 15. Has the LTR project met your expectations for training, support, and engagement? Why or why not? Conclusion 16. Thank you for taking the time to speak to me today about the USAID LTR Activity. Before we finish, is there anything else that I did not cover, that you would like to add? 17. Are there any recommendations you have to address any of the issues we discussed today that would improve the future implementation and sustainability of the Activity? THANK THE RESPONDENT AND TURN OFF THE AUDIO RECORDER 114 | FINAL EVALUATION REPORT USAID.GOV ANNEX E: EVALUATION TEAM EVALUATION TEAM The SI evaluation team (ET) consisted of four core team members, logistical support staff and home office evaluation management staff. Dr. Brenda Sinclair served as Team Leader responsible for overall management of the evaluation and production of deliverables. She is an accomplished Global Education, Gender and Evaluation Consultant with 20 years of experience designing, monitoring, and evaluating education development projects worldwide, including significant experience working with USAID Early Grade Reading programs. Dr. Souhila Messaoud-Galusi served as the Senior Partnerships Specialist bringing over 12 years of experience working with donors, including the past four years spent consulting with the World Bank to design the GPE II Reading Readiness package among other activities. The local evaluation experts included Ms. Salika Khoonbarthao, with 15 years’ experience in monitoring, evaluation and data management, and Ms. Noumandan “La” Kommana with nearly a decade of experience conducting qualitative and quantitative research. Both local evaluation specialists had prior experience working with the BEQUAL project and were responsible for conducting data collection for the LTR project while also contributing to note-taking, data analysis and translation. The team received managerial, technical, and logistical support from SI headquarters staff, namely Project Director Dr. Amy Porter, Project Manager, Mr. Dominick Margiotta, Project Assistant Ms. Jennifer Elkins, Dedoose expert, Sierra Frischknecht, and Gender Equity and Social Inclusion (GESI) Advisor Ms. Lisa Basu. In-country translators and note-takers in Lao PDR also supported the ET during data collection. 115 | EVALUATION FINAL REPORT USAID.GOV ANNEX F: TERMS OF REFERENCE Terms of Reference Mid-Term Evaluation of the USAID’s supported Learn to Read Activity April 2021 to August 2021 I. Background Learn to Read (LTR) is a five-year USAID-funded Activity beginning on August 15, 2018 and ending on August 14, 2023. The goal of the Activity is to improve reading skills for all children in pre-primary through second grade, with a particular focus on non-Lao-Tai-speaking children and children with disabilities. This will be achieved through providing supplemental materials (Result 1), teacher training and support (Result 2) and community engagement (Result 3) programming. LTR is implemented by a consortium led by Save the Children (SC) that includes Room to Read (RtR) and Humanity & Inclusion (HI). School and community-level activities will take place in 16 districts in four provinces of Lao PDR, reaching an estimated minimum of 65,313 children over the project’s life. All programming is carried out in close partnership and a productive working relationship with the units of the Ministry of Education and Sports (MoES) at national, provincial, and district level, with the aim of building system capacity to support children’s reading in the long-term. The units of the MoES that are most relevant to this project including: The Research Institute for Educational Sciences (RIES), the Department of Early Childhood Education (ECE), Department of General Education (GE), Department of Teacher Education (DTE) and the Inclusive Education Center (IEC). The Activity works closely with not only the Ministry of Education and Sports (MoES), but also leading development partners to strengthen the Lao education system with evidence-based approaches. The work with the second Global Partnership for Education (GPE-II) is on PP-RR and work with the Basic Education Quality and Access in Lao PDR project (BEQUAL) supports reading improvement and the roll-out of the national primary curriculum for Grades 1 and 2. A key priority of the Activity is to align and coordinate with these, and other, education initiatives to ensure that LTR complements and augments ongoing literacy improvement efforts and maximizes the return on MoES and donor efforts in the education sector. The Activity supports and advocates for improved, evidence-based pedagogical support for non-Lao-Tai speaking children within the national policy environment, as well as provides a practical model of how existing policies on inclusive education can be implemented at the school level. Inclusive education efforts will take place across all three result areas and will highlight inclusion in materials design, pilot teacher training and support to the main teacher training activities, and community engagement. The Mid-term Evaluation is a requirement of the USAID’s supported Learn to Read Cooperative Agreement No. 72048618CA00002 per the ADS 201.3.6.5 since this Cooperative Agreement is with a $20-million TEC. 116 | FINAL EVALUATION REPORT USAID.GOV II. Mid-term Evaluation objectives The objectives of Mid-Term Evaluation of the USAID’s supported Learn to Read activity are to identify main areas of progress, key challenges - other than the MOU process, and major factors that potentially influence the sustainability of the LTR model within the Lao educational system. Then to make key recommendations for corrective action to resolve outstanding issues and improve performance for the remainder of the project duration and/or beyond. Within this objective, specific issues, and questions to be addressed will include, but not be limited to, the following: 1. Progress towards Activity objectives: How likely will the three key result areas, enumerated in the results framework, be achieved within the set timeframe? a. Which components of the three key result areas are perceived to have most effective/successful during the implementation to date? What is driving these successful components? b. Which components, of the three key result areas, are perceived to have been least successful to date? What is hindering these components? 2. Activity implementation/operations factors: What (and how) are internal operational processes, - other than MoU process, (i.e., funding, management/operations, changes of context, etc.) supporting and/or hindering implementation? a. What LTR operational areas, including: governance, management, monitoring and evaluation, could be changed and/or improved to increase the likelihood of sustainability? 3. Stakeholder dynamics: What evidence exists, and what does it tell us, about how effectively Learn to Read has been engaging the Ministry of Education and Sports and other key stakeholders at central, provincial, and district levels to meet the Activity objectives? a. Are program stakeholders, including the Ministry of Education and Sports, satisfied with their level of participation? Do they perceive that their expectations have met by LTR to date? b. In terms of engagement of stakeholders, what additional/modified support is required, if any, to accelerate the potential for Activity sustainability? 4. Current LTR partners: What aspects of other development partners’ supported programs, such as, the second Global Partnership for Education (GPE-II) and the Basic Education Quality and Access in Lao PDR project (BEQUAL), are most immediately complementary to, and support of the goals and objectives of LTR? a. Among existing LTR partners, if USAID were to support any additional program components, which are most compelling and why? That is, which are most able to provide credible evidence of: (1) effectiveness at supporting the development of reading skills for children at pre-primary, G1, and G2 levels; and (2) organizational/programmatic strengths that could be leveraged during the remaining period and/or beyond? 117 | EVALUATION FINAL REPORT USAID.GOV III. Methodology The Learn to Read MTE will be conducted in Vientiane Capital (where is the main offices of the Implementing Partners located) and two other locations among the four target provinces (one location is either Vientiane Province or Xieng Khouang Province and another location is in either Champasack or Attapeu Province), for data collection among implementing partners and key stakeholders including Government of Laos (GoL) officials in central, provincial and district level. It is anticipated that an external evaluation team will work in close collaboration with USAID Laos Country Representative (USAID LCRP) and the Learn to Read teams throughout the evaluation process, while ensuring independence of the MTE team as well as a firewall between LTR teams to prevent a conflict of interest. The evaluation team will implement at least the following activities and is encouraged to propose additional methods to ensure effectiveness of the MTE. Within this TOR framework, the following specific methods and approaches are recommended as guidance: 1. A desk review of all relevant LTR documents and other relevant education sector documents including policy and legislation (to be identified). 2. Face to face interviews and discussions with all key stakeholders involved in the Learn to Read Activity and relevant Lao reading activities to ensure that the evaluation is carried out in a participatory manner. A list of key partners and stakeholders would be identified at an early stage (see project partners and key stakeholders tentative list @ VI below) and a consultation process developed. All stakeholders consulted should be in a position to present their views in confidence to the MTE team and to identify issues, opportunities, constraints and options for the improvement and future programing. 3. Virtual data collection through teleconference or written comments; where partners and other key stakeholders cannot be reached for face-to-face interviews. 4. Design, prepare, and facilitate at least a workshop (if needed) to obtain feedback and additional inputs, as well as to develop a set of joint recommendations for Activity adjustments and future programming as relevant. 5. Submit an evidence-based evaluation report which draws the relationships between findings, conclusion, and recommendations that responses to the research questions as well as an analysis of the data/information obtained from desk reviews, interviews, and key recommendations. IV. Deliverables Deliverable 1: Design and Implementation Plan o Submit an inception report for MTE for LTR which includes specific details on methodology, data collection instruments, analysis plan and tool, work schedule, roles, and responsibilities for key study team members, taking into consideration the ethical and confidentiality issues and the context of Covid-19 pandemic. o After inception report is approved, prepare a 1–2-page brief in English and Lao that provides an overview of the study to be shared by USAID with the MoES to obtain permission to collect data with key MoES stakeholders and in target LTR schools. 118 | FINAL EVALUATION REPORT USAID.GOV Deliverable 2: Presentation of findings in English and in Lao as and when required • Facilitate a workshop for Learn to Read partners and other relevant stakeholders that includes a presentation synthesizing the initial study findings from the MTE for LTR and then provides a form for stakeholders to validate the findings, obtain additional inputs, and develop joint recommendations; • Present the findings to the USAID LCRP, Learn to Read Executive Committee and other relevant partners as appropriate for feedback. Deliverable 3: Reporting & Datasets • Submit a draft report in English, which will include an Executive Summary in Lao (to be shared with Lao partners, especially the GoL) as described above. • Incorporate feedback from all relevant parties in the final version of the report. Revise Lao Executive Summary to match revised English Executive Summary. • Prepare and present a final presentation based on final version of the MTE report in English with a final Lao version of Executive Summary. V. Timing and Schedules The MTE is scheduled to take place in the month of April to August 2021 with a total of number working days and details to be determined and/or provided by Social Impact (SI) but the final report should be expected to submit to USAID LCRP by end of August 2021. 119 | EVALUATION FINAL REPORT USAID.GOV Timeline Tentative Activities 31 March SI team submits technical and budget for USAID review Week of April 5 Kick off meeting with MTE 28 April Study team submits inception report (detailing methods, sample, limitations, detailed timeline, etc.). Study team submits on 28 April, 1 week for USAID and IP feedback – by 5 May, Study team revises and resubmits final version 12 May. Week of May 10th Discussion and agreement on proposed methodology with USAID LCRP and the Learn to Read partners & preparation for field/partner visits with LTR partners. Weeks of May 17, 24, & 31 Data Collection (3 weeks) Assessment of progress and performance – including field visits and interviews with Learn to Read partners and key stakeholders including concerned MoES officers at all levels and beneficiaries (venues to be determined with LTR partners): - Interview with Learn to Read partners’ staff at central, provincial and district levels - Interview with MoES officers at central, provincial and district levels - Interview with teachers, parents/caregivers at school and community level. June ● Stakeholder workshop ● Presentation of findings (USAID, LTR committee) Week of July 26 Submit draft report, including Executive Summary in English & Lao. Study team submits, 2 weeks for USAID and IP feedback, Study team revises and resubmits final version (2 weeks). Week of Aug 30 Debriefing – Final presentation of completed report to USAID LCRP and the Learn to Read partners V. Team Composition The MTE team is to be proposed by the Contractor and the team composition should be appropriate in accomplishing the tasks at high quality with strong justification. VI. Project partners and key stakeholders – tentative list i. Key and concerned staff from the Save the Children, Room to Read, Humanity & Inclusion (hosted by Save the Children) from central, provincial and district offices. ii. Key and concerned departments [the Research Institute for Educational Sciences (RIES), the Department of Early Childhood Education (ECE), Department of General Education (GE), Department of Teacher Education (DTE) and the Inclusive Education Center (IEC)] of the Ministry of Education and Sports (hosted by the External Relations Department). iii. Key and concerned MoES officers at provincial and district levels. iv. Teachers and parents/caregivers at school and community levels. v. Representatives from the second Global Partnership for Education (GPE-II) through the World Bank for pre-primary reading readiness. 120 | FINAL EVALUATION REPORT USAID.GOV vi. Representatives from the Basic Education Quality and Access in Lao PDR project (BEQUAL) together with DFAT Australia and EU Delegation for reading improvement and the roll-out of the national primary curriculum for Grades 1 and 2. VII. Project partners and key stakeholders – tentative list i. Key and concerned staff from the Save the Children, Room to Read, Humanity & Inclusion (hosted by Save the Children) from central, provincial and district offices. ii. Key and concerned departments [the Research Institute for Educational Sciences (RIES), the Department of Early Childhood Education (ECE), Department of General Education (GE), Department of Teacher Education (DTE) and the Inclusive Education Center (IEC)] of the Ministry of Education and Sports (hosted by the External Relations Department). iii. Key and concerned MoES officers at provincial and district levels. iv. Teachers and parents/caregivers at school and community levels. v. Representatives from the second Global Partnership for Education (GPE-II) through the World Bank for pre-primary reading readiness. vi. Representatives from the Basic Education Quality and Access in Lao PDR project (BEQUAL) together with DFAT Australia and EU Delegation for reading improvement and the roll￾out of the national primary curriculum for Grades 1 and 2.