This publication was produced for review by the United States Agency for International Development. It was prepared by EnCompass LLC for the Monitoring and Evaluation Program for Lebanon (MEPL) under Contract No. 7200AA20D00011, Task Order 72026821F00005. Note: Limited redactions have been made to this version of the report in accordance with the principled exceptions to the presumption in favor of openness established in OMB Bulletin 12-01, “Guidance on Collection of U.S. Foreign Assistance Data.” MID-TERM PERFORMANCE EVALUATION QUALITY INSTRUCTION TOWARDS ACCESS AND BASIC EDUCATION IMPROVEMENT (QITABI-2) FINAL REPORT AUGUST 2023 MID-TERM PERFORMANCE EVALUATION QUALITY INSTRUCTION TOWARDS ACCESS AND BASIC EDUCATION IMPROVEMENT (QITABI-2) WORLD LEARNING FINAL REPORT AUGUST 2023 Prepared by EnCompass LLC for the USAID Monitoring and Evaluation Program for Lebanon (MEPL) under Prime Contract No. 7200AA20D00011, Task Order 72026821F00005. This document is available to the public through the Development Experience Clearinghouse (DEC) repository (dec.usaid.gov). For additional information, please contact: EnCompass LL Corporate Office 1451 Rockville Pike, Suite 600 Rockville, Maryland 20852 USA Or EnCompass LLC Lebanon Office Arz Street Librex Bldg. Bloc 2 – 3rd Floor Zalka, Metn, Lebanon DISCLAIMER: This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the responsibility of EnCompass LLC and its implementing partners and do not necessarily reflect the views of USAID or the United States Government. COVER PHOTO: FGD with Grade-3 students in North Lebanon. (Photo by MEPL) ABSTRACT The USAID-funded Quality Instruction Towards Access and Basic Education Improvement (QITABI) 2 activity collaborates with the Ministry of Education and Higher Education (MEHE) and the Center for Educational Research and Development (CRDP) to improve learning of Literacy, Numeracy and SEL for all primary school students in Lebanon by strengthening the Lebanese education system’s institutional capacity for sustainability and self-reliance. USAID commissioned the Monitoring and Evaluation Program for Lebanon (MEPL) to carry out an evaluation of QITABI-2. This study combines a mid-term performance evaluation design with a research study and seeks to examine the extent of successful project implementation as well as to better understand education sector priorities in Lebanon. The evaluation found that QITABI-2 was well aligned with Ministry priorities and was well regarded and respected throughout the education sector. In addition, the team found that QITABI-2 had supported the development of teaching materials, and approaches that were tailored to the rapidly changing needs of the education sector and its beneficiaries. Ongoing crises such as COVID, the financial collapse, and social and political unrest have aggravated hindering factors to schooling access and quality and resulted in extended school closures. These closures delayed implementation and limited the full integration of QITABI-2’s approaches and the activity’s impact. The report highlights findings and conclusions of the evaluation and provides a series of recommendations for future on-going QITABI-2 work, future USAID supports in Lebanon as well as for other entities involved in education in Lebanon. ACKNOWLEDGMENTS The evaluation team consisted of Amy Mulcahy-Dunn (Team Leader), Karla Giuliano Sarr, and Joanne Chlela (Team Members) and received support from Joanna Khater, Nour El-Zaouk, Sean Kelly, and Ted Rizzo of EnCompass LLC. The evaluation team would like to extend their sincere thanks to the following individuals for their valuable contributions to this evaluation:  USAID/Lebanon’s Education Office, particularly Mr. Timothy Curtin and Ms. Zeina Salame, for making this evaluation possible and for their guidance and support throughout the process.  MEHE and CRDP leadership as well as the representatives of the donor community for sharing their reflections on QITABI-2 and providing their reflections regarding priority issues they see for Lebanon.  The QITABI-2 team for sharing information and insights regarding their program and Lebanon’s education sector and for their willingness to respond to multiple follow-on questions.  The Regional Education Officers, school principals, teachers, caregivers, and students for taking the time to speak with us and share their experience with QITABI-2’s programming. i | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 TABLE OF CONTENTS EXHIBITS .................................................................................................................................... II ACRONYMS ................................................................................................................................ III EXECUTIVE SUMMARY............................................................................................................... IV EVALUATION PURPOSE AND QUESTIONS ................................................................................. 1 EVALUATION PURPOSE ................................................................................................................................. 1 EVALUATION QUESTIONS ............................................................................................................................ 1 BACKGROUND ............................................................................................................................ 2 CONTEXT ....................................................................................................................................................... 2 QITABI-2 PROGRAM DESCRIPTION ........................................................................................................... 3 EVALUATION METHODS AND LIMITATIONS .............................................................................. 4 DOCUMENT REVIEW ..................................................................................................................................... 4 QUALITATIVE DATA COLLECTION ............................................................................................................. 4 SAMPLING ............................................................................................................................................... 4 DATA COLLECTION PROCEDURES ..................................................................................................... 5 DATA ANALYSIS, INTERPRETATION, SYNTHESIS, AND REPORTING........................................................ 5 LIMITATIONS AND BIASES ............................................................................................................................. 6 CONFLICT OF INTEREST................................................................................................................................ 6 FINDINGS .................................................................................................................................... 6 EVALUATION QUESTION 1 (RELEVANCE) .................................................................................................. 6 EVALUATION QUESTION 2 (RELEVANCE) .................................................................................................. 8 EVALUATION QUESTION 3 (RELEVANCE) ................................................................................................ 10 EVALUATION QUESTION 4 (EFFECTIVENESS) ........................................................................................... 12 EVALUATION QUESTION 5 (EFFECTIVENESS) ........................................................................................... 17 EVALUATION QUESTION 6 (SUSTAINABILITY) ......................................................................................... 20 RESEARCH QUESTION 1.............................................................................................................................. 21 RESEARCH QUESTION 2.............................................................................................................................. 24 CONCLUSIONS ......................................................................................................................... 26 RELEVANCE ................................................................................................................................................... 26 EFFECTIVENESS .............................................................................................................................................. 27 SUSTAINABILITY ............................................................................................................................................ 27 RESEARCH QUESTIONS ............................................................................................................................... 27 RECOMMENDATIONS ................................................................................................................ 28 ANNEXES.................................................................................................................................. 30 ANNEX I: EVALUATION TIMELINE ............................................................................................................. 31 ANNEX II: EVALUATION SCOPE OF WORK ............................................................................................. 32 ANNEX III: EVALUATION METHODS AND LIMITATIONS ........................................................................ 33 ii | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ANNEX IV: LIST OF DOCUMENTS REVIEWED .......................................................................................... 43 ANNEX V: COMPREHENSIVE INDICATOR ANALYSIS ............................................................................... 48 EXHIBITS Exhibit 1: Summary of findings, conclusions, and recommendations ............................................................. v Exhibit 2: School-Level Data Collection by Governorate................................................................................. 5 Exhibit 3: Contextual timeline for QITABI-2 (March 2019-2024) .................................................................. 7 Exhibit 4: Alignment of QITABI-2's results framework with the MEHE Plan and the USAID CDCS .... 7 Exhibit 5: Illustration of QITABI-2’s adaptions in response to changing needs ......................................... 11 Exhibit 6. Stakeholder perspectives on ways QITABI-2 adaptations met their needs ............................ 12 Exhibit 7. Percent of students who improved by at least one reading level .............................................. 14 Exhibit 8: Evaluation Design Matrix ...................................................................................................................... 33 Exhibit 9: Question Legend .................................................................................................................................... 35 Exhibit 10: Detailed Document Review List ...................................................................................................... 35 Exhibit 11: School-Level Data Collection by Governorate ............................................................................ 37 Exhibit 12: School Sample Achievement of Sample Targets ........................................................................... 37 Exhibit 13: Principal, Teacher, Caregiver, and Student Samples Achievement of Sample Targets ....... 38 Exhibit 14: Progress towards Objective-1 indicators ...................................................................................... 48 Exhibit 15: Progress towards Objective-2 indicators ...................................................................................... 50 Exhibit 16: Progress towards Objective-3 indicators ...................................................................................... 51 iii | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ACRONYMS ALLC American Language Learning Center CBA Curriculum-Based Assessment CDCS Country Development Cooperation Strategy CRDP Centre de Recherche et Développement Pédagogique (Center for Educational Research and Development) D-RASATI Developing Rehabilitation Assistance to Schools and Teacher Improvement DAIS Data Analysis, Interpretation, and Synthesis DOPS Département d’Observation Pédagogique Scolaire DGE Director General of Education EGMA Early Grade Mathematics Assessment EGRA Early Grade Reading Assessment EQ Evaluation Question FGD Focus Group Discussion HGSF Home Grown School Feeding HICD Human Institutional Capacity Development IP Implementing Partner IRC International Rescue Committee KFW Kreditanstalt für Wiederaufbau KII Key Informant Interview LOI Language of Instruction LRP Learning Recovery Program LRPM Learning Recovery Program Method M&E Monitoring and Evaluation MEHE Ministry of Education and Higher Education MEPL Monitoring and Evaluation Program for Lebanon MSI Management Systems International NYU New York University/Global TIES for children QITABI Quality Instruction towards Access and Basic Education Improvement REO Regional Education Officer RQ Research Question SEL Social Emotional Learning TLM Teaching and Learning Material TVET Technical and Vocational Education Training UNICEF United Nations International Children's Emergency Fund USAID United States Agency for International Development USDA United States Department of Agriculture USG United States Government WFP World Food Programme iv | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 EXECUTIVE SUMMARY The United States Agency for International Development (USAID)-funded Quality Instruction Towards Access and Basic Education Improvement (QITABI-2) Activity works to build the Lebanese education system’s institutional capacity for sustainability and self-reliance. QITABI-2 runs from March 31, 2019 to September 30, 2024 and is implemented by World Learning alongside its partners Ana Aqra Association, the American Language Learning Center (ALLC), the International Rescue Committee (IRC), Management Systems International (MSI), and New York University/Global TIES for children (NYU). Through its three outcomes, QITABI-2 seeks to 1) improve student performance in reading, math, and writing, 2) improve Social and Emotional Learning (SEL) skills, and 3) improve national-level service delivery of education. QITABI-2 works with all primary public schools in Lebanon, including those that host second-shift learning for refugee students. QITABI-2’s ultimate goal is to improve learning for all primary school students, both Lebanese and non￾Lebanese. This study was conducted by USAID’s Monitoring and Evaluation Program for Lebanon (MEPL), implemented by EnCompass LLC and MSI.1 It combines a mid-term performance evaluation of the QITABI-2 Activity to examine the extent to which project implementation has been successful and a research study aimed at better understanding the priorities of Lebanon’s education sector. The study took into account the changing Lebanese context during the period of implementation. EVALUATION QUESTIONS The study aimed to answer six evaluation questions (EQs) covering relevance, effectiveness, and sustainability and two research questions (RQs). In order to respond to these questions, the evaluation team conducted a desk review and primary qualitative data collection through a series of key informant interviews and focus group discussions with ministry, donor, regional and school-level staff as well as parents and Grade 3 and Grade 6 students from 15 schools across the country. Data collection occurred in two phases between December 2022 and March 2023 and involved both in￾person and virtual work. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Exhibit 1 distills the key findings, conclusions, and recommendations from this study. Where recommendations are not applicable (N/A), the activity should continue with its current design and implementation. 1 MSI, which is a subcontractor to World Learning on QITABI-2, is also a subcontractor to EnCompass LLC on MEPL. As such, to ensure objectivity in design, analysis, and reporting, MEPL has taken special precautions not to involve its MSI-affiliated staff in this evaluation work. v | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Exhibit 1: Summary of findings, conclusions, and recommendations EQ1. Relevance: How relevant was the Activity to the needs of the direct and indirect beneficiaries? Findings Conclusions Recommendations  The three pillars central to the MEHE General Education Plan (2021-2025) are nearly identical to the 2021-2026 Country Development Cooperation Strategy (CDCS). QITABI-2 programming aligns exactly with Pillars 2&3/Results IR2.2 and IR2.3.  Areas of synergy are improving learning outcomes, strengthening the education system, and integrating social and emotional learning (SEL) across the curriculum.  QITABI-2 builds on the strong foundation of relevance and collaboration established by QITABI-1.  QITABI-2 programming demonstrates relevance to the needs of beneficiaries through its alignment with core frameworks, focus on government priority areas, and continuation of elements deemed relevant during the previous activity.  N/A EQ2 Relevance: In light of the crises, what do primary and secondary beneficiaries (teachers, Center for Educational Research and Development (CRDP) staff, MEHE staff, students, caregivers) and other stakeholders now see as the priority challenges for primary education? Findings Conclusions Recommendations  Interviewed stakeholders focused on learning loss as a priority.  Stakeholders of all types identified teacher salary reductions as the greatest immediate challenge to primary education.  The MEHE General Education Plan integrated SEL programming.  The learning recovery study found high levels of anxiety and frustration among teachers, a point that arose as well during focus groups with teachers.  Nearly all school-level stakeholders agreed parents struggle to pay the cost of transportation for their children to attend school.  Interviews revealed children  Six priority challenges emerge regarding the primary education sector in Lebanon: o Learning loss, o Teacher salary reductions, o SEL needs of students and teachers, o Transportation cost limiting access to school, o Food insecurity, and o Insufficient resources at MEHE, CRDP, DOPS, REOs, school: electricity, heating, and stationery. (Please see the research questions for relevant recommendations.) vi | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Findings Conclusions Recommendations and teachers not having sufficient food. Most caregiver focus groups described how priorities shifted to ensuring children had a decent meal.  Critical resources like electricity, heating, and paper are lacking at many schools. EQ3. Relevance: A) To what extent has QITABI-2’s implementation adapted to the changing priority needs that have emerged since the original project design? Were these adaptations successfully applied? Findings Conclusions Recommendations  QITABI worked with MEHE, CRDP with support from UNICEF to introduce several adaptations in response to the beneficiaries’ changing needs, including: o Implementing “Distance Learning” and “Back to School” initiatives. o Developing E-content, training on production of digital lessons, supporting teachers in distance-learning approaches. o Creating Learning Recovery Program (LRP) to help students re￾enter school after a long learning disruption.  QITABI successfully pivoted programming to respond to the crises and the changing needs of direct and indirect beneficiaries, demonstrating collaborative partnership and responsiveness, great cultural awareness, and adaptive management.  USAID should leverage QITABI-2’s example of partnership, project delivery, and management within a crisis setting as a best practice. EQ3. Relevance: B) To what extent do beneficiaries feel that the adapted QITABI-2 activity design effectively addresses their needs related to primary education? Findings Conclusions Recommendations  Participants, cited several QITABI-2 adapted programs including the Learning Recovery Program that integrates SEL and , the Summer School catch-up Programs, the digitization of learning materials, and the distribution of food parcels as being crucial in light of the crises.  Direct and indirect beneficiaries felt that QITABI￾2’s adaptations were responsive to their needs.  N/A vii | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 EQ4. Effectiveness: To what extent has the Activity achieved its intended outputs and outcomes in terms of improving student performance in reading, math, and writing; instilling students with SEL skills; and improving the national-level service delivery of education? Findings Conclusions Recommendations  QITABI-2 exceeded seven of its output indicator targets and achieved between 79 and 91 percent of the four remaining output indicators.  Data for outcome indicators for student and teacher performance were not available due to COVID￾related delays.  Qualitative data indicate that QITABI-introduced approaches show promise.  QITABI-2 has successfully achieved the majority of its intended outputs.  Delays due to COVID, teacher strikes, and other crises have limited QITABI-2’s impact and made measurement of student performance impossible during the drafting of this report.  Given the continuing threat of strikes and schools closures, in addition to the challenges faced by students and teachers in accessing e￾learning materials, USAID should work with QITABI-2, MEHE, CRDP, and UNICEF to explore solutions for equitable remote learning options that could be delivered centrally when teachers are not available. EQ5. Effectiveness: What were the factors that contributed to or hindered the a) adoption of education interventions promoted by QITABI-2, and b) the achievement of the intended results? Findings Conclusions Recommendations Contributing Factors:  QITABI-2 team is considered technically strong, well￾organized, and communicative.  The activity applies a research-based approach throughout the design, evaluation, and adaptation of programming.  Activities were co-designed with MEHE and CRDP  Activities were integrated into existing systems.  Holistic approach that considered students’ and teachers’ well-being. Several factors contribute to the effectiveness of QITABI-2:  Technically strong team and research-based approaches garnered respect for QITABI-2 o Close collaboration with MEHE and CRDP ensured alignment with system priorities and integration within existing systems. o The trust and respect established in this close collaboration facilitated adaptation of new materials and approaches.  N/A Hindering factors:  Repeated school closures reduced instructional time and disrupted the Learning Recovery Program (LRP).  Teachers and principals indicated a need for additional training and mentoring to support teachers’ adoptions of LRP approaches.  Increased transportation costs negatively impacted  Several factors hinder QITABI￾2’s effectiveness: o Financial crisis and the government’s limited funding for education, o Prolonged school closures with limited access to viable remote learning options, o High transportation costs.  Learning Recovery Program Method (LRPM) not fully  Given the need for more teacher mentoring and limited resources, QITABI-2 should work with MEHE to identify alternate approaches to providing regular teacher coaching and mentoring.  QITABI-2 should provide training to principals so that they can more effectively support viii | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Findings Conclusions Recommendations student and teacher attendance.  Interviews indicate connectivity issues and/or weak digital literacy limited access to remote learning materials.  Teachers shared how limited physical space and time prevented them from fully implementing LRP.  Though LRP materials were designed for students who were performing below grade level, some teachers and principals indicated that materials were too easy.  Teachers, CRDP, and MEHE staff noted that the outdated 1997 national curriculum limited the changes that were possible as innovative QITABI approaches, though supported by the Ministry, did not align with existing materials. There are indications that curriculum reform is underway, and this would substantially reduce this barrier. integrated into standard teaching practice because of: o Limited time to apply lessons learned, o Lack of training for principals, o Need for more extensive and on-going teacher training and coaching. teachers and the implementation of the LRP.  Explore ways to strengthen digital literacy and increase access to and use of remote learning materials.  QITABI -2 should determine if the perception among some educators that LR materials were too easy implies the need for additional teacher training or if the materials may need to be adjusted. Teacher training might seek to ensure that teachers are clear in the purpose, know when and how to target students, and how to differentiate instruction for more advanced learners.) EQ6. Sustainability: At this mid-term point, what signs emerge that project activities and results may continue after the project ends? What factors may hinder the sustainability of interventions? Findings Conclusions Recommendations  There are multiple examples of lasting impact of QITABI-2: o MEHE’s integration of QITABI-2’s approaches into the five-year. General Education Plan, o MEHE’s adoption of the SEL framework to align all future SEL work. o QITABI-2’s support of the new curricular framework. o MEHE and CRDP’s desire for QITABI-2’s support of the new curricular reform.  Alignment with MEHE’s priorities and integration  Several elements speak to the lasting impact of QITABI-2 including:  QITABI-2 has worked to maximize the sustainability of its programming by working collaboratively with MEHE, CRDP, and UNICEF in delivering the LRP and in integrating QITABI-2’s interventions and supports within existing systems.  N/A ix | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Findings Conclusions Recommendations within existing systems enhance the sustainability of education programming.  Close collaboration with MEHE and CRDP in QITABI￾2’s activities is a critical element for sustainability according to many stakeholders.  MEHE’s financial resources are limited, and foreign aid is needed to cover school operation costs.  MEHE’s DOPS directorate does not have the human resources needed to coach teachers at all public primary schools with the intensity of teacher coaching visits provided within a sample of 120 schools.  CRDP does not have human resources needed to conduct EGRAs, EGMAs, and CBAs conducted with QITABI-2 support.  The public education sector cannot function effectively without substantial foreign aid.  The current economic and financial state of affairs severely impacts the sustainability of QITABI-2’s activities and threatens to erase the gains achieved by the project and other donor-funded projects.  QITABI-2 should work with MEHE to identify alternate approaches to fund and staff teacher coaching and mentoring.  Given limited human resources at CRDP, explore alternate, less labor-intensive student assessment approaches.  USAID should continue to coordinate with UNICEF and World Bank to ensure the alignment of donors’ different learning recovery efforts in Lebanon. RQ1. Looking broadly at the basic education sector (pre-primary, primary, secondary, and technical vocational education, and higher education) what do key local and international education stakeholders with a global understanding of the sector feel are priority challenges? Within particular education levels? At a sector-wide or systemic level? How do these perceptions align with the most recent USAID Lebanon CDCS and MEHE education sector strategic plan? Findings Conclusions Recommendations  MEHE faces profound financial and resource limitations.  Addressing learning loss is the primary concern.  Priority needs broadly noted by respondents include: o Finding more effective and sustainable ways of addressing student nutrition needs. o Keeping schools open and addressing barriers to school access. o Addressing school quality via the development of the new curriculum. o Strengthening the capacity of principals.  Priority needs include: o Finding more effective and sustainable ways of addressing student nutrition needs. o Keeping schools open and addressing barriers to school access. o Addressing school quality via the development of the new curriculum. o Strengthening the capacity of principals.  Additional issues flagged include: o Strengthening early childhood education and TVET. o Developing programming  USAID continue discussions with the Education Donor Working Group and MEHE/CRDP in order to alleviate teacher salary problems to avoid future teacher strikes and school closures.  (See RQ2 for additional recommendations) x | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Findings Conclusions Recommendations  Donors and IPs noted priority areas such as: o Strengthening early childhood education and TVET,  Developing programming for out-of-school youth. for out-of-school youth. RQ2. Where is USAID best placed to make an impact considering where the government of Lebanon and stakeholders are already putting their resources? What do both key informant interviews and related emerging data say about potential future interventions related to but not part of the existing QITABI-2 activity? Findings Conclusions Recommendations  MEHE, CRDP, and donor respondents advocated for the need to continue and/or expand USAID’s current education programming.  QITABI-2 baseline assessment indicates that students were still performing below grade level and teachers were applying approaches inconsistently.  MEHE requested QITABI-2’s technical support in the development of the new curriculum.  MEHE requested support in expanding the LRP programming to Cycle III; MEHE feels that the lower￾secondary schools are in need of support.  Respondents highlighted the need to establish school feeding programs to provide students with meals at school.  USAID occupies a unique space among education actors in Lebanon in its focus on improving student performance in reading, math, and SEL.  USAID should continue to support: o Cycles I and II including Summer School Programming. o Development of the new curriculum.  Cycle III is seen a critical transition phase; Cycle III is not currently supported by donor funding.  There is an urgent and increasing need to establish school feeding programs.  USAID should continue supporting foundational learning in Cycle I and Cycle II and consider expanding support to Cycle III.2  QITABI-2 should contribute meaningfully to the development of the new curriculum under MEHE and CRDP’s leadership.  USAID should consider researching alternate and sustainable school feeding options.  USAID should consider supporting CRDP in identifying solutions to make CRDP's eight resource centers for students with disabilities operational again. 2 Cycle I is Grades 1-3, Cycle II is Grades 4-6, and Cycle III is Grades 7-9. UNESCO, Cartographie de la CITE du Liban (Montreal: UNESCO Institute for Statistics 2020), https://uis.unesco.org/sites/default/files/documents/isced_2011_mapping_fr_lebanon_0.xlsx. 1 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE A wave of social, political, and financial crises and disasters beginning in 2019 have rocked Lebanon’s educational system and substantially changed the realities of everyday life for many Lebanese. These crises have severely weakened the education system. In March 2019, USAID launched the five-year (2019-2024) $90 million Quality Instruction Towards Access and Basic Education Improvement (QITABI) 2 Activity that works to build the Lebanese education system’s institutional capacity for sustainability and self-reliance. World Learning implements the project alongside its partners Ana Aqra Association, the American Language Learning Center (ALLC), the International Rescue Committee (IRC), Management Systems International (MSI), and New York University/Global TIES for Children (NYU). Through its three outcomes, QITABI-2 seeks to 1) improve student performance in reading, math, and writing, 2) improve Social and Emotional Learning (SEL) skills, and 3) improve national-level service delivery of education. QITABI-2 works with all primary public schools in Lebanon and engages with low-cost private schools to enhance all primary students’ learning outcomes. QITABI-2’s ultimate goal is to improve learning for all primary school students, both Lebanese and non-Lebanese. This study was conducted by USAID’s Monitoring and Evaluation Program for Lebanon (MEPL). It pairs a mid-term performance evaluation of QITABI-2 with diagnostic research questions and seeks to examine the extent to which project implementation has been successful as well as to better understand the education sector’s priorities in Lebanon. The study took into account the changing Lebanese context during the period of implementation. The evaluation and diagnostic research considered the successes, systemic constraints/bottlenecks, and perceived value among stakeholders and beneficiaries. While USAID/Lebanon and its implementing partners are the principal audiences for the study, the report may also benefit QITABI-2’s critical partners, the Lebanese Ministry of Education and Higher Education (MEHE), and other key education partners, including the donor community, researchers, and other key stakeholders working in education. The study offers recommendations that focus specifically on mid-course corrections for the QITABI-2 Activity to maximize its benefits, as well as suggestions for future USAID programming. EVALUATION QUESTIONS The evaluation questions (EQs) explored QITABI-2’s relevance to beneficiary needs, effectiveness in achieving outputs and outcomes, effectiveness in responding to the changing context in Lebanon, and possibilities for sustaining change. The research questions (RQs) look at broader challenges within the Lebanese education system. Recommendations emerged across the questions for course correction and further programming.  EQ1. Relevance: How relevant was the activity to the needs of the direct and indirect beneficiaries?  EQ2 Relevance: In light of the crises, what do primary and secondary beneficiaries (teachers, Center for Educational Research and Development (CRDP) staff, MEHE staff, students, caregivers) and other stakeholders now see as the priority challenges for primary education?  EQ3. Relevance: A) To what extent has QITABI-2’s implementation adapted to the changing priority needs that have emerged since the original project design? Were these adaptations successfully applied? B) To what extent do beneficiaries feel that the adapted QITABI-2 activity design effectively addresses their needs related to primary education?  EQ4. Effectiveness: To what extent has the Activity achieved its intended outputs and outcomes in terms of improving student performance in reading, math, and writing; instilling students with SEL skills; and improving the national-level service delivery of education?  EQ5. Effectiveness: What were the factors that contributed to or hindered the a) adoption of education interventions promoted by QITABI-2, and b) the achievement of the intended results? 2 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023  EQ6. Sustainability: At this mid-term point, what signs emerge that project activities and results may continue after the project ends? What factors may hinder the sustainability of interventions?  RQ1. Looking broadly at the basic education sector (pre-primary, primary, secondary, and technical vocational education, and higher education) what do key local and international education stakeholders with a global understanding of the sector feel are priority challenges? Within particular education levels? At a sector-wide or systemic level? How do these perceptions align with the most recent USAID Lebanon CDCS and MEHE education sector strategic plan?  RQ2. Where is USAID best placed to make an impact considering where the government of Lebanon and stakeholders are already putting their resources? What do both key informant interviews and related emerging data say about potential future interventions related to but not part of the existing QITABI-2 activity? BACKGROUND CONTEXT Since 2019, Lebanon has been battered by multiple years of overlapping crises, including the nationwide uprising that started on October 17, 2019, the COVID-19 crisis and ensuing lockdowns, the Port of Beirut explosion on August 4, 2020, socio/political unrest, and a financial and economic crisis that brought hyperinflation, extreme currency devaluation, power-outages, and fuel-shortages. These crises have severely impacted all facets of Lebanese life and weakened the country’s education system. School closures due to COVID-19 persisted during the 2020-2021 school year with 35 weeks of complete closure and another 15 weeks of partial closure across the 2019-2020 and 2020-2021 school years. Like in many other parts of the3 world, learning shifted from in-person to distance learning modalities. The school system was ill-equipped for distance learning and teachers lacked training in remote teaching approaches and student support4. In addition to quality issues, the school system faced insufficient access to devices, high connectivity costs, frequent power outages limiting students’ access to distance learning and teachers’ implementation of adaptive strategies, especially for the most marginalized. School enrollment and achievement were doubly affected as the crises led to increased poverty and resulted in lower demand for education. The extreme currency devaluation has meant that monthly teacher salaries have dropped from the equivalent $1,000/month in 2019 to an estimated $60-$90/month at the beginning of January 20235. As a result, school personnel and teachers are frequently unable to afford school transport costs and suffer from a loss of motivation that often results in poor attendance. Intermittent and prolonged educator strikes have occurred as teachers demand to offset the devolution of their salaries. Efforts to recoup the COVID learning loss were undermined by extensive school closures during the 2021-22 and 2022-23 school years. In the 2021-2022 school year, it is estimated that the total number of teaching days in public schools did not exceed 59 out of the 104 days set by MEHE1. In 3 https://covid19.uis.unesco.org/global-monitoring-school-closures-covid19/country-dashboard/ 4 Youssef, S., Jones, N., Małachowska, A., and Saleh, M. (2022) Adolescents in the abyss of Lebanon’s worst economic crisis: a focus on Lebanese and Palestinian adolescents’ education, and voice and agency. London: Gender and Adolescence: Global Evidence July 2022 Lebanon-Girls-Education-and-VA-WEB.pdf (odi.org) 5 Based on information shared during KIIs with MEHE, donor representatives, and QITABI team members in January 2023. The currency continued to drop further reducing the value of teacher salaries. As of the end of January 2023, the official central bank exchange rate had devalued the Lebanese Pound (LBP) by 90 percent dropping it from 1500 LB per US$ pre￾crises to 15,000 LBP/US$ in February 2023. The unofficial rate used in business in February was 57,000 LPB/US$ (Reuters). 3 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 the 2022-2023 school year, extended teacher strikes delayed the start of the second semester by two months (January 8-March 6, 2023) further disrupting schooling. The financial crisis with its triple-digit inflation and a currency devaluation of 98 percent compared to 2019 rates, has pushed more families into poverty with 74 percent of the population estimated to be living in poverty6, and the World Food Programme (WFP) estimating that 60 percent of the population was “multi-dimensionally deprived” 7. Families face greater difficulty paying school fees, transport costs, uniforms, and school supplies. Many families shifted their children from private to public schools, increasing enrollment in under-resourced public schools by 11 percent in the 2020- 2021 school year. In addition, students from poorer families continue to drop out of school, as families can no longer afford the direct or indirect costs of schooling. Both these trends are anticipated to continue as poverty rates rise8. School closures, teacher and student absenteeism, student drop-out, as well as limited access to distance learning have resulted in substantial learning loss and will have lasting negative impacts on the quality of learning and learning outcomes. Teachers in 2022 estimated that their students were 1-2 years behind where they should be.9 The 2022 QITABI-2 baseline assessment results confirmed teachers’ estimates as the majority of 2nd, 3rd, and 6th-grade students demonstrated a lack of foundational skills in Arabic, French, English, and math. 10 QITABI-2 PROGRAM DESCRIPTION QITABI-2 programming targets primary school teachers in grades 1-6 and school principals. The program supports all 88711 public schools in Lebanon. These schools include first and second-shift schools. Activities involve teacher training and coaching; development and provision of educational materials and equipment, including e-conte, classroom libraries and math manipulatives; support for learning recovery programming; support for the development of SEL framework and SEL materials, instructional approaches, and assessment measures; and systems-strengthening and capacity development for MEHE and the Center for Educational Research and Development (CRDP). QITABI-2 seeks to ensure that educational institutions have the capacity to develop, implement, and monitor policies and curricula and enable teacher capacity to support learning outcomes in literacy, numeracy, and SEL. QITABI-2’s early grade reading assessment (EGRA) and early grade math endline assessment (EGMA) are planned in April 2024 to measure the impact of the intervention in literacy and numeracy. As various crises have developed, QITABI-2 has needed to adjust its approach. It adapted several of its planned activities to respond to its beneficiaries’ changing needs. 6 Youssef, S., Jones, N., Małachowska, A., and Saleh, M. (2022) Adolescents in the abyss of Lebanon’s worst economic crisis: a focus on Lebanese and Palestinian adolescents’ education, and voice and agency. London: Gender and Adolescence: Global Evidence July 2022 Lebanon-Girls-Education-and-VA-WEB.pdf (odi.org)” 7 WFP Lebanon Situation Report - January 2023. https://bit.ly/3q0xrrU. Note, data refer to August 2022 figures this same figure was 53% of the population in December of 2021. 8 World Bank “Foundations for BUILDING FORWARD BETTER An Education Reform Path for Lebanon”. 2021. World Bank Document 9 QITABI 2 “Literacy and Numeracy Baseline Report Evaluation of Student Performance In Primary Public Schools In Lebanon.” SEPTEMBER 2022 10 ibid 11 Note that this number represents the number of operational schools at the time of the evaluation. This number continues to fluctuate as school strikes, staffing shortages, and limited resources impact schools’ ability to remain open. 4 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 EVALUATION METHODS AND LIMITATIONS The evaluation team conducted a two-phase process that was largely qualitative and included a document review and both remote and in-person data collection. The design matrix (see Annex III, Exhibit 8) maps how data sources responded to the evaluation and research questions. The data collection started with collecting secondary data for the document review and secondary quantitative data. Primary qualitative data collection occurred in two phases: Phase 1 was remote data collection with QITABI-2 and USAID staff, while Phase 2 was in-person data collection for school-level stakeholders, government partners, and other donors. DOCUMENT REVIEW The document review of 56 documents included QITABI-2 program documents, existing QITABI-2 assessments, evaluations of previous USAID education activities in Lebanon (i.e., QITABI-1 and Developing Rehabilitation Assistance to Schools and Teachers Improvement 2, or D-RASATI 2), and literature on Lebanon’s education sector (see Annex IV for a complete list). The document review started in October 2022 and ran iteratively throughout the entire data collection period until March 2023. The document review mostly preceded the implementation of primary data collection to inform the design and continued throughout in order to aid in analysis. QUALITATIVE DATA COLLECTION Data collection took place in two phases. The first phase of qualitative data collection began in December 2022 with QITABI-2 staff and continued with USAID staff in January 2023. The second phase of data collection began in January 2023 with in-person key informant interviews (KIIs) with government and donor representatives. Focus group discussions (FGDs) with regional and school￾level stakeholders (i.e., Regional Education Officers (REOs), teachers, principals, caregivers, and students) was postponed until March given teacher and ministry strikes and school closures. Data collection was completed in March 2023. Remote data collection employed purposeful sampling to target individuals who played critical roles in the development and implementation of the QITABI-2 activity. The evaluation team interviewed two key individuals from USAID and 13 QITABI-2 staff. The identification of interviewees reflected inputs from USAID and QITABI-2 staff. The number of interviewees per organization corresponded with the degree of involvement in the activity. In-person data collection built on Phase I to collect perspectives from stakeholders involved in QITABI-2 activities as well as government and donors. SAMPLING The sampling of schools followed a purposive approach that included schools with characteristics important to QITABI-2 activities—such as governorate, location, secondary language of instruction (LOI), shift, schools participating in the Learning Recovery Program (LRP) evaluation, and inclusive school—as well as the guiding inquiry questions. The evaluation team selected 16 QITABI-2 first￾and second-shift public schools from a list stratified by governorate, school shift, and secondary LOI. Principals from selected schools were asked if their school would participate in the evaluation. All but one of the selected schools participated. Among the 15 schools, there were two in each governorate (except Mount Lebanon, which had one school). The evaluation team conducted KIIs with school principals, and FGDs with teachers, caregivers, and students. The team engaged with principals to select teachers, caregivers, and students for participation in the evaluation in each school. There were 15 principal KIIs, 15 teacher FGDs (73 teachers – 69 women and 4 men), seven caregiver FGDs (31 caregivers – 24 women and 7 men), and eight student FGDs covering Grade 3 (27 students – 12 girls and 15 boys) and Grade 6 (20 students – 12 girls and 8 boys). 5 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 The evaluation team purposively selected government counterparts – nine from MEHE and five from CRDP – for KIIs about QITABI-2 activities. In addition, the evaluation team purposively selected and interviewed one representative each from Kreditanstalt für Wiederaufbau (KfW), United Nations International Children's Emergency Fund (UNICEF), World Bank, and Amideast to understand the education sector in Lebanon in terms of its needs, challenges, and opportunities. Exhibit 2: School-Level Data Collection by Governorate DATA COLLECTION PROCEDURES Data collection during Phase 1 was remote, using secure platforms, such as Microsoft Teams, Google Meets, and Zoom, per the preferences of the interviewees. If the interviewee consented to recording, it happened through the interview platform. During the Phase 2 in-person data-collection process, the team leader and the local education specialist led data-collection activities, with the team leader focusing on interviews with government officials and donors, while the local education specialist focused on school-based data collection. MEPL’s Senior Monitoring and Evaluation (M&E) Specialist and Data Analyst provided support and, in all cases, focus groups required both the local education specialist and at least one other team member as notetaker. Please see Annex 3 for details on ethics and security protocols as well as informed consent. DATA ANALYSIS, INTERPRETATION, SYNTHESIS, AND REPORTING The evaluation team developed a desk review matrix to track documents collected and coded information from documents against evaluation and research questions. The team applied standardized data extraction guidelines to facilitate a systematic process for document review and analysis. They recorded interviews and focus groups when agreed to by participants. Detailed notes were taken during interviews, and interviews were recorded to ensure accuracy and completeness of notes. Analysis was conducted on notes from data collection. Next, the evaluation team applied a combination of deductive and inductive coding approaches. The evaluation team developed transcripts directly within a Microsoft Excel database tailored to the data collection instruments. 6 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Once transcripts were completed and cleaned, the database served as a data analysis tool to code data based on the preliminary codebook. The database included a section for recording answers to informed consent questions, relevant quotes, internal notes, and iterative coding structures. Following preliminary analysis, the evaluation team gathered for an internal data analysis, interpretation, and synthesis (DAIS) meeting to consolidate findings on qualitative data. To validate and refine these preliminary findings, the team held a virtual finding meeting with USAID. The research team presented preliminary findings in May 2023. Following the participatory findings discussion, they developed the draft report for review by the USAID/Lebanon Education Office. A validation session with host country counterparts and QITABI-2 is planned as part of the feedback on and revisions to the draft report. LIMITATIONS AND BIASES The evaluation team faced delays in obtaining MEHE’s approval to collect school-level data in addition to delays resulting from public school strikes throughout Lebanon early on in the data collection phase. The KIIs were conducted over the telephone or Microsoft Teams, and planned meetings and site visits were rescheduled, some multiple times. The FGDs were conducted remotely in response to principals’ concerns about either taking teachers out of the classroom to participate or asking teachers to come to school early or remain after school to participate. The rescheduled field work was completed four weeks later than planned. Recall Bias. Beneficiaries may respond with answers that blend their experiences into a composite memory. The evaluation team mitigated this risk by conducting as many KIIs and FGDs as possible within the available evaluation period to triangulate responses and increase the validity of the findings. Response/Social Desirability Bias. This is the risk that key informants may provide responses that are socially desirable or influential in obtaining donor support. At the beginning of each KII and FGD, interviewers clarified that the responses would not influence the possibility of future assistance. Similarly, consent scripts were used to inform KII and FGD respondents that their answers were confidential. Selection Bias. The principals helped the evaluation team in selecting participant parents, teachers, and students. There was a risk that the principals would favor the most active, responsive, or engaged beneficiaries—meaning that the evaluation would hear from key informants who report positive experiences. CONFLICT OF INTEREST MSI, which is a subcontractor to World Learning on QITABI-2, is also a subcontractor to EnCompass LLC on MEPL. As such, to ensure objectivity in design, analysis, and reporting, MEPL has taken special precautions not to involve its MSI-affiliated staff in this evaluation work. FINDINGS EVALUATION QUESTION 1 (RELEVANCE) EQ1: How relevant was the activity to the needs of the direct and indirect beneficiaries? As indicated above in the Background section, the QITABI-2 Activity was awarded in March 2019, and a series of shocks to the education system occurred seven months after, beginning with the national uprising in October 2019. The timeline in Exhibit 3 illustrates some of the most challenging disruptions that during the award period alongside key policy developments. 7 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Exhibit 3: Contextual timeline for QITABI-2 (March 2019-2024) To answer EQ1, the evaluation team compared the QITABI-2 design with two key reference documents: the MEHE General Education Plan (2021-2025), and the USAID/Lebanon 2021-2026 Country Development and Cooperation Strategy (CDCS). (See EQ2 below for primary and secondary beneficiary reflections on their priority needs in light of the crises beginning in 2019, and the Effectiveness section for a discussion on project adaptations.) Exhibit 4 shows how the three pillars central to the MEHE General Education Plan are nearly identical with the 2021-2026 CDCS. Exhibit 4: Alignment of QITABI-2's results framework with the MEHE Plan and the USAID CDCS MEHE General Education Plan (2021-2025) USAID CDCS (2021-2026) QITABI-2 Results framework Pillar 1: To increase equitable access, participation, and completion of education for all learners in Lebanon, with a focus on the most vulnerable CDCS IR 2.1: Access to quality education expanded (p. 24) Pillar 2: To ensure improved learning outcomes for children and youth through enhanced quality of education services CDCS IR 2.2: Learning outcomes improved (p. 25) 1) improve student performance in reading, math, and writing 2) improve Social and Emotional Learning (SEL) skills Pillar 3: To ensure that the education system is effectively planned, managed, and governed, so that resources are transformed into high-level results. (p. 14) CDCS IR 2.3: Capacity to provide quality educational management systems improved (p. 26) 3) improve national-level service delivery of education. The QITABI-2 results framework aligns with the second and third areas of focus aligning with improving learning outcomes and strengthening the education system. The MEHE plan also 8 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 specifically mentions SEL as the “new curriculum will also emphasize the integration of Social and Emotional Learning (SEL) across the curriculum” and makes specific mention to QITABI-2 in that the SEL framework will be “co-constructed with DOPS [Département d’Observation Pédagogique Scolaire] and the QITABI II project” (p. 34). Finally, QITABI-2 also builds upon the strong foundation of relevance and collaboration established by the previous QITABI1 project, as concluded in its midterm evaluation12. EVALUATION QUESTION 2 (RELEVANCE) EQ2: In light of the crises, what do primary and secondary beneficiaries (teachers, CRDP staff, MEHE staff, students, caregivers) and other stakeholders now see as the priority challenges for primary education? A QITABI-2 staff member articulated how, due to the crises, Lebanon today is not the same Lebanon that the QITABI-2 original design had in mind. The response to EQ2 seeks to understand school-level beneficiaries’ as well as ministerial partners’ perspectives on evolving challenges. Most school directors, teachers, and caregivers interviewed stated that the Covid closures, the economic crisis, and the continuous teachers’ strikes, were all challenging for students. Further analysis shows the six priority areas that emerged across all interviews and focus group discussions. Learning loss: Within the context of overlapping crises since 2019 and various school closures, numerous interviewed stakeholders focused on learning loss as a priority for primary education. These comments aligned with the 2021 Learning Recovery baseline study that found that “parents, teachers and school principals expressed grave concerns regarding the quality of student learning in Lebanon over the past two school years” (p. 11). Caregivers continued to express this concern during mid-term evaluation activities. For example, one caregiver argued that… “…They did not get their rightful education due to the crises and strikes.” (Caregiver, FGD, Beqaa) “I would like to talk about the last three years. The primary classes can be followed up and they can catch up on everything and work successfully. As for the cycle 2 from grade 4 and above, they need more work. They will face more challenges and difficulties to catch up because three essential teaching years were lost on them. Their competencies are not fully acquired, they are struggling. They need a lot more work.” [FGD_50] In addition, three school principals also contrasted public school realities with private schools that have not felt the impact of strikes. Most of the teachers interviewed stated that the strikes had resulted in the deterioration of students’ competency levels and negatively affected their academic performance. A MEHE official interviewed mentioned how students’ learning loss had been devastating during the past two years, an opinion shared by at least one USAID staff member interviewed. A few stakeholders also emphasized that the learning gap affected the most vulnerable children the most. Teacher salary reduction: Stakeholders of all types identified teacher salary reductions as the greatest immediate challenge to primary education. As the Background section noted, teachers have seen a significant reduction in their salaries since 2019 due to the financial crisis. MEHE officials are well aware of the significance of this issue. As one official stated, 12 QITABI-1 Midterm Performance Evaluation. September 2017. Social Impact. 9 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 “The crisis is a big detour for the education sector. The teachers are demotivated. Due to the currency devaluation, their salaries lost their values.” (MEHE Official, KII) According to a variety of participants, including a school principal, three teacher FGDS, a CRDP official, two donors, and three implementing partners taking part in the mid-term evaluation, many teachers find themselves demotivated, and some have left teaching all together, resulting in a loss of critical human resources. Many prefer to stop working than spend their meager salary on transportation. As discussion in one teacher FGD shared “Honestly, what we are getting paid isn’t fair. The last two years we barely got paid, and we aren’t getting any help with transportation money.” (Teachers FGD, South, School #2) SEL needs of students and teachers: The need for greater attention to students’ and teachers’ social emotional learning and well-being also arose as a priority area for primary education. A CRDP official interviewed underlined the importance of fostering student and teacher well-being, highlighting the fact that the MEHE General Education Plan integrated SEL programming for students to this end. Regarding teachers, the learning recovery study found that “84.5 percent of the teachers surveyed reported being anxious about their current living situation; 72.3 percent experienced frustration as a result of the current crises; 39.9 percent noticed that they have become moodier; 30.8 percent reported feeling worthless; and 10.5 percent felt total emptiness.” Teachers from a mid-term evaluation focus group corroborated this point, provided additional detail, and expressed urgency for supporting teachers’ well-being: “Honestly, during these times, even the teachers are not relaxed and need social and emotional support. We are not in a good psychological or social emotional state.” (Teachers FGD, Hermel School #2) Lacking transportation: Nearly all school-level stakeholders agreed that transportation was a common challenge. Financial challenges mean that many parents who used to pay the cost of transportation for their children to attend school may now struggle. Several teacher focus groups observed that there was a lot of absenteeism and students were dropping out due to the high transportation fees. The two quotes below illustrate this point of view: “Most of the challenges are related to the financial problems because a big number of students come to school on foot (…) because they cannot afford to pay for transportation.” (Teachers FGD, Tripoli, School #1) “After our return, following the strikes, several students dropped out, and their parents told the principal that they could not afford to send their children to school because of the expensive transportation fees.” (Teachers FGD, Beirut, School #2) Food insecurity: Interviews also frequently revealed concerns around children and teachers not having sufficient food to be able to focus on learning. Most of the caregiver focus groups described how their priorities had shifted to ensuring that their children had a decent meal. The majority of caregiver groups stated that it was hard for them to cover their children’s needs because costs had increased drastically. Similarly, many teachers and school principals interviewed also expressed concerns that students struggled with the lack of food and snacks at school, which made it difficult for them to focus on learning. As one teacher expressed during a focus group in Hermel, “Some students come in without even having a sandwich to eat. One of the students was crying, and when we asked her why, she answered because she did not have a sandwich and could not afford to buy something eat.” (Teachers FGD, Hermel #1) 10 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Some interviewed ministerial staff and implementing partner staff also raised concerns about children not having enough to eat. As one CRDP official indicated, “We have a lot of parents [that are] very depressed because they don't work. They don't get good salaries due to the crisis, and the children know that their parents cannot buy bread for them… What's happening is very sad” (CRDP Official, Interview) Lacking resources at school: Due to the financial crisis, critical resources like electricity, heating, and paper are lacking at many schools, per numerous teachers and school principals. During nearly half of the teacher FGDs, teachers lamented not having access to supplies like they used to prior to the crisis. Reduced access to photocopying is the most common problem, due to limited ink and paper. Electricity is also a related and significant concern, as power cuts are frequent and teachers are not able to utilize projectors, further compounding the need for photocopies and counteracting teacher training content. According to one teachers FGD, “We are now able to display lessons on screen in class, but most of the time there is no electricity, so we go back to the old teaching ways.” (Teacher FGD, South, School #2) Another school principal explained how the school was going into debt to pay for fuel for heating. This principal stated, “Providing fuel for the central heating system in our schools is a challenge as well. Our school is located at a high altitude and requires a lot of heating. UNICEF provides around 3,500 liters per year; however, we need around 600 liters per day to light and heat the schools.” (School principal KII, Baalbek) EVALUATION QUESTION 3 (RELEVANCE) EQ3 A: To what extent has QITABI-2’s implementation adapted to the changing priority needs that have emerged since the original project design? Were these adaptations successfully applied? QITABI-2 integrated several program adaptations to support MEHE’s response to crises and the system’s changing needs. During the COVID lockdowns, QITABI-2 worked closely with MEHE and CRDP (Pre-Service and In-service Training Bureau and Joint Academic Departments) to implement their “Distance Learning” initiative and subsequently, the “Back to School” initiative. Examples of support during COVID include the development of E-content (13-week abridged content) and delivering training to CRDP on the production of digital lessons and to DOPS on supporting teachers in distance-learning approaches. QITABI-2 also collaboratively developed SEL materials, including brief animated videos and DOPS messages for parents on “self-care for caregivers, signs of stress and anxiety, coping mechanisms for children, challenges of distance learning, and online Bullying.”13 To support students’ reintegration into schools, QITABI-2 again worked with MEHE, CRDP, and UNICEF to deliver the Summer School Program in both 2021 and 2022. The most significant example of QITABI-2’s adaptation was the development of the LRP, which was launched in the fall of 2021. The LRP was created to help students re-enter school, and its design was informed by QITABI-2’s Learning Recovery Study. The plan was designed and developed collaboratively with 13 QITABI QR 7 October-December 2020. 11 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 MEHE, CRDP, and UNICEF. Though designed initially as a four-week program, the LRP was expanded to a two-year program halfway through the fall semester. The LRP encompassed a five-phase national training and coaching strategy for all public-school teachers for Arabic, English, French, and Mathematics for grades 1-6. It included a diagnostic assessment to allow for differentiated learning, an abridged 13-week curriculum, formative assessment, and teaching and learning resources and instructional approaches that included active learning, additional learning time, and summer school. SEL instruction for teachers was also integrated into the LRP to help teachers strengthen learning environments (LRP evaluation report 12.22) by helping teachers to better understand and support students’ socio-emotional needs and to introduce teachers to strategies for their own self-care.14 Examples of QITABI-2’s program adaptations are listed below. Exhibit 5: Illustration of QITABI-2’s adaptions in response to changing needs EQ3 B: To what extent do beneficiaries feel that the adapted QITABI-2 activity design effectively addresses their needs related to primary education? Data from KIIs and FGDs with school-level and government stakeholders show that many beneficiaries felt that QITABI-2’s adaptations responded to their needs. Generally, there was consensus among MEHE, CRDP, and other donor stakeholders participating in the mid-term evaluation that QITABI-2’s support was very relevant and addressed MEHE’s evolving priorities. 14 QITABI 2 Learning Recovery Program Evaluation Report. December 6, 2022. Need Illustration of QITABI adaptions in response to changing needs Providing access to education during school closure QITABI-2 supported MEHE Distance Learning Project:  Development of E-content (13-week abridged content).  Training to CRDP on the production of digital lessons.  Training to DOPS on supporting teachers in distance-learning approaches.  Educational Boxes distributed. Addressing increased anxiety and depression among children during school closure Integration of SEL approaches in LRP  SEL materials for DOPS and parents to support their children (during school closure).  Educational boxes. Addressing learning loss and anxiety and depression once schools re-opened Support to MEHE Back-to￾School Plan: Summer School Program Learning Recovery Program  Summer School Program.  Learning Recovery Program.  Integration of SEL approaches in LRP materials and e￾content. Need to provide school materials and food for homes given financial crisis Provision of resources  Food parcels.  Educational boxes. 12 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Exhibit 6 pairs four key themes of relevance to beneficiary needs that surfaced from stakeholders with illustrative supporting quotes. Exhibit 6. Stakeholder perspectives on ways QITABI-2 adaptations met their needs Development of the summer school program: Very important in helping children transition back to school – especially the second iteration. "The [QITABI-2] summer program helps kids refresh their memory and be ready for school. We assessed the kids and fixed the issues they had. (…) It has a positive impact on the teachers, and it helps them financially, which encourages them even more.” (Teachers FGD, South, School #2) “…the content was effective and helped students acquire what they had missed to reach their grade levels.” (School principal, Interview, Akkar) Materials developed and digitized: MEHE and CRDP officials interviewed emphasized the value of the materials developed and how their digitization was crucial to prepare for future crises and school closures. “…there were two things with which QITABI complied quickly. These were the digital resources and the eBook (…) That's why we created the eBook because we couldn't print books for all students in Lebanon since no company would take bids and there were no books for students. This led to creating eBooks, and it evolved with the help of QITABI.” (CRDP Official, Interview) Provision of food parcels and educational boxes: Many school-level actors found the food parcels and the educational boxes to be responsive to their needs. However, the decision to target students in Cycle I and Cycle II created equity concerns. “All materials provided, such as books, tools, and educational boxes, were of high quality and motivated both the teachers and the students." (School principal, Interview, Beirut) “The educational boxes and the food parcels, as well as the provision of stationery, were also an answer to the needs. Thanks to the flexibility of QITABI-2.” (MEHE Official, Interview) “We appreciate having all cycles included in the program. Yesterday, there was food parcel distribution for cycles 1 & 2, the parents of early childhood education and Cycle 3 did not receive anything, and it was very difficult, especially now due to the ongoing economic crisis we are facing." (Principal, Interview, Nabatieh) The three themes above emphasize key positive ways in which QITABI-2’s adaptations aligned with beneficiaries’ needs. At the same time, some stakeholders also expressed limitations to the interventions and desires to expand the scope and strengthen the program. The next two evaluation questions on effectiveness examine more deeply the successes and limitations of QITABI-2’s interventions once implemented. EVALUATION QUESTION 4 (EFFECTIVENESS) EQ4: To what extent has the Activity achieved its intended outputs and outcomes in terms of improving student performance in reading, math, and writing; instilling students with SEL skills; and improving the national-level service delivery of education? Various sources informed the evaluation team’s findings regarding effectiveness, including project assessments, quarterly reports, indicator data, and qualitative data from stakeholder interviews. It should be noted that these findings reflect quarterly reports and indicator data available at the time of the evaluation data collection (December 2022-March 2023). 13 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Objective 1: Improved Student Performance Of the 1115 indicators related to Objective 1, QITABI-2 exceeded three output indicators: 1) the number of primary or secondary educators who complete professional development activities with U.S. Government (USG) assistance (KPI 4), 2) the number of primary school educators completing professional development activities on evidence-based math instruction (Ind-1c), and 3) the number of teaching and learning materials (TLMs) provided (KPI 6). QITABI-2 also came close (80 percent or more of target) to meeting three other output indicators and one outcome indicator. These output indicators refer to the number of schools receiving USG assistance (KPI 3), the number of schools receiving educational materials and equipment (Ind-1d), and the number of learners in primary schools or equivalent non-school based settings reached with USG assistance (KP 5). The outcome indicator that nearly met the FY22 target (at 99.5 percent) concerns the number of learning environments that have improved safety (KPI 916). One output (standard) indicator related to the number of education institutions/environments supported with preparedness, prevention and/or response interventions (KP7) was not reported on by QITABI-2 due to the lack of a precise definition for it. In addition, reporting on the outcome indicator measuring the percent of learners who demonstrate reading fluency and comprehension of grade-level text at the end of Grade 2 with USG assistance (KP1) is scheduled for Year 4. A similar indicator measuring the percent of learners attaining a minimum grade-level proficiency in reading at the end of Grade 3 (KP8) and reading and fluency at the end of primary (KP2) are scheduled for Year 5. (See Annex 4 for Comprehensive Indicator Analysis.) COVID closures delayed QITABI-2’s assessment of students’ reading and mathematics skills in Arabic, English, and French. The EGRA, the EGMA, and the curriculum-based assessment (CBA) baseline study were postponed from the fall of 2019 to April of 2022. As the endline student assessment is scheduled for the spring of 2024, the evaluation team cannot measure QITABI-2’s impact on student performance. The impact of COVID, the economic and financial crisis, and the ongoing school closures have resulted in extensive learning loss that is outside of the project’s control. According to the 2022 QITABI-2 baseline, teachers estimated that at the beginning of the 2021-2022 school year, students in public schools were performing 1 to 2 years below grade level. School closures due to teacher strikes continued throughout the 2021-2022 school year, reducing effective school days to 59 and severely limiting opportunities for students to catch up. The April 2022 baseline showed that the majority of 2nd, 3rd, and 6th-grade students lacked foundational skills in Arabic, French, English, and Math. The impact of the closures appears to be strongest among earlier grades where students did not have foundational skills to build on that older students were able to acquire before the pandemic.17 Despite the overall low performance, an internal evaluation indicated that the application of the Learning Recovery Program (LRP) showed promise as an approach to reintegrating students into schools, providing teachers the structure needed to support student re-entry, beginning to improve teacher practice, and beginning to strengthen student performance18. The LRP evaluation was conducted in 120 QITABI-2-supported schools and investigated changes in student performance and teaching practice between February and June of 2022. Student performance in all literacy and 15 Thirteen indicators are included within the QITABI 2 indicator tracking table and MEL plan. Two of them (Ind 1.1 and Ind 1.2) report on improvements in reading and math levels respectively. The values are the same as the LRPE values presented in the exhibit within this section. The evaluation team did not have enough information to confidently include mention of the two indicators within the report. 16 The QITABI-2 team clarified that this target was not fully met due to the decrease in the number of operating schools. 17 QITABI 2 “Literacy and Numeracy Baseline Report Evaluation of Student Performance In Primary Public Schools In Lebanon.” SEPTEMBER 2022 18 QITABI 2 Learning Recovery Program Evaluation Report. December 6, 2022. 14 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 numeracy domains improved. Assessed schools saw an increase in the percentage of students performing at the previous grade level (as opposed to below the previous grade level). For example, close to half of students improved their reading comprehension in the previous grade level’s reading comprehension standards by at least one reading level (Exhibit 7).19 Not surprisingly given the level of learning loss and the short evaluation period, most students were still performing at or below the previous grade level. It should also be noted that this evaluation was conducted in schools that were receiving more frequent teacher mentor and DOPS visits, and, without the benefit of a counterfactual, it is difficult to definitively attribute these results to the LRP. Exhibit 7. Percent of students who improved by at least one reading level Category Percentage of Students Who Improved by at Least one Reading Level Overall 48% Arabic 47% English 41% French 57% Source: QITABI-2 LRP Evaluation 2022 Similarly, the LRP evaluation indicated that teacher performance, based on both the teachers’ self￾assessment and DOPS observations, improved over the course of the brief (8-10 week) evaluation interval. According to the teachers’ self-assessment, more than 80 percent of teachers were adequately applying the LRPM. DOPS observed classrooms and rated teacher performance along four domains (i) Planning and preparation, (ii) Classroom Environment, (iii) Teaching and Assessment, and (iv) Professional Development, and the evaluation recorded improvements. However, despite these improvements, results indicate a need for continued support, as DOPS results showed teacher performance at or just above 50 percent of the potential maximum score for each domain assessed. In addition, FGDs from both the spring 2022 LRP evaluation as well as the current performance evaluation indicate that teachers’ integration of LRP materials and instructional approaches varied considerably. Encouragingly, student performance was positively correlated with teachers’ adequate adoption of the LRP, indicating the promise that the LRP holds.20 School-level interview data further provide indication of improved teaching and learning despite lacking outcome results. Three aspects of QITABI 2’s approach emerged as particularly effective: • Active Learning Approach: Almost all students stated that they had benefitted from QITABI-2’s activities and found them enjoyable. Teachers similarly stated that students retained information when explained through an activity rather than a didactic and theoretical way. Also, the majority of teachers said that the usage of flashcards, role-playing, storytelling, and fun games with their students had helped them perform better. Caregivers also noted that their children had enjoyed the variety of activities. The following excerpts illustrate these points: “They really like reading stories or listening to me reading the stories out loud. They keep asking me ‘When are you going to read us a story? When are we going to read stories?’ I have noticed a big difference in this regard.” (Teachers, FGD, Beirut School #1) • LRP Materials: Many teachers and school principals interviewed reported that the teaching and learning materials were effective and enjoyable. Caregivers mentioned the classroom libraries and noted that their children enjoyed reading stories both in class and at home. The majority of student 19 QITABI 2 Learning Recovery Program Evaluation Report. December 6, 2022. 20 ibid 15 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 focus groups agreed that the books were useful and discussed how when teachers integrated activities from QITABI 2’s approach, they understood the lessons better. For example: “I loved QITABI-2’ books. They helped me a lot to learn how to read and pronounce letters.” (Grade 3 Student, FGD, Chehim) “The Arabic box was distributed to all classes which is a box that has activities. This box has alphabets in it. It would be on the board and read every day. The kids really liked it. They would even remind me if we didn’t read the sounds of the letters or the names of the letters. It indeed affected us and the students positively.” (Teachers FGD #70) “If we apply all the materials and use the tools provided in the Box, we can achieve better results. We started working on some stories, checking the general ideas (…) but the box needs to be used to achieve more progress.” (Teachers FGD #60) • Diagnostic Assessment: A fundamental component of the LRP is the integration of diagnostic assessment, which teachers administer to their students to measure their competency level and determine what differentiated instruction is needed. Nearly half of the teacher focus groups from a variety of governates showed that the teachers were able to identify their students’ weaknesses and work with them after receiving training from QITABI-2. As teachers explained, “[The assessment] was really helpful because it allowed us to test the students objectively and explore their levels and true needs. The assessment is very accurate and helps us discover the real needs and gaps of each student." (Teachers FGD, Hermel, School #1) “This year was easier than the previous one because we were able to assess the kids’ levels from the beginning, and I changed a lot in my teaching strategies, but I stopped them after a few months." (Teachers FGD, Tripoli, School #2) At the same time, some stakeholders indicated concerns they had with both materials and assessments. EQ5 will explore these observations as part of hindering factors. Objective 2: Improved Social and Emotional Learning There are seven objective 2 indicators. QITABI-2 achieved its three output indicators (Ind 2.1, 2a and 2e) which reflected the incorporation of SEL-related skills in learning content and teacher training in LRP and the summer catch-up program. QITABI-2 exceeded its 2022 targets for two of these indicators (Ind 2.1 and 2a) by a large margin. The third indicator (Ind-2e) was focusing on private school students and was no longer relevant as priorities shifted towards food parcels and the LRP and away from private schools. QITABI-2 is still rolling out the SEL program, as this was delayed by COVID, so no outcome data are currently available. However, qualitative findings indicate that many beneficiaries believe that QITABI-2’s SEL activities are helping to address a stated need. There was agreement among all school principals interviewed that the well-being of students was very low because of COVID-19 and the financial crisis. MEHE staff said that the SEL component, implemented with DOPS, had helped teachers respond to different student needs. Teachers and principals indicated via KIIs and FGDs that the integration of SEL in their teaching benefited students and encouraged them to be more open about their feelings. As students became more expressive, teachers gained a better understanding of the students’ needs. Teachers shared that the SEL curriculum provided practical skills and positive motivational ways of teaching. SEL relates to real life situations and helps in learning life skills, such as how to make decisions, how to deal with certain situations, and how to manage the reactions of others. Example quotes related to the benefits of QITABI 2’s integration of SEL include: “Previously, if students had disruptive behavior in class, teachers would ask them to leave the classroom. Now, teachers are dealing better with the students because of QITABI-2.” (School principal, Interview, Beqaa) 16 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 “The students are able to express their feelings and opinions, and this is mainly obvious in the writing exercises. I think these activities have a very positive impact on our students.” (Teachers, FGD, Hermel, School #2) “The yarn ball game is used with students to express their feelings. When throwing the ball, each student is asked to express his/her problem or feelings related to school.” (Teachers, FGD, South, School #1) “We always try to lighten the mood in class for the students and to have them do fun activities because we know that they are emotionally tired.” (Teachers, FGD, Beirut, School #1) In addition to the direct beneficiaries’ statements noted above, MEHE and CRDP noted the importance of QITABI-2’s role in establishing the National Social and Emotional Framework. The framework has informed the training content developed in collaboration with CRDP. According to at least one CRDP official, it was a huge success. Objective 3: Improved National Level Service Delivery of Education Only two output indicators inform Objective 3, and QITABI-2 has exceeded its 2022 targets for both. The first (Ind-3.1) refers to the number of new management and monitoring systems, procedures and tools recommended and/or introduced as a result of USG assistance. The second refers to the number of education administrators and officials completing professional development activities. There are no outcome indicators for Objective 3. In order to identify and prioritize activities needed to strengthen the national-level service delivery of MEHE and CRDP, QITABI-2 carried out Human and Institutional Capacity Development (HICD) studies and developed a Performance Improvement Action Plan.21 Many of the planned training activities related to these action plans were postponed given the shifting priorities. For example, funding originally slated for capacity-strengthening was reallocated to fund the food parcel following COVID and the financial crisis.22 The HICD capacity-strengthening activities that did take place shifted from the original focus on the Holistic Learning Approach (HLA)/Balanced Literacy Approach (BLA) in favor of more urgent trainings related to distance learning, HICD, and English communication skills (Indicator 3.1). Despite the changes in programming due the crises, close collaboration between QITABI-2, MEHE, and CRDP provided extensive capacity- and system-strengthening opportunities. For example, the MEHE-General Directorate of Education assisted with planning for the summer school and setting up DOPS online coaching system,23 CRDP teacher trainers collaborated in the development of the LRP teacher training materials and approach, and CRDP staff worked with QITABI-2 on digitizing LRP materials. Other examples of Objective 3 accomplishments cited by QITABI-2 include the project’s support to CRDP in the development of both the Lebanese Curriculum Framework and CRDP’s five-year strategic plan24. QITABI-2 also supported the development of the DOPS Coaching Framework and put in place plans to support the implementation of the Document Management System for MEHE.25 21 QITABI 2, Quality Instruction towards Access and Basic Education Improvement 2 Fifteenth Quarterly Report | October-December 2022. January 2023. 22 Discussion with USAID and QITABI team. 23 QITABI 2 Thirteenth Quarterly Report | April-June 2022. July 2022. 24 ibid 25 QITABI 2 Fifteenth Quarterly Report | October-December 2022. January 2023. 17 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 EVALUATION QUESTION 5 (EFFECTIVENESS) EQ5: What were the factors that contributed to or hindered the a) adoption of education interventions promoted by QITABI, and b) the achievement of the intended results? The response to this question seeks to identify underlying factors that supported or weakened QITABI-2’s effectiveness. Findings build upon the relevance and effectiveness sections above and draw from interview data as well as document review. Contributing Factors Many factors seem to have enhanced the effectiveness of QITABI-2’s programming. These include: • Alignment with education system goals and priorities: QITABI-2’s programming, as noted in EQ1, closely aligns with the MEHE’s goals and the education system’s shifting priority needs. By co￾designing many of its interventions with MEHE, CRDP/DOPS, teacher trainers, and UNICEF, QITABI-2 gave MEHE and CRDP ownership of these interventions, which are aligned with the systems and approaches used by the ministry and its staff, thus facilitating adoption and sustainability. Close collaboration with UNICEF facilitated the leveraging of UNICEF funding in support of the LRP and summer programming. • Highly regarded project team: Discussions with MEHE and CRDP officials as well as donors underline how QITABI-2 is highly regarded and how its team is considered strong technically, well￾organized, and communicative, with regular contact with MEHE staff. QITABI-2’s team members themselves appreciated the project’s leadership, which inspired them to keep on looking for solutions. MEHE and CRDP/DOPS see QITABI-2 as a true partner. As a MEHE official indicated, the strong relationships and respect that QITABI-2 developed across MEHE and CRDP leadership has allowed its team to move things forward and get them done. • Holistic approach: Evaluation respondents also appreciated QITABI-2’s holistic approach, noting that QITABI-2 took into consideration students’ and teachers’ well-being as well as students’ academic needs. According to some teachers, “In summer school, there was a support class for mental health where kids were able to let off a lot of things, and it's one thing we didn’t have in regular school.” (Teachers FGD, Tripoli, School #2) “What I liked the best is the variety of activities that could be done in each session. The activities that were provided by QITABI-2’s programs and the variety they added to the classes were the main benefits. In addition, the SEL and care related to the students’ emotional and psychological states are also important benefits and gains.” (Teachers FGD, Hermel, School #2) MEHE and CRDP respondents also noted that QITABI-2 worked across administrative groups within MEHE, CRDP, and DOPS as well as with school principals, teachers, and students. The incorporation of active learning strategies (see EQ3) also contributes to a holistic approach. • Research-based approach: MEHE and CRDP officials interviewed said they appreciated the fact that QITABI-2 applied a research-based approach to their work. Examples include QITABI-2’s piloting, evaluating, and adapting their interventions; analyzing DOPS data; holding reflection sessions; and conducting large-scale student assessments. As one MEHE official emphasized, “QITABI’s assessment of their own progress and their own results is very scientific and allows for better progress towards results.” (MEHE Official, Interview) 18 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Hindering Factors Several factors seem to limit the effectiveness of QITABI-2, including: • School closures: As indicated earlier, repeated and lengthy school closures have reduced the amount of instructional time available for learning and have disrupted the adoption of the LRP. Some teachers indicated that they had not had time to apply the complete LRP approach. As two principals shared, “Everyone was interested in the trainings and the tools provided, but the teachers said they had not had time to apply much of what was learnt because they had a condensed curriculum to follow.” (School principal, Interview, Beqaa) “When the teachers use the new techniques in their work, the students benefit a lot, but due to the continuous strikes, improvement was not very noticeable.” (School principal, Interview, Beirut) • Transportation costs: The increase in transportation costs as a result of the financial crisis negatively impacted student and teacher attendance at school (see EQ1), in turn limiting QITABI 2’s effectiveness. • Challenges with distance learning due to COVID: Data from stakeholder interviews point to connectivity issues and/or weak digital literacy, which limited access to distance learning programming. Almost all teacher focus groups identified the lack of devices, internet, and electricity as a huge limitation to the distance learning process. “During the online teaching period, most students did not have access to phones or the internet. These accumulated reasons had extremely negative repercussions.” (Teacher FGD, Hermel #2) “In a context where there is inconsistent access to electricity (…) and the internet, what does remote learning look like? How much support can we expect families to give to their kids in light of this economic situation (…) Families need to think about putting food on the table first.” (Donor, Interview) Of the 15 principals interviewed, two spoke negatively of distance learning in general. Similarly, three caregiver FGDs indicated that children had been negatively affected during online learning, as they had not been able to attend the sessions due to the limited number of devices at home. It is possible these sentiments may be more widespread. • Lack of time and space and teacher training and mentoring: During the FGDs, some teachers shared how various factors had prevented them from fully implementing the LRP approach, including the limited physical space in the classrooms and the limited time available for learning and teaching. The LRP evaluation also found that while teachers liked the program and the training content, they often reverted to more classic instructional methods due to time restraints, resistance to change. When asked about recommendations for future programming, about half of QITABI-2’s staff, REO staff, principals, and teacher focus groups talked about the need to continuously improve training, including training to successfully integrate the LRP. IP and MEHE respondents pointed to the important role of principals in the LRP’s effectiveness and the need for dedicated principal training. As a QITABI staff member indicated, “Principals are in need of capacity building...” (Interview) The respondents also focused on the need to improve the logistics of teacher training. For example, they requested that books be distributed earlier, that sessions target fewer teachers at a time, that the length of training sessions be reduced, and that training be differentiated to accommodate teachers’ differing levels of teaching experience. To a lesser extent, teachers suggested ways to improve the content of the training, including allocating more time for writing in Arabic, placing more emphasis on English and French language skills, and adding material related to the appropriate 19 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 use of technology and digital learning and how to deal with bullying. Providing teachers with more support and feedback through coaching is another area that various types of stakeholders emphasized when asked about ways to improve QITABI-2. Two MEHE staff emphasized coaching as an area for improvement and indicated the potential for working more with DOPS to systematize the use of observation tools and results. • Procurement delays and difficulty level undermine full effectiveness of TLMs: Although QITABI-2’s Teaching and Learning Materials (TLMs) contributed to perceived improvements in teaching and learning experiences (see EQ4), findings indicate areas that require additional attention. First, hyperinflation and the devaluation of the national currency delayed procurement, which caused delays in the distribution of TLMs to some schools. Four of the 15 school principals interviewed across three governates (Baalbek-Hermel, Beqaa, Nabatieh) stated that the TLMs26 had arrived after the start of the academic year. School principals in Akkar, Beqaa, and Nebatieh also noted that the insufficient numbers of TLMs delivered meant that photocopies had to be made to cater to all students, which added to the financial burden. Furthermore, mixed commentary emerged among school-level stakeholders about the difficulty levels of the TLMs. For example, while two school principals from Mount Lebanon and South Lebanon said the books corresponded to students’ levels, two school principals and three teacher focus groups from Baalbek-Hermel and Beirut said the books were actually too easy, yet still beneficial to students. Two other school principals from Akkar and South Lebanon said the books were much below students’ levels. “We need more activities, and we need them to be a bit harder because they were very simple. We need it to be a bit harder than before.” (Teacher FGD, Beirut, School #1 “Books were at the right level especially for students who were not high achievers.” (Principal, Interview, South) It is unclear how widespread these reflections are. The TLMs were designed to support students who are performing below grade level. Therefore, the Evaluation team is uncertain if these reflections indicate that the materials are too easy, or that additional training and mentoring to ensure that teachers and principals are aware of the intent of the LRP TLM materials. • Outdated curriculum: Many stakeholders including teachers, CRDP and MEHE staff, and implementing partners noted that the outdated national curriculum prevented the full realization of QITABI-2’s promise for teaching and learning reform. The curriculum dates back to 1997. Quotes from the teacher FGDs provide an illustration of these frustrations: “All the training we are undergoing is useless if the official books are not changed to accommodate the changes and adapt to the new teaching and learning techniques and processes.” (Teachers FGD, Hermel #1) “What matters to us is the curriculum (…) I started teaching in 1997, and the curriculum has been the same ever since. There are some things that should be canceled, things that should be easier. There are some things that should be changed.” (Teachers FGD, Kab Elias) Current on-going efforts and working groups to revise the curriculum, including QITABI’s support of the curriculum mapping, seem to be promising. 26 TLMs were printed and distributed by UNICEF. 20 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 EVALUATION QUESTION 6 (SUSTAINABILITY) EQ6: At this mid-term point, what signs emerge that project activities and results may continue after the project ends? What factors may hinder the sustainability of interventions? Analysis of interview data demonstrates that there is great enthusiasm among many teachers, school principals, and government officials that educators can sustain their improved capacity into the future. Through QITABI-2’s training program, teachers gained new techniques and tools that they can continue to use as a reference after the project ends to carry on developing their competencies and skills. According to one principal, “Teachers now have resources and techniques that they can use as a reference later on even when the project ends.” (Principal, Interview, Baalbek) Moreover, system-level factors associated with QITABI-2’s effectiveness also contribute to the sustainability of its programming. Alignment with MEHE’s priorities and integration within existing systems are factors recognized to enhance the sustainability of education programming27. The close collaboration with MEHE and CRDP in the design and development of QITABI-2’s activities is also a critical element for sustainability, according to many stakeholders. For instance, one MEHE official stated that sustainability was ensured, “… because it [QITABI 2] answers MEHE’s needs and works with us. The partnership with MEHE is very crucial to ensure the sustainability of the interventions.” (MEHE Official, Interview) Additionally, many components of QITABI-2 were designed to rely on existing systems and resources. For example: 1) The LRP teacher training was delivered by CRDP trainers rather than by QITABI-2 itself. 2) DOPS (rather than QITABI-2’s Learning Facilitators) conduct classroom observations and provide teacher support. 3) QITABI-2 helped strengthen the local production of reading materials by recruiting and training Lebanese volunteer authors to write stories that are more relevant for Lebanese students. 4) A QITABI-2 team members embedded at CRDP supports the center’s staff to digitize teaching materials and the creation of e-learning products. CRDP noted that the use of CRDP staff in the design and implementation of much of QITABI-2’s programming was a conscious decision designed to enhance the sustainability of the project’s activities. One CRDP official was particularly enthusiastic about this aspect of QITABI-2’s activities, saying: “…the resources are already there on the website, so everybody can use them now or later. They are almost entirely validated by CRDP. The ones that are validated will appear automatically on the website, and we are in the process of validating all the lessons, but it will take time because there are many (…) So, each time we validate a new lesson, it will appear on the website. (…) Any teacher or student can access our platforms to see the lessons.” (CRDP Official, Interview) In addition to the reliance on existing systems, QITABI-2’s efforts are being integrated into education sector planning and reform efforts. For example, MEHE noted that QITABI-2’s approaches were integrated within its five-year education plan. QITABI-2’s work on the development of the National SEL Framework, input for the development of the new curriculum framework, and 27 QITABI 2 “Literacy and Numeracy Baseline Report Evaluation of Student Performance In Primary Public Schools In Lebanon.” SEPTEMBER 2022 21 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 involvement in the co-development of the LRP promise to have a long-lasting and profound impact on the education sector in the years to come. Potential Hindering Factors The most significant barriers to QITABI-2’s sustainability are financial. Current MEHE financial resources and financial and accountability systems necessitate dependence on foreign aid to cover school operation costs. This dependence hinders the sustainability of any education programming. The devaluation of teacher salaries and MEHE’s inability to independently supplement teachers’ salaries to help offset some of this devaluation have resulted, as previously noted, in teacher strikes and extended school closures. World Bank funding was needed to provide the necessary supplementation of teacher salaries. With very limited time in class, teachers’ full adoption of QITABI-2’s approaches and student learning will be limited. Despite integration into existing systems, the lack of financial resources meant that QITABI-2 programming was dependent on UNICEF funding for operational needs in addition to planned USAID funding. For example, in addition to USAID support, UNICEF financed the summer school by covering teachers’ incentives, students’ transportation fees, and the costs of printing learning recovery classroom materials. With regard to specific QITABI-2 activities, the Ministry’s DOPS directorate does not have the human resources needed to coach teachers at all public primary schools with the intensity of teacher coaching visits provided within a sample of 120 schools. As such, support needs to be provided to DOPS in the use of data to help identify schools that should be prioritized for support. Similarly, CRDP does not have the human resources needed to conduct the EGRA and the EGMA independently. Similarly, the work of the learning facilitators depends entirely on the project. As one MEHE official stated, “The work with the learning facilitators is not sustainable, since they are paid by QITABI; and once QITABI ends, they won’t continue, and all the learning will not be sustained.” (MEHE, Interview) Though there may be great enthusiasm for the e-learning and digital materials that are being developed, few teachers and students are accessing these due to a lack of electricity, internet connectivity, and digital literacy. RESEARCH QUESTION 1 RQ1: Looking broadly at the basic education sector (pre-primary, primary, secondary, and technical vocational education, and higher education) what do key local and international education stakeholders with a global understanding of the sector feel are priority challenges? Within particular education levels? At a sector-wide or systemic level? How do these perceptions align with the most recent USAID Lebanon CDCS and MEHE education sector strategic plan? The responses to this research question reflect findings similar to those of EQ2 but broaden the perspective to include education sectors beyond primary. The findings draw from interview data as well as from project documents and published research studies. MEHE respondents, donors, and other national and international stakeholders cited common priority issues for Lebanon’s education system. Many of these reflect the on-going crises while others reflect underlying challenges with the quality of the education system itself. Addressing learning loss was a primary concern of the respondents. However, they noted even more fundamental concerns that needed to be prioritized before addressing learning loss. One ministry respondent encapsulated the situation in this way: “We need to eat and keep schools open” (MEHE, 22 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Interview). Securing access to nutrition is becoming an increasingly urgent issue in Lebanon. WFP estimates more than 2 million people in Lebanon are facing food insecurity and are in need of aid.28 • Addressing nutrition needs. Families who struggle to put food on the table are less likely to have money to pay for school. Students who are hungry tend to have lower cognitive function, are less able to concentrate, put forward less effort while at school,29 and are less able to retain information or engage while in class. Lack of nutrition, especially among younger children, can have a lasting impact on their cognitive development.30 Some respondents noted the need to establish feeding programs at school in order to ensure that children are getting at least one good meal a day. “We hear stories of families having to prioritize one meal per day or even, unfortunately, having to select only one member to eat today and the other to eat the next day. (…) There is a need to support schools with a hot decent meal for all children.” (IP, Interview) “Providing meals or snacks is needed in school as well, as some students come to school without a sandwich or snack.” (School principal, Interview, Beirut) • Addressing barriers to school access. After COVID, the primary challenge to keeping Lebanese schools open has been the extreme devaluation of teacher salaries and the resulting teacher strikes and school closures that left students out of school for months at a time. For these reasons, many respondents prioritized the need to increase teacher salaries. Even when schools are open, ensuring enrollment and attendance in school is becoming increasingly challenging. “The most crucial thing is to bring kids back to learning or ensure that they can catch up on missed learning. The second crucial thing is to ensure that they have access to schools.” (Donor, Interview) MEHE noted that the dropout rate before the completion of Grade 9 was a significant problem for Lebanon’s education system in the past three years. “Students who are 13 to 18 (…) are most prone to leaving school to work and support their families” (Donor, Interview) “Covering children’s personal needs is a burden on parents. Some students are skipping school to go to work. Boys in cycle 3 are working in restaurants at night and then coming to school the next morning.” (School principal, Interview, North) MEHE attributes the recent increase in dropout to “students’ need to work, long distances to travel to school, and the pressure of costs of education for textbooks, transportation, and uniforms.”31 UNICEF confirms that rising inflation and poverty is making education unaffordable and driving students to work instead of attending school.32 A MEHE official noted that dropout rates tend to be greater in secondary school. Similar to EQ2 findings, transportation cost was frequently cited by respondents as a major challenge to the attendance of both students and teachers. 28 World Food Program, “WFP Lebanon Country Brief – December 2022,” https://docs.wfp.org/api/documents/WFP￾0000145889/download/?_ga=2.202755667.356002948.1684509867-2065345087.1683296491 29 Afridi, Farzana. Barooah, Bidisha., Somanathan, Rohini. “Hunger and Performance in the Classroom.” Indian Statistical Institute and IZA SEPTEMBER 2019. 30 The Effects of Nutritional Interventions on the Cognitive Development of Preschool-Age Children: A Systematic Review - PMC (nih.gov) 31 Ministry of Education and Higher Education, Lebanon Five Year General Education Plan 2021-2025 (Beirut: Ministry of Education and Higher Education, 2021), 9 https://www.mehe.gov.lb/ar/SiteAssets/Lists/News/AllItems/5YP%20MEHE￾GE%20__amend1_%20Feb%202022.pdf. 32 UNICEF, Searching for Hope: A Grim Outlook for Youth as Lebanon Teeters on the Brink of Collapse (Beirut: UNICEF, 2022), 5, https://www.unicef.org/lebanon/media/7746/file/Searching%20for%20Hope%20Youth%20report%20EN%20pdf.pdf. 23 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 • Low quality school. Lagging school quality is an issue that preceded the current crises. “Even pre￾crisis, pre-pandemic, we were seeing very low levels of foundational learning,” as one donor representative put it. According to the MEHE General Education Plan, “the period following the 2019 crises compounded a crisis in effective learning time, which was estimated at only 6.3 years of learning within the 10.2 years of schooling that an 18-year-old should experience in Lebanon.”33 Poor school quality also leads parents of some means to send their children to private schools, which creates an imbalance, according to implementing partners and donors. In addition to schools lacking resources and teachers being demotivated, one of the most commonly cited concerns was the outdated nature of the national curriculum. “The state of the educational system is really bad in terms of the relevance of the learning delivered to children (…) in the public sector (…) there is poor quality, and it is not leading anywhere. Many kids drop out after grade 9.” (Donor, Interview) • Curriculum. MEHE, CRDP, donor, implementing partner, and teacher respondents noted the urgent need to develop the new curriculum. The current curriculum was developed in 1997 and, according to the General Education Plan, “falls short in terms of modern pedagogical approaches and appropriate assessment bylaws for development.”34 Evaluation respondents noted that it lacked relevance and was not at the right level of difficulty for students. They noted that the new curriculum should integrate 21st century skills and SEL and reflect an inclusive approach to education. The excerpts below illustrate these points among donors and teachers: “The curriculum needs massive investment in terms of making sure that all the books are relevant and at the right level for kids and that the teachers understand the curriculum.” (Donor, Interview) “(…) making sure that the system is adjusting to meet children where they are, that the curriculum is designed and implemented so that it is feasible and recognizes the current state of how things are and not just trying to go back to business as usual, as quickly as possible.” (Donor, Interview) According to MEHE respondents, the Curricula Higher Committee was established in March 2021, and CRDP has since developed a curriculum workplan that the Minister of Education endorsed. The curriculum reform for Grades K-12 will include 21st century competencies, assessment frameworks and integration of SEL.35 At the time of data collection, working groups were active, and the curriculum reform seemed to be underway. MEHE respondents also noted that the public lower secondary schools (Grades 7-9) were not supported by any donor programs and that the shifts in instructional approaches introduced by QITABI-2 in grades 1-6 were not reflected in the lower secondary level, which resulted in considerable differences in instructional approaches between cycles 1 and 2. • Inconsistent quality of school principals. Another factor impacting school quality is the inconsistent skills and capacities of school principals. MEHE respondents noted that many principals were in need of capacity strengthening, adding that the skills and motivation of school principals varied considerably and that the rigorous criteria were not being properly observed during the selection of principals. They noted that in the past, principals had been required to go through one year of specialized university training before being selected based on set criteria. This training and selection process are no longer followed, and appointments are often made based on political affiliations. MEHE respondents argued that principals should be trained in school leadership and 33 Ministry of Education and Higher Education, Lebanon Five-Year General Education Plan 2021-2025, 9. 34 Ministry of Education and Higher Education, Lebanon Five-Year General Education Plan 2021-2025, 10. 35 Ministry of Education and Higher Education, Lebanon Five-Year General Education Plan 2021-2025, 11. 24 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 planning, and that only those who showed promise and motivation should be considered. IP staff shared similar reflections noting that: “Many principals require some training. Instructional leadership is needed, and we need to improve their capacity to run their schools (…) in relation to literacy and numeracy and how to set goals, how to hold their teachers accountable (…), and how to evaluate their own school performance rather than just at the student level or the teacher level. The project could have done better in terms of preparing principals to respond to such a huge difference in instruction by teachers (…) But none of the projects addressed this directly.” (IP, Interview) • Other issues identified by respondents. Donors and IPs noted that early childhood education, technical and vocational education training (TVET), and non-formal education were also priority areas for the education sector. They highlighted the lack of quality and relevance in TVET programming, noting that the TVET curriculum had not been updated since the 1970s.36 It should be noted though that these issues tended to be raised by individuals already focusing on these education levels. “We can’t ignore the transition to secondary and TVET. This is really important, in particular for the most marginalized children. I would say that the issue of out-of-school children is really urgent and needs attention” (Donor, Interview) Some respondents, namely a CRDP official and two IP staff, also noted concerns about the safety of building infrastructure, citing the recent collapse of a school in Tripoli that killed two students on November 2nd, 2022. They mentioned a need to assess the safety of school buildings. • Increasing efficient use of limited funding. As noted before, the financial crisis has severely impacted government revenue, and the government lacks the resources needed to independently finance the education system. One respondent noted that it would be important to strengthen the education monitoring systems so that MEHE could more easily track the number of teachers by contract type, number of hours they are contracted to work, and number of hours they work. Though this will not address the funding shortfall, it could help the ministry use its limited funding more efficiently. RESEARCH QUESTION 2 RQ2: Where is USAID best placed to make an impact considering where the government of Lebanon and stakeholders are already putting their resources? What do both key informant interviews and related emerging data say about potential future interventions related to but not part of the existing QITABI-2 activity? In answering this question, the evaluation team focused on participants with a broad perspective of the education sector in Lebanon, especially donors intervening in education, as well as those familiar with USAID’s current portfolio. Findings echo and extend many of the results reached under EQ2 and RQ1. • Continue to support primary cycles I and II. When asked about future USAID activities, MEHE, CRDP, and donor respondents advocated for the need to continue and/or expand USAID’s current education programming in Lebanon. A MEHE official stated simply but firmly, “Stay with us.” (Interview). Although stakeholders noted that the LRP materials and approaches were well regarded, the QITABI-2 baseline assessment indicates that students were still performing below grade level and teachers were applying these approaches inconsistently. More support is therefore needed to 36 Implementing partner interview 25 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ensure the full integration and a uniform and routine adoption of these materials and teaching approaches to help students catch up. The importance of the integration of SEL was highlighted by most respondents. As the 30-school SEL pilot was delayed until the current school year (2022- 2023), more support will be needed to evaluate and then roll out the SEL program. “The foundational learning is what needs to be prioritized in the current context. Getting basic numeracy and literacy skills is important, and a lot of kids are missing out.” (Donor, Interview)” “I think USAID’s focus should remain on basic education, but I would include early childhood education. I think we have only just started this agenda around foundational learning because it is so important. We underestimate the amount of lost education in Lebanon in the past few years, so I think in the coming three to five years, we really need to push (…) USAID’s agenda (…) it is really relevant.” (Donor, Interview) • Continue to support Summer School Program. Given the success of the 2021-2022 summer school, respondents said USAID should continue to support summer school programming and explore ways to make the summer school available for expanding numbers of students. “The summer school was good for tackling the learning loss.” (Donor interview) • Support Cycle III. MEHE requested support in expanding the LRP programming to grades 7-9. Both an IP staff and a USAID staff suggested emphasizing soft skills and positive youth development, for instance. As noted under RQ1, MEHE feels that the lower-secondary schools are in need of support. Cycle III is seen a critical transition phase that is a determinant of students’ continuation to upper secondary and tertiary education. Yet, Cycle III is not currently supported by donor funding. Except for World Bank’s Emergency Social Safety Net (ESSN) project, which (among other things) provides education cash transfers to cover school fees and recurrent expenses, such as transport and books37 for 13-18-year-old students from “extreme poor Lebanese households at risk of dropping out.”38 “Everybody is supporting primary and basic education. We also need to work on the secondary level.” (MEHE Official, Interview) • Support the new curriculum. Building on QITABI-2’s support of the new curriculum framework, MEHE requested QITABI-2’s support in the development of the new curriculum. • Provide capacity strengthening to principals. Given the pivotally important role that principals play in a school’s performance and the importance of teachers’ adoption of new teaching approaches and observations, as noted in the RQ1 section above, respondents recommended the inclusion of principals in future QITABI-2 training programs. They also suggested the introduction of a principal performance evaluation and of greater principal accountability systems. • Support the operationalization of inclusive education. Though Lebanon has an inclusive education policy, the inclusive schools and resource centers for students with disabilities lack the trained staff, resources, and infrastructure needed to effectively integrate students with disabilities in their classrooms. Lebanon has eight resource centers for students with disabilities. As of January 2023, CRDP noted that these were all closed, as there was no funding to pay their staff. MEHE and CRDP noted that support was needed to operationalize Lebanon’s inclusive education. 37 Key Informant interview with World Bank representative. 38 World Bank. “The World Bank Lebanon Emergency Crisis And Covid-19 Response Social Safety Net Project (P173367). 27-Oct-2020 | Report No: PIDISDSA30792. World Bank Document 26 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 “MEHE doesn’t have a well-structured unit and trained teachers. There is a policy on inclusion, but teachers are not well equipped.” (MEHE, interview) “We are currently not paying the staff [at our center for students with learning difficulties] so the center closed because we don’t have money to pay experts to work with students with difficulties from public schools or even private ones. So, parents with no money who have children with difficulties don’t know what to do, and now we have suspended work at the center because we cannot pay the staff. This is too bad.” (CRDP, Interview) • Explore sustainable school feeding programs. MEHE identified school feeding as “one of a few activities for bolstering the access, participation, and completion of students from vulnerable groups, and as a key component of Pillar 1 in its General Education Plan.”39 Respondents highlighted the urgent and increasing need to establish school feeding programs to provide students with meals at school. Evidence from low- and middle-income countries indicates that school feeding programs strengthen enrollment and learning outcomes, but there is mixed evidence for attendance.40 Home Grown School Feeding (HGSF) programs intentionally procure a portion of their food from small￾holder farmers so that the school feeding program benefits local markets, locally sourcing familiar and nutritious food for students.41 One modality of HGSF is farm-to-school in which local farmers or communities directly grow the food for schools. School gardens are part of this modality and encouraged by WFP and school feeding activities supported by the U.S. Department of Agriculture (USDA)’s McGovern Dole Food for Education Program, among others. CONCLUSIONS The findings above lead to a number of conclusions, which the evaluation organized here according to thematic criteria. RELEVANCE  QITABI-2 programming aligns closely with the MEHE General Education Plan and the USAID CDCS. These two documents are complementary, with the CDCS results framework and the MEHE plan articulating pillars and results framework being nearly identical. QITABI-2 programming demonstrates relevance to the needs of beneficiaries through its alignment with core frameworks (the MEHE General Education Plan and the USAID CDCS), focus on government priority areas, and continuation of elements deemed relevant during the previous activity.  Six priority challenges emerge regarding the primary education sector in Lebanon. They are: learning loss; teacher salary reductions; SEL needs of students and teachers; lacking transportation; food insecurity’ and lacking resources at school: electricity, heating, and stationery.  QITABI-2 successfully pivoted its programming to respond to the crises and the changing needs of direct and indirect beneficiaries, demonstrating collaborative partnership and responsiveness, great cultural awareness, and adaptive management. 39 Ministry of Education and Higher Education, Lebanon Five Year General Education Plan 2021-2025, 50. 40 Adetoun Dapo-Famodu, School Feeding Programmes in Low- and Middle-Income Countries: A Focused Review of Recent Evidence from Impact Evaluations (Rome: World Food Programme, 2021), 18, https://www.wfp.org/publications/school￾feeding-programmes-low-and-lower-middle-income-countries 41 Food and Agriculture Organization and World Food Programme, Home-Grown School Feeding Resource Framework Synopsis (Rome: Food and Agriculture Organization and World Food Programme, 2018), 2, https://www.wfp.org/publications/home￾grown-school-feeding-resource-framework. 27 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023  Specific examples of adaptation are QITABI-2’s development of the LRP, major contribution to the summer school programs, digitization of materials, distribution of food parcels and educational boxes, integration of SEL materials and approaches in the LRP, and integration of teacher wellbeing in the teacher training sessions. EFFECTIVENESS  QITABI-2 has successfully achieved the majority of its intended outputs. Delays due to COVID, teacher strikes, and other crises have limited QITABI-2’s impact and made measurement of student performance impossible during the drafting of this report. Qualitative data from school￾level actors, in particular, indicate some achievements, such as the adaptation of more active learning approaches appreciated by students and teachers alike, the adoption of a more robust approach to integrating SEL, and taking important steps to improve service delivery.  The LRP successfully introduced new teaching practices, including the use of diagnostic assessment and differentiated learning and the integration of active learning and SEL in the classroom. The full adoption of these approaches and the anticipated impact on student performance remains to be seen, as the on-going crises, the extended school closures, and the very limited instructional time have greatly hindered the application of this program.  Several factors contribute to the effectiveness of QITABI-2, including aligning with the needs of MEHE, CRDP, DOPS, teacher trainers, and UNICEF and co-designing activities with them; integrating within existing systems; having a technically strong, well-organized, and communicative implementing partner consortium, led by World Learning; applying a research-based design; and evaluating interventions.  Several factors hinder QITABI-2’s effectiveness, including the financial crisis and the government’s unreliable funding for education, the prolonged school closures, the high transportation costs, and the limited time to apply lessons learned and fully integrate the LRP into standard teaching practice. The lack of training for principals and the need for more extensive and on-going teacher training and coaching were also noted as factors hindering the adoption of LRP approaches. SUSTAINABILITY  By working collaboratively with MEHE and CRDP in the design of the LRP and in integrating QITABI-2’s interventions and support within existing systems, QITABI-2 has worked to maximize the sustainability of its programming.  The public education sector cannot function effectively without substantial foreign aid.  The current economic and financial state of affairs severely impacts the sustainability of QITABI￾2’s activities and threatens to erase the gains achieved by the project and other donor-funded projects. RESEARCH QUESTIONS  Overall sector priorities: RQ1. In addition to the profound financial and resource limitations faced by MEHE, priority needs identified by respondents include addressing student nutrition needs, keeping schools open and addressing barriers to school access, addressing school quality via the development of the new curriculum, and strengthening the capacity of principals. Additional issues flagged include a need for strengthening early childhood education and TVET and developing programming for out-of-school youth.  Suggested future focus of USAID: RQ2. USAID occupies a unique space among education actors in Lebanon in its focus on improving student performance in reading, math, and SEL. USAID should continue to support cycles I and 2, including Summer School programming and support the development of the new curriculum. Cycle III is seen as a critical transition phase, but 28 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 it is not yet currently supported by donor funding. There is also an urgent need to establish school feeding programs. RECOMMENDATIONS The recommendations below emerge from the findings and conclusions and reflect consultations between the evaluation team and USAID. Validation sessions with partners may elicit additional recommendations and/or revisions that respond to the perspectives and considerations of a broader stakeholder group.  USAID should stay the course in its support of improving the first and second cycles of education to further support the use of materials and approaches developed under QITABI-2 and enhance student performance and system delivery. In addition, in response to need, USAID should explore working with MEHE and other donors to integrate QITABI-2’s approaches into the third cycle to ensure that students will continue to benefit from these approaches at the secondary level.  QITABI-2 should provide technical support to MEHE and CRDP in the development of new curriculum.  As some principals and teachers expressed concern that the TLMs were too easy, QITABI-2 and MEHE/CRDP should determine if the perception among some educators that LR materials were too easy implies the need for additional teacher training or if the materials may need to be adjusted. Teacher training might seek to ensure that teachers are clear in the purpose, know when and how to target students, and how to differentiate instruction for more advanced learners. .  Given MEHE will continue to require substantial foreign aid, QITABI-2 should work with MEHE to identify alternate approaches to fund and staff teacher coaching and mentoring. These could include providing additional targeted DOPS visits to schools that are most in need, training school principals or resource teachers at the school to act as teacher mentors, and organizing WhatsApp group calls and chats with teachers and a teacher mentor to discuss teaching challenging and strategies.  Similarly, given the limited human resources at CRDP, explore alternate, less labor-intensive national standardized testing approaches, such as written and group-administered tests that could be more sustainably conducted.  Given the continuing threat of strikes and schools closures and the challenges faced by students and teachers in accessing e-learning materials, USAID should work with QITABI-2, MEHE, CRDP, and UNICEF to explore ways to strengthen digital literacy and increase access to and use of remote learning materials.  QITABI-2 should continue discussions with the Education Donor Working Group and MEHE/CRDP in order to alleviate teacher salary problems and avoid future teacher strikes and school closures.  In addition to continuing the close collaboration with MEHE and CRDP, USAID should continue to coordinate with UNICEF and the World Bank to ensure the alignment of donors’ different learning recovery efforts in Lebanon.  Considering the severity of the food insecurity issue and the challenges faced with the distribution of food parcels, USAID should research alternate and sustainable school feeding options to ensure the proper nutrition for students to help them perform better at school and encourage them to remain in school. For sustainability, USAID should work closely with communities and family volunteers to establish school kitchens, strengthen local food chains, and 29 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 explore the feasibility of integrating school garden programming. Potential funders include WFP and USDA. Danish NGO “Zaher – Grow to Learn” and Lebanese NGO “SOILS,” which works with Syrian children and youth in the Beqaa, may offer inspiration. Other school feeding programs, such as numerous WFP/World Bank initiatives and the Slow Food school garden programs in Africa are also worth looking at. School gardens also offer additional benefits for socio-emotional health and can provide important learning opportunities regarding agricultural practice, biology, and the environment.  CRDP’s resource centers could play an important role in supporting students with disabilities and in making schools more inclusive. The centers are now closed due to lack of resources. USAID should consider supporting CRDP in identifying solutions to make these resource centers operational again.  USAID should leverage QITABI-2’s example of partnership, project delivery, and management within a crisis setting as a best practice. 30 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ANNEXES Annex I : Evaluation Timeline Annex II : Evaluation Scope of Work Annex III : Evaluation Methods and Limitations Annex IV : List of Documents Reviewed Annex V : Comprehensive Indicator Analysis 31 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ANNEX I: EVALUATION TIMELINE 32 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ANNEX II: EVALUATION SCOPE OF WORK 33 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ANNEX III: EVALUATION METHODS AND LIMITATIONS Evaluation Design The evaluation team conducted a two-phase process that was largely qualitative and included document review and both remote and in-person data collection. The design matrix (see Exhibit 8) maps how data sources responded to evaluation and research questions. Exhibit 8: Evaluation Design Matrix Question Questions, Sub-questions Document review Data Collection Methods and Sources Rel - EQ1 How relevant was the activity to the needs of the direct and indirect beneficiaries? Cooperative agreement, case studies, quarterly and annual reports, project workplans, project MEL plans, QITABI1 mid-term evaluation, QITABI2 baseline evaluation, UNICEF Deprived Childhood research, USAID Gender assessment, USAID Arabic Literacy and Numeracy State of the Art Conference Desk Review KIIs: USAID Lebanon, IPs, Ministerial partners (Central level, REOs, DOPS), instructional coaches, and school principals. Focus groups: teachers, students, and caregivers. Rel - EQ1a What were the primary education needs reflected in QITABI’s TOC? How closely do these identified needs align with the pre-crises needs expressed by beneficiaries? Do beneficiaries feel that the QITABI-2 activity design effectively addressed these pre-crises priority needs? Rel – EQ2 In light of the crises, what do primary and secondary beneficiaries (teachers, CRDP staff, MEHE staff, students, caregivers) and other stakeholders now see as the priority needs for primary education? Quarterly and annual reports, project workplans, QITABI2 baseline evaluation, UNICEF Deprived Childhood research, USAID Arabic Literacy and Numeracy State of the Art Conference Desk Review KIIs: USAID Lebanon, IPs, Ministerial partners (Central level, REOs, DOPS), instructional coaches, and school principals. Focus groups: Teachers, students, and caregivers. Eff – EQ3 To what extent has the Activity achieved its intended outputs and outcomes in terms of improving student performance in reading, math, and writing; instilling students with SEL skills; and improving the national-level service delivery of education? QITABI-2 1 baseline EGRA, QITABI-2 1 endline EGRA, 2021 Learning Recovery report, QITABI-2 2 2022 Baseline evaluation, Cooperative agreement, indicator tracking table, MEL plans KIIs: USAID Lebanon, IPs, Ministerial partners (Central level, REOs, DOPS), instructional coaches, and school principals. Focus groups: Teachers, students, and caregivers. Eff – EQ4 What were the factors that contributed to or hindered the a) adoption of education interventions promoted by QITABI? and b) the achievement of the intended results? Which of these factors are independent from the crises experienced in Lebanon since 2019? QITABI-2 1 baseline EGRA, QITABI-2 1 endline EGRA, 2021 Learning Recovery report, QITABI-2 2 2022 Baseline evaluation, Cooperative agreement, case studies, quarterly and annual reports, project workplans, QITABI1 final evaluation, USAID Arabic Literacy and KIIs: USAID Lebanon, IPs, Ministerial partners (Central level, REOs, DOPS), instructional coaches, and school principals. Focus groups: Teachers, students, and caregivers. 34 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Question Questions, Sub-questions Document review Data Collection Methods and Sources Numeracy State of the Art Conference Desk Review Eff – EQ5 To what extent has QITABI-2’s implementation adapted to the changing priority needs that have emerged since the original project design? Were these adaptations successfully applied? Once applied, were these adaptations effective in addressing the stated need? Cooperative agreement, case studies, quarterly and annual reports, project workplans, QITABI2 baseline evaluation, Learning Recovery report KIIs: USAID Lebanon, IPs, Ministerial partners (Central level, REOs, DOPS), instructional coaches, and school principals. Focus groups: Teachers, students, and caregivers. Sust – EQ6 At this mid-term point, what signs emerge that project activities and results may continue after the project ends? What factors may hinder the sustainability of interventions? 2021 Learning Recovery report, QITABI-2 2 2022 Baseline evaluation, Cooperative agreement, case studies, quarterly and annual reports, project workplans KIIs: USAID Lebanon, IPs, Ministerial partners (Central level, REOs, DOPS), instructional coaches, and school principals. Focus groups: Teachers, students, and caregivers. RQ1 Looking broadly at the basic education sector (pre-primary, primary, secondary, and technical vocational education, and higher education) what do stakeholders with a global understanding of the sector feel are priority challenges? Within particular education levels? At a sector-wide or systemic level? How do these perceptions align with the most recent USAID Lebanon CDCS and MEHE education sector strategic plan? USAID Lebanon CDCS; Lebanon National Education Plan; QITABI-2 MEL plans, UNICEF Deprived Childhood research, USAID Gender assessment, USAID Arabic Literacy and Numeracy State of the Art Conference Desk Review KIIs: USAID Lebanon, select IPs, Ministerial partners (Central level), and donors. Findings workshop: USAID Lebanon, select IPs, Ministerial partners (Central level), and donors. RQ2 Where is USAID best placed to make an impact considering where the government is already putting their resources and what do both key informant interviews and related emerging data say about potential future interventions related to but not part of the existing QITABI-2 activity? USAID Lebanon CDCS; Lebanon National Education Plan; UNICEF Deprived Childhood research, USAID Gender assessment, USAID Arabic Literacy and Numeracy State of the Art Conference Desk Review KIIs: USAID Lebanon, select IPs, Ministerial partners (Central level), and donors. Findings workshop: USAID Lebanon, select IPs, Ministerial partners (Central level), and donors. The data collection started with collecting secondary data for the document review and secondary quantitative data. Primary qualitative data collection occurred in two phases: Phase 1 was remote data collection with QITABI-2 and USAID staff, while Phase 2 was in-person data collection for school-level stakeholders, government partners and other donors. 35 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Note that the evaluation team changed the order of the evaluation questions during the writing phase in order to ensure a fluid flow of argumentation within the report. Exhibit 9 matches the number of the inception report question with the number of the question in the final report. Exhibit 9: Question Legend Draft Report # Inception Report # EQ1 EQ1 EQ2 EQ2A EQ3A EQ5 EQ3B EQ2B EQ4 EQ3/EQ5B EQ5 EQ 4 EQ6 EQ6 RQ1 RQ1 RQ2 RQ2 Document Review The document review of 52 documents included QITABI 2 program documents, existing QITABI 2 assessments, evaluations of previous USAID education activities in Lebanon (i.e., QITABI 1 and D￾RASATI 2), and literature on Lebanon’s education sector (see Exhibit 10 and Annex IV for detailed list). Document review started in October 2022 and ran iteratively throughout the entire data collection period until March 2023. The document review mostly preceded the implementation of primary data collection to inform the design as well as throughout to aid in analysis. Exhibit 10: Detailed Document Review List Document Type Documents of Interest QITABI 2 Program Documents QITABI-2 award and modifications, annual workplans, monitoring, MEL plans, and quarterly reports. QITABI 2 Assessments Baseline report, Learning Recovery Program evaluation, education box assessment, qualitative study to develop the national SEL framework, Rapid Education and Risk Analysis (RERA) study, Human and Institutional Capacity Development (HICD) analysis documentation, GESI Analysis (using World Learning’s Transforming Agency, Access, and Power (TAAP)1 tool), and analyses of classroom observations. Evaluations of QITABI 1 and D-RASATI 2 Midline evaluation of QITABI 1 and D-RASATI 2 evaluation Literature Review of Education USAID/Lebanon Country Development Cooperation Strategies (CDCS) 2014-2018 and 2022-2026, Arabic Literacy and Numeracy State of The Art Conference Desk Review (USAID, Middle East Education, Research, 36 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Document Type Documents of Interest Training, and Support, 2019), Political Economy of Education in Lebanon - Research for Results Program (World Bank 2020), Foundations for building forward better An Education Reform Path for Lebanon (World Bank 2021), Public Finance Review - Ponzi Finance? – Education Chapter (World Bank, 2022), Multidimensional Child Poverty: A Qualitative Overview of Lebanon (UNICEF, 2022), 2018 USAID Lebanon gender assessment (education chapter), 2018 PISA report for Lebanon, and further literature to fill gaps not discussed by these materials and not attainable through primary data collection. Qualitative Data Collection The data collection started in December 2022 and concluded in March 2023. The first phase of qualitative data collection began in in December 2022 with QITABI-2 staff and continued with USAID staff in January 2023. The second phase of data collection began in January 2023 with in￾person including KIIs with government stakeholders from the Ministry of Education and Higher Education (MEHE), the Center for Educational Research and Development (CRDP), and donor representatives such as the Kreditanstalt für Wiederaufbau (KfW), UNICEF, and World Bank. FGDs with regional and school-level stakeholders (i.e., REOs, teachers, principals, caregivers and students) was postponed until March given teacher and ministry strikes and school closures. Data collection was completed in March 2023. Remote data collected employed purposeful sampling to target individuals who played critical roles in the development and implementation of the QITABI-2 activity. The evaluation team interviewed two key individuals from USAID and 13 QITABI-2 staff (six from World Learning, three from International Rescue Committee, one from Chemonics, one from Amideast, one from Management Systems International, and one from Ana Aqra Association). The identification of interviewees reflected inputs from USAID and QITABI-2 staff. The number of interviewees per organization corresponded with the degree of involvement in the activity. In-person data collection built on Phase I to collect perspectives from stakeholders involved in QITABI-2 activities as well as government and donors. Sampling of Schools Sampling of schools followed a purposive approach that included schools with characteristics important to QITABI-2 activities (e.g., governorate, location, secondary language of instruction (LOI), shift, LF intensive visit, and inclusive school) as well as the guiding inquiry questions. The evaluation team selected 16 QITABI-2 public first and second shift schools from a list stratified by governorate, school shift, and secondary LOI. Principals from selected schools were asked if their school would participate in the evaluation all but one of the selected schools participated. Among the 15 schools, there were two in each governorate (except Mount Lebanon which had one school). Exhibit 11 shows the distribution of the schools across Lebanon. 37 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Exhibit 11: School-Level Data Collection by Governorate Exhibit 12: School Sample Achievement of Sample Targets Governorate First Shift Second Shift Total Target Achieved Target Achieved Target Achieved Akkar 1 1 1 1 2 2 Baalbeck-Hermel 1 1 1 1 2 2 Beirut 1 0 1 2 2 2 Bekaa 1 0 1 2 2 2 Mount Lebanon 1 0 1 1 2 1 North Lebanon 1 0 1 2 2 2 Nabatiyeh 1 2 1 0 2 2 South Lebanon 1 0 1 2 2 2 Total 8 5 8 11 16 15 Sampling of Principals, Teachers, Caregivers, and Students The evaluation team conducted KIIs with school principals, and FGDs with teachers, caregivers, and students. The evaluation team selected teachers, caregivers, and students for participation in the evaluation with the assistance of the school principal in each school. There were 15 principal KIIs, 15 teacher FGDs (73 teachers – 69 women and 4 men), and seven caregiver FGDs (31 caregivers – 24 women and 7 men), and eight student FGDs covering Grade 3 (27 students – 12 girls and 15 boys) and Grade 6 (20 students – 12 girls and 8 boys). The division of students allowed us to tailor our 38 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 process and student interaction to their grade and age group as well as to capture their different impressions and challenges. Exhibit 13: Principal, Teacher, Caregiver, and Student Samples Achievement of Sample Targets Governorate Principal KIIs Teacher FGDs Caregiver FGDs Student FGDs Target Achieved Target Achieved Target Achieved Target Achieved Akkar 2 2 2 2 1 1 1 1 Baalbeck-Hermel 2 2 2 2 1 1 1 1 Beirut 2 2 2 2 1 1 1 1 Bekaa 2 2 2 2 1 1 1 1 Mount Lebanon 2 1 2 1 1 0 1 1 North Lebanon 2 2 2 2 1 1 1 1 Nabatiyeh 2 2 2 2 1 1 1 1 South Lebanon 2 2 2 2 1 1 1 1 Total 16 15 16 15 8 7 8 8 Note. FGDs with Grade 3 students in Akkar, Beirut, Mount Lebanon, North Lebanon, and South Lebanon. FGDs with Grade 6 students in Baalbeck-Hermel, Bekaa, and Nabatiyeh Key Informant Interviews The evaluation team conducted 15 KIIs with principals to obtain information about QITABI-2 activities and outcomes to inform the evaluation. Interviews were carried out using a semi￾structured interview guide. Principals were asked about the benefits of QITABI-2 programming, the extent of the adoption of QITABI-introduced systems, approaches, and materials, training initiatives, pivots in QITABI-2 programming, and the extent to which they feel learning recovery shifts helped to alleviate crises related issues at the school. KII questions are included in Annex 4. Focus Group Discussions The research team alternated between caregiver and student focus groups such that each of the selected schools would have either a caregiver FGD or a student FGD. FGDs with caregivers inquired about caregiver experiences with their children’s schooling and the challenges they faced. Caregivers were invited to FGDs conducted at the beginning of the school day when caregivers were likely to travel to the school for student drop-off. The student FGD questions explored students’ daily experiences at school (e.g., preferences, level of satisfaction, challenges, and transition back to school post-reopening) and on the way to school. Grade 3 and Grade 6 students were identified to participate as the grades align with those that participated in baseline and learning recovery studies. The evaluation team obtained caregiver consent for each student to participate in FGDs as well as students’ assent to participate. Sets of questions for each of those group discussions are included in Annex 4. Government The evaluation team purposively selected government counterparts: 9 from MEHE and 5 from CRDP. The MEHE interviewees included three staff from MEHE headquarters and six Regional Education Officers. The MEHE and CRDP counterparts selected were the most involved in QITABI￾2 activities to participate in KIIs. Selection was based on those who are best placed to respond to evaluation and research questions. Input from USAID and QITABI-2 staff informed the list. Donors Discussions with donors focused on the research questions and their understanding of the education sector in Lebanon, its needs, challenges, and opportunities. Selection of individuals for KIIs was purposive based on those who may have the experience and visibility necessary on the education 39 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 sector to allow them to most comprehensively participate. The evaluation team selected and interviewed one official each from KfW, UNICEF, and World Bank. Data Collection Procedures Data collection during Phase 1 was remote, using secure platforms such as Microsoft Teams, Google Meets, or Zoom, per the preferences of the interviewees. If the interviewee consented to recording, it happened through the interview platform. The evaluation team asked the interviewee to identify a place where they would have privacy and a strong internet connection in order to participate in the interview. At the start of data collection, the interviewer told the participant that if they thought the conversation was no longer private and confidential because someone had approached them or for any reason to alert the interviewer using the chat function in order to change the topic of conversation. During all remote interviews, two members of the evaluation team participated. All team staff were seasoned interviewers who had experience in maintaining the privacy and confidentiality of interviews. They confirmed procedures together in order to ensure they kept participants and their data safe. During the Phase 2 in-person data collection process, the team leader and the local education specialist led data collection activities with the team leader focusing on interviews with government officials and donors, while the local education specialist focused on school-based data collection. The senior M&E specialist and the data analyst supported and, in all cases, focus groups required both the local education specialist and at least one other team member as notetaker. School-level data collection instruments were tested in a school in Beirut to ensure that the instruments and protocols functioned well. Any challenges discovered in this application were reflected in revised instruments and data collection protocols. Ethics and Security Protocols The evaluation team employed a multi-pronged strategy and high ethical standards for this evaluation to ensure no individuals experienced additional risk or undue harm as a result of their involvement in the data collection processes. APPROVAL FROM IRB. The evaluation team submitted the evaluation design to EnCompass’s internal IRB to ensure the design met strong methodological and ethical standards. The evaluation team submitted a complete methodology, data collection guide, data collection instruments, security protocols, and other relevant documents to the IRB committee for review. Data collection instruments and protocols were submitted for approval by the EnCompass IRB only. There is no national IRB in Lebanon; accredited private institutions’ IRBs in Lebanon are dedicated for clinical research and trials. The US based evaluation team conferred with the Lebanon￾based team and conducted an internet search to confirm this understanding. INFORMED CONSENT. The evaluation team followed a rigorous informed consent procedure that included consent forms that were concise and easy to understand. RISKS, BENEFITS, AND COMPENSATION. The chief risk was a breach in confidentiality or privacy. The evaluation team used best practices in data management and security to reduce the risk of a data breach. While there was no anticipated direct benefit from the program, the evaluation team hoped that the evaluation would improve QITABI-2 programming. Given the low risk to participants and the opportunity for strengthening the Lebanese education system, the study provided important findings that outweighed the risk. Participants were not compensated to participate in the interviews. DATA SECURITY. There was a risk of a data breach that would reveal the identities of participants and connect them clearly with interviews notes, transcripts, and recordings. All virtual data collection activities prioritized the protection of all participants in the evaluation. The following approaches mitigated risks:  The handling of sensitive programmatic information was limited to evaluation team staff.  The evaluation team used an encrypted folder on the EnCompass server to store all data. 40 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 After each primary data collection event (interview, focus group, etc.), a team member transferred the audio recording and notes to the secure folder for data storage.  Personal information was not included during data collection, except in a secure tracker. This tracker was the only place where the name of a participant was linked to their data’s unique code.  The transcript was stored in the same folder. All interviews were saved using a consistent naming protocol that did not identify the participant.  The data analysis platform, Excel, resided on the encrypted EnCompass server. Only individuals involved in analysis had access to this file. Unique codes were in a separate folder under password.  The evaluation team used secure communication channels as requested by the respondent.  The evaluation team will apply a detailed protocol for data retention and destruction, including destroying all data after three years of secured storage. Informed Consent Process The evaluation team followed an informed consent process for all data collection activities. For remote interviews, the evaluation team e-mailed scheduled interviewees the informed consent form for remote data collection. For in-person interviews, the evaluation team reached out to stakeholders via email or phone to schedule meetings. The evaluation team reviewed the process at the beginning of the discussion. The language of the consent form was tailored to the language of the participant. In the case of remote data collection participants, the form was provided in English as all remote participants were fluent in English. Interviews were conducted in English. The evaluation team conducted in-person interviews and focus groups in English and/or Arabic. In most cases, interviews with government officials and donors took place in English. The school-level interviews and FGDs and regional education official interviews were conducted in Arabic. Consent forms and data collection tools were translated into Arabic in advance of field work. The focus group discussions with students required both caregiver informed consent as well as assent from the students. This process allowed parents to decide whether or not they wanted their child to participate and for the child to also make a decision. For students, school principals or teachers informed caregivers and the evaluation team sought caregiver consent for participation in advance of the FGD. Parameters of Informed Consent For both remote and in-person data collection, the informed consent forms included comprehensive information about the evaluation and data use so the participant could make an informed decision about participating in the evaluation. In addition to describing the purpose, reason for selection, procedures and duration of data collection, potential risks and benefits, and any compensation provided, the informed consent includes language that clarified:  Information about the level of anonymity/confidentiality and protection the respondent could expect—specifically, comments were not attributed to a specific individual or organization; to the extent possible all identifying information was removed from any report;  That participation was entirely optional and interviewees could stop participating at any time;  That participation means they could refuse to respond to any question for any reason;  That the respondent had the option to refuse having the interview audio-recorded;  That the respondent could raise any questions or concerns about the study with the data collectors or the manager of the study at any time, during or after the interview; and  That their responses were carefully secured and remained confidential. The evaluation team asked the participant to review the informed consent form and let the team know if they had any questions. At the beginning of each KII or FGD, the interviewer reviewed key 41 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 points from the consent form and ensured that the respondents understood risks and benefits. After allowing time for the participant(s) to ask questions, the interviewer asked for consent (or assent, in the case of students). This consent was oral consent because this portion of the study presented no more than minimal risk of harm to subjects and involved no procedures that required written consent outside of the research context. During the consent process, the interviewer asked if the interviewee would agree to be recorded. In the case that the recording was not allowed, the interviewer made note that the interviewee did not consent to the recording and took written notes from the discussion. Answers to the consent/assent questions were documented as part of the interview notes/transcript. The evaluation team subsequently recorded responses within the data analysis database. Data Analysis, Interpretation, Synthesis, and Reporting DOCUMENT REVIEW. The evaluation team developed a desk review matrix to track documents collected and coded information from documents against evaluation and research questions. The team applied standardized data extraction guidelines to facilitate a systematic process for document review and analysis. Insights from document review informed the development of tools as well as a preliminary codebook for qualitative analysis. PRIMARY QUALITATIVE DATA COLLECTION. The team recorded interviews and focus groups when agreed to by participants. Detailed notes were taken during interviews and interviews were recorded to ensure accuracy and completeness of notes. Analysis was conducted on notes from data collection. Next, the evaluation team applied a combination of deductive and inductive coding approaches. An initial coding scheme was developed based on evaluation questions and topics identified during document review and developed within the data collection instruments. The team updated the codebook iteratively as the team identified topics outside the content of the instruments. Inter-coder reliability checks and regular, frequent meetings with the evaluation team members participating in the coding facilitated a shared understanding of the coding scheme. The evaluation team developed transcripts directly within an Excel database tailored to the data collection instruments. Once transcripts were completed and cleaned, the database served as a data analysis tool to code data based on the preliminary codebook. The database included a section for recording answers to informed consent questions, relevant quotes, internal notes, and iterative coding structures. INTERPRETATION AND SYNTHESIS. Following preliminary analysis, the evaluation team gathered for an internal data analysis, interpretation, and synthesis (DAIS) meeting to consolidate findings on qualitative data. To validate and refine these preliminary findings, the evaluation team held a virtual findings meeting with USAID. The research team presented preliminary findings in May 2023. REPORTING AND DISSEMINATION. Following the participatory findings discussion, the team developed the draft report for review by the USAID/Lebanon Education Office. A validation session with host country counterparts and QITABI-2 is planned as part of the feedback and revisions to the draft report. Limitations and Biases The evaluation team faced delays obtaining MEHE’s approval to collect school-level data and to public school strikes throughout Lebanon early in the data collection phase. KIIs were conducted by telephone and on Microsoft Teams, and planned meetings and site visits were rescheduled, some multiple times. FGDs were conducted remotely in response to principals’ concerns about either taking teachers out of the classroom to participate or asking teachers to come to school early or remain after school to participate. The rescheduled field work was completed four weeks later than planned. RECALL BIAS. Beneficiaries may respond with answers that blend their experiences into a composite memory. Those who participated in similar activities may not distinguish their QITABI experience. Additionally, perceptions of events may change over time, and the ability to remember 42 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 specific details may fade. This is particularly probable in this situation when FGDs were conducted shortly after schools re-opened in March. RESPONSE/SOCIAL DESIRABILITY BIAS. Response bias is the risk that key informants may provide responses that are socially desirable or influential in obtaining donor support. For example, respondents may believe that negative findings could reduce future assistance. At the beginning of each KII and FGD, interviewers clarified that the responses would not influence the possibility of future assistance. Similarly, consent scripts were used to inform KII and FGD respondents that their answers were confidential. SELECTION BIAS. Selection bias is an inherent risk when implementers help to facilitate contact with project beneficiaries. The evaluation team used information provided by QITABI to organize KIIs with school principals and obtained the teachers and parents contacts from the latter. There was a risk that the principals would favor the most active, responsive, or engaged beneficiaries— meaning that the evaluation team would hear from key informants who report positive experiences. LIMITED RESOURCES. The research questions complement the evaluation questions, but the effort is limited by time and resources. DATA COLLECTION INSTRUMENTS 43 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ANNEX IV: LIST OF DOCUMENTS REVIEWED 1. Abdul-Hamid, H., and Yassine, M. “International Development in Focus. Political Economy of Education in Lebanon: Research for Results Program.” World Bank Group, 2020. 2. Afridi, F., B. Barooah, and R. Somanathan. “Hunger and Performance in the Classroom.” Indian Statistical Institute and IZA September 2019. 3. Boyle, Helen. Arabic Literacy and Numeracy State of the Art Conference Desk Review” (2019). https://www.edu￾links.org/sites/default/files/media/file/SOTA%20Desk%20Review%20Report_Executive%20Su mmary_English.pdf 4. Central Administration of Statistics. “Education Statistics.” Presidency of the Council of Ministers – Lebanese Republic. 23 May 2023. http://www.cas.gov.lb/index.php/demographic￾and-social-en/education-en. 5. Curriculum Plan, Action Plan for Curriculum Development 6. Curriculum Plan: Workflow and Distribution of Tasks 7. Dapo-Famodu, A. School Feeding Programmes in Low- and Middle-Income Countries: A Focused Review of Recent Evidence from Impact Evaluations (Rome: World Food Programme, 2021), https://www.wfp.org/publications/school-feeding-programmes-low-and-lower-middle￾income-countries. 8. ECW Multi-Year Resilience Programme (MYRP). Lebanon, 2022-2024. 9. Education Donor Working Group. (2022). "Meeting Minutes: July 14, 2022." Minutes prepared by EnCompass LLC for USAID/Lebanon. Beirut: Lebanon, Encompass LLC, 2022. 10. Education Donor Working Group. (2022). "Meeting Minutes: October 06, 2022." Minutes prepared by EnCompass LLC for USAID/Lebanon. Beirut: Lebanon, Encompass LLC, 2022. 11. Education Donor Working Group. (2022). "Meeting Minutes: September 02, 2022." Minutes prepared by EnCompass LLC for USAID/Lebanon. Beirut: Lebanon, Encompass LLC, 2022. 12. El Ghali,Hana Addam; Bend,Mary; Fazili,Sheena; El Franji,Nadine Joseph; Lahire,Nathalie; Hosseini Matin,Anahita; Sedmik,Elisabeth. “Schools, Students, and Teachers in Lebanon : Research for Results Program 2021” (English). Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/099430012202120328/P1758140fff6250300b8c0 06a941a84da7c 13. Food and Agriculture Organization and World Food Programme, Home-Grown School Feeding Resource Framework Synopsis (Rome: Food and Agriculture Organization and World Food Programme, 2018), https://www.wfp.org/publications/home-grown-school-feeding-resource￾framework. 14. General Directorate of Education. (2022). “MEHE Back to School Plan 2022-2023” 44 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 15. Hunter, D., E. Monville-Oro, B. Burgos, C. Nyrhia Rogel, B. Calub, J. Gonsalves, and N. Lauridsen. Agrobiodiversity, School Gardens, and Healthy Diets: Promoting Biodiversity, Food, and Sustainable Nutrition. New York: Routledge, 2020. 16. Lebanon PISA 2018, Middle East & North Africa, World Bank Group. 17. Management Systems International. (2021). "QITABI 2 Learning Recovery Study Report." Prepared for the United States Agency for International Development. Beirut, Lebanon, MSI, 2021. 18. Ministry of Education and Higher Education. Lebanon Five Year General Education Plan 2021- 2025 (Beirut: Ministry of Education and Higher Education, 2021), https://www.mehe.gov.lb/ar/SiteAssets/Lists/News/AllItems/5YP%20MEHE￾GE%20__amend1_%20Feb%202022.pdf. 19. Reaidi R., O. Salha, and R. El Khoury. "QITABI 2 Literacy and Numeracy Baseline Report: Evaluation of Student Performance in Primary Public Schools in Lebanon." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: Management Systems International, 2022. 20. Roberts, M., T. Tolar-Peterson, A. Reynolds, C. Wall, N. Reeder, and G.R. Mendez. “The Effects of Nutritional Interventions on the Cognitive Development of Preschool-Age Children: A Systematic Review.” Nutrients 14(3): 532. 21. Simmons, K., J. Pendelton, D. Smith, A. Spangler, S. Smith-Lunsford, and N. Andraos. (2019). "USAID Lebanon Gender Assessment: Final Assessment Report." Prepared for the United States Agency for International Development. Social Impact Inc. and Encompass LLC, 2019. 22. Social Impact. “Arabic Literacy and Numeracy State of the Art Conference Desk Review: Final Report.” Prepared for the United States Agency for International Development, October, 2019. 23. Social Impact. “Developing Rehabilitation Assistance to Schools and Teachers Improvement (D-RASATI 2). Final Performance Evaluation Report.” Prepared for the United States Agency for International Development, October, 2016. 24. Social Impact. “Midterm Performance Evaluation: Quality Instruction Towards Access and Basic Education Improvement (QITABI) Final Evaluation Report.” Prepared for the United States Agency for International Development, September 2017. 25. UNICEF. "Deprived Childhoods: Child Poverty in crisis-wracked Lebanon." Prepared for the United Nations Children Fund. Beirut: Lebanon, UNICEF, 2022. 26. UNICEF. "Multidimensional Child Poverty: A qualitative overview of Lebanon." Prepared for the United Nations Children Fund. Beirut: Lebanon, UNICEF, 2022. 27. UNICEF, Searching for Hope: A Grim Outlook for Youth as Lebanon Teeters on the Brink of Collapse (Beirut: UNICEF, 2022). 45 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 28. USAID/Lebanon. Country Development Cooperation Strategy (CDCS) 2021-2026. (2021). https://www.usaid.gov/lebanon/cdcsWorld Bank. “Foundations for Building Forward Better: An Education Reform Path for Lebanon.” 2021 29. World Bank. “Lebanon Public Finance Review: Ponzi Finance?” July, 2022. 30. World Bank. “Project Information Document: World Bank Lebanon Emergency Crisis and COVID-19 Social Safety Net Project (P173367). 27 October 2020. https://documents1.worldbank.org/curated/en/216001603902307374/pdf/Project￾Information-Document-LEBANON-EMERGENCY-CRISIS-AND-COVID-19-RESPONSE￾SOCIAL-SAFETY-NET-PROJECT-P173367.pdf 31. World Bank and Ministry of Education and Higher Education. (2021). Teachers in Lebanon: Understanding the Quality of Teachers’ Instructional Practices and Performance: Research for Results Program 2021. 32. World Food Programme. “WFP Lebanon Country Brief – December 2022,” https://docs.wfp.org/api/documents/WFP￾0000145889/download/?_ga=2.202755667.356002948.1684509867- 2065345087.1683296491. 33. World Food Programme. “WFP Lebanon Situation Report - January 2023.” https://reliefweb.int/report/lebanon/wfp-lebanon-situation-report-january￾2023#:~:text=Based%20on%20WFP%E2%80%99s%20Household%20Deprivation%20Score% 2C%2060%20percent,2022%2C%20up%20from%2053%20percent%20in%20December%2020 21. 34. World Learning. "QITABI 2 - Quality Instruction Towards Access and Basic Education Improvement 2 - Monitoring, Evaluation and Learning Plan." Prepared for the United States Agency for International Development, September 2019. 35. World Learning. "QITABI 2 - Quality Instruction Towards Access and Basic Education Improvement 2 - Monitoring, Evaluation and Learning Plan." Prepared for the United States Agency for International Development, September 2020. 36. World Learning. "QITABI 2 - Quality Instruction Towards Access and Basic Education Improvement 2 - Monitoring, Evaluation and Learning Plan." Prepared for the United States Agency for International Development, November 2021. 37. World Learning. "QITABI 2 - Quality Instruction Towards Access and Basic Education Improvement 2 - Monitoring, Evaluation and Learning Plan." Prepared for the United States Agency for International Development, V1. October 28, 2020 - V2. January 12, 2021. 38. World Learning. "QITABI 2: First Quarterly Report: April - June 2019." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2019. 39. World Learning. "QITABI 2: Second Quarterly Report: July - September 2019." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2019. 46 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 40. World Learning. "QITABI 2: Third Quarterly Report: October - December 2019." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2019. 41. World Learning. "QITABI 2: Fourth Quarterly Report: January - March, 2020." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2020. 42. World Learning. "QITABI 2: Fifth Quarterly Report: April - June, 2020." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2020. 43. World Learning. "QITABI 2: Sixth Quarterly Report: July - September 2020." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2020. 44. World Learning. "QITABI 2: Seventh Quarterly Report: October - December, 2020." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2020. 45. World Learning. "QITABI 2: Eighth Quarterly Report: January - March, 2021." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2021. 46. World Learning. "QITABI 2: Ninth Quarterly Report: April - June, 2021." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2021. 47. World Learning. "QITABI 2: Tenth Quarterly Report: July - September, 2021." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2021. 48. World Learning. "QITABI 2: Eleventh Quarterly Report: October - December, 2021." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2021. 49. World Learning. "QITABI 2: Twelfth Quarterly Report: January - March, 2022." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2022. 50. World Learning. "QITABI 2: Thirteenth Quarterly Report: April - June 2022." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2022. 51. World Learning. "QITABI 2: Fourteenth Quarterly Report: July - September 2022." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2022. 47 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 52. World Learning. "QITABI 2: Fifteenth Quarterly Report: October - December 2022." Prepared for the United Stages Agency for International Development. Beirut, Lebanon: World Learning, 2022. 53. World Learning. "QITABI 2: Year 1 Work Plan: April 2019 - September 2020." Prepared for the United States Agency for International Development. Beirut, Lebanon: World Learning, 2019. 54. World Learning. "QITABI 2: Year 2 Work Plan: October 2020 - September 2021." Prepared for the United States Agency for International Development. Beirut, Lebanon: World Learning, 2020. 55. World Learning. "QITABI 2: Year 3 Work Plan: October 2021 - September 2022." Prepared for the United States Agency for International Development. Beirut, Lebanon: World Learning, 2021. 56. Youssef, S., N. Jones, A. Małachowska, and M. Saleh, Adolescents in the abyss of Lebanon’s worst economic crisis: a focus on Lebanese and Palestinian adolescents’ education, and voice and agency. London: Gender and Adolescence: Global Evidence July 2022. 48 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 ANNEX V: COMPREHENSIVE INDICATOR ANALYSIS This section provides a more detailed analysis of the QITABI 2 output and outcome indicators, complementing the response to EQ4 within the main body of the report. Exhibit 14: Progress towards Objective-1 indicators No. Indicator FY20 Actuals FY21 Actuals FY22 Targets FY22 Actuals (Up to Q4) % of FY22 target achieved LOP Target (Y5) KP1 Percent of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with USG assistance (Standard ES. 1-1) N/A N/A N/A N/A N/A 50% KP2 Percent of learners who demonstrate reading fluency and comprehension of grade level text at the end of primary school (or grade 6, whichever comes first) with USG assistance (Standard ES. 1-2) N/A N/A N/A N/A N/A 50% KPI 3 Number of public and private schools receiving USG assistance (ES. 1- 50)* 887 1,015 1,017 927 91.2% 1,337 KPI 4 Number of primary or secondary educators who complete professional development activities with USG assistance (Standard ES.1-6) N/A 1,763 7,505 13,550 180.5% 8,714 KPI 5 Number of learners in primary schools or equivalent non-school based settings reached with USG education assistance (ES.1-3) 148,202 159,315 211,391 169,902 80.4% 340,306 KPI 6 Number of primary or secondary textbooks and other teaching and learning materials (TLM) provided with USG assistance (Standard ES.1-10) 297,526 191,919 576,522 616,960 107.0% 1,110,197 49 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 No. Indicator FY20 Actuals FY21 Actuals FY22 Targets FY22 Actuals (Up to Q4) % of FY22 target achieved LOP Target (Y5) KP7 Number of education institutions/environments supported with preparedness, prevention and/or response interventions (pre-primary to higher education/formal or non￾formal) (Standard ES.1&ES.2 N/A N/A N/A N/A N/A 987 KP8 Percent of learners who attain a minimum grade￾level proficiency in reading at the end of grade 3 (end of cycle 1) with USG assistance N/A N/A N/A N/A N/A 50% KP9 Number of learning environments supported by USG assistance that have improved safety, according to locally￾defined criteria (ES. 1- 51) N/A N/A 887 883 99.5% 887 Ind￾1c Number of primary school educators who complete professional development activities on implementing evidence-based math instruction with USG assistance N/A 317 2,206 3,061 138.8% 2,206 Ind￾1d Number of schools receiving educational materials and equipment 887 1,012 1,017 914 89.9% 1,017 Ind￾1.1 Percentage of learners showing improvement of at least one reading level N/A N/A 80.0% 48.1% 60.1% 85.0% Ind￾1.2 Percentage of learners showing improvement in math N/A N/A 80.0% 61.0% 76.3% 85.0% Analysis of Objective-1 indicators based on FY2022 Q4 valuesi demonstrates that QITABI-2 exceeded, or was close to exceeding, nearly all of its output targets as well as one outcome target (KP9), the latter related to school safety (Exhibit 14). Two outcome indicators (Ind-1.1 and Ind-1.2) that report on improvements in reading and math respectively fall short from meeting targets. Overall, less than half of learners show improvement in at least one reading level though nearly two- 50 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 thirds (61.0 percent) did so for math. One output indicator and three outcome indicators could not be reported on: KPI 7 due to lacking a precise definition of the standard indicator while it is too soon in the project life to report on KP1 (planned for Year 4), KP2 and KP8 (both Year 5). Exhibit 15: Progress towards Objective-2 indicators Indicator FY20 Actuals FY21 Actuals FY22 Targets FY22 Actuals (Up to Q4) % of FY22 target achieved LOP Target (Y5) Ind￾2.1 Percent of teachers who apply knowledge and skills in the classroom to provide a safe, predictable and caring learning environment N/A N/A 60% 74.90% 124.8% 70% Ind￾2.2 Percent of learners that demonstrate developmentally appropriate SEL skills N/A N/A 50% N/A N/A 70% Ind￾2a Number of teachers who have received training in SEL N/A 522 7,505 11,302 150.6% 8714 Ind￾2b Number of schools with leadership trained to promote SEL N/A N/A 35 N/A N/A 621 Ind￾2c Number of parents receiving PLC Kit N/A N/A 1,750 N/A N/A 31,050 Ind￾2d Number of parent teacher associations (PTAs) or community governance structures engaged in primary or secondary education supported with USG assistance N/A N/A 35 N/A N/A 621 51 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Indicator FY20 Actuals FY21 Actuals FY22 Targets FY22 Actuals (Up to Q4) % of FY22 target achieved LOP Target (Y5) (StandardES.1- 13) Ind￾2e Indicator YOUTH-1: Number of youths trained in soft skills/life skills through USG-assisted programs N/A 12,177 72,102 56,775 78.7% 131,843 As Exhibit 15 shows, Objective-2 indicator data exist for three of seven indicators while the remaining four cannot be calculated due to COVID-19 school closures’ postponement of the in￾school SEL program. Analysis of the three informed indicators (Ind 2.1, 2a and 2e) reflect LRP and the summer catch-up program incorporation of SEL-related skills in learning content and teacher training. These three indicators are output indicators. No SEL-related outcome data are available. QITABI has exceeded its FY2022 targets for two of these indicators (Ind 2.1 and 2a) by a large margin but has not achieved its target for the third (Ind-2e). Exhibit 16: Progress towards Objective-3 indicators # Indicator FY20 Actuals FY21 Actuals FY22 Targets FY22 Actuals (Up to Q4) % of FY22 target achieved LOP Target (Y5) Ind￾3.1 Number of new management and monitoring systems, procedures and tools recommended and/or introduced to (MEHE/DOPS and CRDP) as a result of USG assistance(custom) 10 11 12 13 108.3% 54 Ind￾3a Number of education administrators and officials who complete professional development activities with USG assistance (Standard ES.1-12) 170 296 277 572 206.5% 1061 52 | QITABI-2 MID-TERM PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – AUGUST 2023 Only two indicators inform Objective 3. Both have been calculated and show that QITABI exceeded its FY22 targets. They also represent adaptive management and pivots made as a result of the various crises the project and context face, such as the development of a new e-content system for CRDP that arose during HICD implementation. The professional development activities that Ind 31 references refer to the development of e-content and reflect how the original focus on HLA/BLA was postponed in favor of more urgent trainings on distance learning, HICD, and English communication skills.