SEPTEMBER 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Jehanzaib Khan, Leah Ghoston, Rana Taher, Tamara Nassereddine, Zeina Helou, and Nabil Constantine of Social Impact. MIDTERM PERFORMANCE EVALUATION QUALITY INSTRUCTION TOWARDS ACCESS AND BASIC EDUCATION IMPROVEMENT (QITABI) FINAL EVALUATION REPORT Performance Management and Support Program for Lebanon (PMSPL II) PHOTO COURTESY OF QITABI I | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION QUALITY INSTRUCTION TOWARDS ACCESS AND BASIC EDUCATION IMPROVEMENT (QITABI) Midterm Performance Evaluation Performance Management and Support Program for Lebanon (PMSPL II) for USAID/Lebanon Cover Photo: A public school teacher using the Balanced Literacy Approach in an Arabic-language class This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract/Order No. AID-268-C-15-00001. Limited redactions have been made to in accordance with the principled exceptions to the presumption in favor of openness established in OMB Bulletin 12-01, “Guidance on Collection of U.S. Foreign Assistance Data This document is available in printed or online versions. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com II | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ABSTRACT To expand equitable access and improve learning outcomes for vulnerable students in Lebanese public schools, USAID/Lebanon awarded the Quality Instruction Towards Access and Basic Education Improvement (QITABI) project to World Learning (WL) on September 30, 2014. QITABI is a four-year, $41.2 million education initiative, implementing activities under three components: Improving reading outcome, expanding access to safe and relevant education, and strengthening management in the education system. This mid-term evaluation for QITABI Project addresses the following three questions: 1. How relevant is QITABI project to the needs of the public basic education sector in Lebanon? 2. To what extent is QITABI on track to achieve its objectives—i.e., improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools? 3. To what extent are the project activities and results likely to continue after the project ends? The evaluation employed a qualitative approach, conducting 29 key informant interviews (KIIs) 38 focus group discussions (FGDs) with the different project stakeholders. The team triangulated data where possible from interviews and focus groups with the project’s quantitative monitoring and evaluation data. The evaluation concluded that QITABI interventions are aligned with national education strategies to directly respond to the priorities identified by the Lebanese government and USAID. Results thus far indicate that the project is on track to achieve its objectives; The EGR training was effective in improving the teaching skills of Arabic-language teacher, The ICT resource rooms have a positive effect on the learning environment, ICT training to DOPS and CERD ensures their professional development and continued ability to serve as mentors to teachers for ICT use. Project is facing many challenges; the significant delay in the delivery of the school buses will restrict the project in providing post-delivery support to schools and MEHE in their effective use. Although transferring skills related to EGR has been successful, challenges such as overcrowded classrooms can limit the efficacy of BLA. In addition, the infrastructure shortfalls challenge overall efficacy in some schools to benefit from the ICT equipment. In addition, key stakeholders have a limited understanding of the scope of Component 3, its activities, and/or how they contribute to QITABI’s objective. Evaluation team recommends QITABI to consider providing capacity-building to MEHE and CERD to implement EGRA and analyze the assessment data. QITABI should also consider providing refresher EGR training to teachers during the final year of the project. QITABI should consider providing DOPS with capacity development to build a core group of senior ICT coaches QITABI should consider conducting an assessment of enrollment and attendance in the selected schools to establish a baseline before delivering the buses to schools, and another one toward the end of the project. To help achieve the expected outcomes for Component 3 by the end of the project, QITABI should develop a detailed and actionable plan, with input from USAID, MEHE, and CERD, with clear activity descriptions, implementation strategies, and pathways. III | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION TABLE OF CONTENTS ABSTRACT........................................................................................................................................................II TABLE OF CONTENTS ................................................................................................................................III List of Figures..............................................................................................................................................................iv List of Tables...............................................................................................................................................................iv ACRONYMS..................................................................................................................................................... V EXECUTIVE SUMMARY............................................................................................................................... VI Evaluation Purpose and Evaluation Questions....................................................................................................vi Project Background ...................................................................................................................................................vi Evaluation Designand MethodsLimitations...........................................................................................................vi HIGHLIGHTS of the evaluation RESULTS...........................................................................................................vi Key Findings................................................................................................................................................................vii Conclusions ................................................................................................................................................................ ix Recommendations...................................................................................................................................................... x INTRODUCTION ...........................................................................................................................................1 Project Description ................................................................................................................................................... 1 Development Hypothesis......................................................................................................................................... 2 Critical Assumptions................................................................................................................................................. 2 EVALUATION PURPOSE AND QUESTIONS .........................................................................................3 Evaluation Question 1: Relevance.......................................................................................................................... 3 Evaluation Question 2: Effectiveness ..................................................................................................................... 3 Evaluation Question 3: Sustainability..................................................................................................................... 4 METHODS AND LIMITATIONS..................................................................................................................5 Desk Review................................................................................................................................................................ 5 Qualitative Data Collection..................................................................................................................................... 5 Limitations.................................................................................................................................................................... 6 FINDINGS AND CONCLUSIONS..............................................................................................................8 Question 1: Relevance .............................................................................................................................................. 8 Question 2: Effectiveness....................................................................................................................................... 11 Question 3: Sustainability....................................................................................................................................... 19 RECOMMENDATIONS................................................................................................................................22 ANNEXES ........................................................................................................................................................24 ANNEX I: INCEPTION REPORT ..............................................................................................................25 ANNEX 2: DOCUMENTS REVIEWED ....................................................................................................48 ANNEX 3: DATA COLLECTION INSTRUMENTS ..............................................................................49 ANNEX 4: QITABI INTENDED RESULTS/SCOPE OF WORK.........................................................95 IV | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION LIST OF FIGURES Figure 1: QITABI Activity Locations............................................................................................................................. v Figure 2: QITABI Logical Framework..............................................................Error! Bookmark not defined. LIST OF TABLES Table 1: FGD Sample........................................................................................................................................................ 6 Table 2: EGRA Baseline, Number of Students Assessed ....................................................................................... 11 Table 3: EGR Training Beneficiaries............................................................................................................................ 12 V | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ACRONYMS ALC Arabic Language Committee AMIDEAST America-Mideast Educational and Training Services, Inc. BLA Balanced Literacy Approach CDCS Country Development Cooperation Strategy CERD Center for Educational Research and Development COP Chief of Party COR Contract Officer Representative DEC Development Experience Clearinghouse DO Development Objective DOPS Département d’Orientation Pédagogique Scolaire/Guidance and Counseling Department EGR Early Grade Reading EGRA Early Grade Reading Assessment EQUI Evaluation Quality Use and Impact ESDP Education Sector Development Plan ET evaluation team EWS early warning system FGD focus group discussion GD General Director GOL Government of Lebanon ICT information communication technology IP implementing partner IR Intermediate Result LOE level of effort LOP life of project KII key informant interview KPI key performance indicator M&E monitoring & evaluation MCE Microsoft Certified Educator MEHE Ministry of Education and Higher Education MOS Microsoft Office Specialist MoU Memorandum of Understanding MSI Management Systems International NETSP National Education Technology Strategic Plan NFE non-formal education O&M operation and maintenance PMSPL II Performance Management and Support Program for Lebanon PRC parent reading circle QITABI Quality Instruction Towards Access and Basic Education Improvement RACE II Reaching All Children with Education II SI Social Impact, Inc. SOW Statement of Work TLM teaching and learning materials TOT training of trainers UPS Uninterruptible Power Supply USAID United States Agency for International Development USG United States Government VAT value-added tax WL World Learning Figure 1: QITABI Activity Locations VI | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The objectives of this midterm performance evaluation of the Quality Instruction Towards Access and Basic Education Improvement (QITABI) project are as follows: (1) to determine the relevance, effectiveness, and sustainability of progress toward reaching the project’s expected results; (2) to provide pertinent information about the project results thus far; and (3) to offer practical recommendations for maximizing the benefits of the project. The evaluation addresses the following three questions: 4. How relevant is QITABI project to the needs of the public basic education sector in Lebanon? 5. To what extent is QITABI on track to achieve its objectives—i.e., improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools? 6. To what extent are the project activities and results likely to continue after the project ends? PROJECT BACKGROUND The Lebanese government faces multiple barriers to providing access to quality education for all children. The public education system is struggling to keep pace with the increasing demands of providing access to quality education due to the Syrian refugee inflow since 2011, straining Lebanon’s already limited public education system. To expand equitable access and improve learning outcomes for vulnerable students in Lebanese public schools, USAID/Lebanon awarded the QITABI project to World Learning (WL) and its implementing partners on September 30, 2014. QITABI is a four-year, $41.2 million education initiative, with a period of performance from September 30, 2014 through September 29, 2018. QITABI is implementing activities under the following three components: • Component 1: Improving reading outcomes for primary-level public school students • Component 2: Expanding access to safe and relevant education for vulnerable public school students • Component 3: Strengthening management (resilience) in the education system to better direct and monitor education EVALUATION DESIGNAND METHODSLIMITATIONS The evaluation employed a qualitative approach, conducting 29 key informant interviews (KIIs) 38 focus group discussions (FGDs) with the different project stakeholders. Data analysis included summarizing FGD and KII notes categorized by evaluation question, coding key themes, organizing these themes in a detailed analysis matrix, and developing findings based on these themes. Following the analysis, the evaluation team prepared a detailed outline of key findings, conclusions for each evaluation question, and overall recommendations. The team triangulated data where possible from interviews and focus groups with the project’s quantitative monitoring and evaluation data. HIGHLIGHTS OF THE EVALUATION RESULTS This section provides a brief outline for QITABI’s current achievements and the proposed recommendations for improvement. VII | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION What is going well: • QITABI successfully implemented the Early Grade Reading Assessment (EGRA) baseline, developed reading benchmarks, and exceeded its target of assessing 3,840 students in Arabic reading in Grades 2 and 3. • Scaling up EGRA and adopting reading benchmarks has been accepted by MEHE and CERD. Implementing the EGRA tool, benchmarks, and EGR modules will continue • Improved reading outcomes in Arabic language previously ignored in the public sector is now considered a necessity by MEHE. The Balanced Literacy Approach (BLA) has been accepted. • MEHE’s General Director reactivated the Arabic Language Committees (ALCs), which are required to support improvements in Arabic-language teaching in their respective schools. • The Early Grade Reading (EGR) intervention effectively built DOPS and CERD capacity to coach/train Arabic-language teachers, enhancing teachers’ assessment knowledge and improving their teaching techniques. • Implementation of classroom libraries and ICT equipment encouraged innovation in teaching, increased enthusiasm for reading, and improved the school learning environment. • QITABI developed an early warning system (EWS) for the teachers to identify students at risk of developing reading difficulties and to address student needs with support from the school and community. The EWS is to be piloted in 20 schools along with parent reading circles (PRCs) in 10 schools. • Four hundred ICT resource rooms have been provided to 300 schools exceeding the targeted rollout. The ICT resource rooms improve the school learning environment and facilitate more interactive teaching methods, benefiting both students and teachers. How things can be improved: • Provide MEHE and CERD with the needed scale up support for EGRA and the capacity￾building for its implementation and analysis. • Provide refresher EGR training to teachers during the final year of the project that include practical applications of the BLA, allowing trainees to observe sample lessons in overcrowded classroom settings. • Add necessary trainings to improve ICT use in public school and complement DOPS support. • Strengthen data collection, monitoring, resource management, and the sensitivity of MEHE units to informed decision-making. • Increase collaboration with MEHE and USAID to initiate the year 5 activities early in the current academic year. Examples include detailed actionable plan for component 3, the pilot EWS and PRC, and an assessment to measure the impact of buses on enrollment and attendance. KEY FINDINGS Following are the key findings for each evaluation question by project component. QUESTION 1: HOW RELEVANT IS THE QITABI PROJECT TO THE NEEDS OF THE PUBLIC BASIC EDUCATION SECTOR IN LEBANON? Component 1: Improving reading outcomes for primary-level public school students Improving reading outcomes in Arabic language was previously ignored in the public sector and is considered a necessity by MEHE and USAID. It contributes toward improving the quality of VIII | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION education in public schools, which is supported both by Education Sector Development Plan (ESDP) and Reaching All Children with Education II (RACE II). Component 2: Expanding access to safe and relevant education QITABI initially planned to focus on expanding access to education for vulnerable children through non-formal education (NFE), but MEHE did not consider it a priority.1 After 18 months’ delay in implementation and extensive negotiations with MEHE, QITABI replaced NFE with the provision of ICT resource rooms to schools; professional development in ICT to CERD, DOPS, and teachers; and buses to 100 schools. These interventions are in line with the award, remain relevant, support MEHE’s needs, and respond to USAID’s Country Development Cooperation Strategy (CDCS/Lebanon) and Education Strategy for expanding access to education for vulnerable children in Lebanon. Component 3: Strengthening management (resilience) in the education system to better direct and monitor education Component 3 supports MEHE and CERD to develop policies/guidelines through research, policy workshops, and technical support from consultants. In addition, it seeks to strengthen data collection, monitoring, management, and the sensitivity of MEHE units to informed decision-making. Strengthening management in the education system aligns with MEHE’s needs as highlighted in its ESDP and RACE II. It also supports USAID’s CDCS. QUESTION 2: TO WHAT EXTENT IS QITABI ON TRACK TO ACHIEVE ITS OBJECTIVES? Component 1: Improving reading outcomes for primary-level public school students QITABI successfully implemented the EGRA baseline, developed reading benchmarks, and exceeded its target of assessing 3,840 students in Arabic reading in Grades 2 and 3. The EGR intervention effectively built DOPS and CERD capacity to coach/train Arabic-language teachers, enhancing teachers’ assessment knowledge and improving their teaching techniques. However, access to libraries and ICT equipment is primarily for Shift 1 students, as principals in many schools restrict the use of these resources in Shift 2 for maintenance purposes. QITABI has overachieved its Year 2 target for teachers trained (achieved, 639; target, 294) and is progressing to achieve its Year 3 target of completing training for 1,295 teachers. QITABI developed an early warning system (EWS) for the teachers to identify students at risk of developing reading difficulties and to address student needs with support from the school and community. QITABI will pilot the EWS in 20 schools along with parent reading circles (PRCs) in 10 schools in September once MEHE approves. Component 2: Expanding access to safe and relevant education QITABI has achieved its Year 3 target of providing 400 ICT resource rooms to 300 schools. Schools appreciate the ICT resource rooms because they improve the school learning environment. Yet in overcrowded conditions in some schools, teachers fear that frequent use among a higher number of students will wear out the equipment earlier than would happen otherwise. MEHE would like intensive training for DOPS coaches to form a small senior ICT team to provide training to fellow coaches. ICT resource room equipment enables CERD centers to provide comprehensive training on equipment with which they now have firsthand experience. The purchase of 100 buses is pending a waiver from USAID for non-US manufactured vehicles. Developing the operation and maintenance (O&M) plan, along with other factorshave delayed the procurement for almost a year. Completing the procurement and delivery of buses will likely take 1 MEHE has not yet developed an NFE framework; as a result, the Ministry is not supporting NFE activities in Lebanon. IX | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION another year, disallowing any time for QITABI to assess their use, adherence to the O&M plan, and effectiveness in improving access to education. Component 3: Strengthening management (resilience) in education system to better direct and monitor education On QITABI’s recommendation, MEHE’s General Director reactivated the Arabic Language Committees (ALCs). These committees are required to support improvements in Arabic-language teaching in their respective schools. QITABI plans to mobilize ALC support for the BLA implementation in Year 4. QITABI is working with the Director of Basic Education at MEHE to develop a simple monitoring and evaluation (M&E) system for the directorate to collect data from the schools about QITABI interventions and access QITABI data collected throughout the implementation. Several key stakeholders (from MEHE and CERD) expressed concerns that they have limited knowledge, clarity, or understanding of Component 3 activities and its scope or purpose. QUESTION 3: TO WHAT EXTENT ARE THE PROJECT ACTIVITIES AND RESULTS LIKELY TO CONTINUE AFTER THE PROJECT ENDS? Component 1: Improving reading outcomes for primary-level public school students MEHE and CERD will continue to implement the project with the EGRA tool, benchmarks, and EGR modules. CERD has formally expressed interest, through an official letter to USAID, in scaling up EGRA and adopting the reading benchmarks. Although CERD and MEHE recognize its value, it is not ready to support EGRA scale-up due to lack of capacity and resources. Regarding sustainability of libraries and ICT equipment, many report that overcrowded classrooms may inhibit maintenance of the libraries because of overuse In contrast, the classroom ICT equipment is under warranty for up to three years. In addition, many schools have an ICT point person who serves as a key resource to troubleshoot and maintain equipment. However, this person’s longevity as a school resource depends on allocation of funds from the school budget. Without ICT training or funding, the use of ICT in BLA will be limited. Component 2: Expanding access to safe and relevant education QITABI is in the process of building DOPS capacity in ICT to enable them to coach teachers.DOPS will receive further ICT training during the final year of the project. CERD training centers have ICT resource room equipment that their trainers have been trained to use. CERD plans to offer ICT training as part of its program during and after the project ends. The O&M plan for the school buses provides a structured framework for MEHE and schools. however, the remaining project time is limited for QITABI to provide support to MEHE and schools after the delivery of the buses. Component 3: Strengthening management (resilience) in education system to better direct and monitor education The ALCs have been reactivated to support Arabic-language teaching and EGR interventions at the school level. Some stakeholders expressed a lack of clarity about the ALCs’ usefulness to project interventions CONCLUSIONS 1. QITABI interventions are aligned with national education strategies to directly respond to the priorities identified by the Lebanese government and USAID. QITABI’s initial plan to focus on NFE, however, did not match MEHE’s priorities for development projects. QITABI’s flexibility resulted in MEHE’s buy-in. The government (MEHE and CERD) recognizes the importance of EGRA, EGR, classroom libraries, and the reading benchmarks as necessary initiatives; while X | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION CERD has formally expressed to USAID interest in supporting the scale-up of EGRA and adopting the reading benchmarks, MEHE is concerned that it lacks the capacity and resources to expand EGRA. 2. Overall, QITABI’s leadership, strong project management, and approach to working collaboratively with MEHE and CERD have contributed to the project’s progress thus far and to building strong relationships with key stakeholders. Results thus far indicate that the project is on track to achieve its objectives. This is especially true for Components 1 and 3. QITABI is on track to achieve Component 2 results associated with the ICT resource rooms and professional development in ICT. However, the significant delay in the delivery of the school buses will restrict the project in providing post-delivery support to schools and MEHE in their effective use and operation and in assessing their effects on expanding access to education for vulnerable children. 3. The EGR training was effective in improving the teaching skills of Arabic-language teachers. Students’ fundamental skills in Arabic-language reading, level of interest, and relationship with teachers and other students have shown improvement. Although transferring skills related to EGR has been successful, challenges such as overcrowded classrooms can limit the efficacy of BLA. 4. The ICT resource rooms have a positive effect on teaching, student learning, engagement in classroom activities, and the school learning environment, but infrastructure shortfalls challenge overall efficacy in some schools. Providing ICT training to DOPS and CERD ensures their professional development and continued ability to serve as mentors to teachers for ICT use. 5. Key stakeholders have a limited understanding of the scope of Component 3, its activities, and/or how they contribute to QITABI’s objective. Although coordinating directly with the General Director for Component 3 can generate swift decision-making, it may add to a lack of understanding among key MEHE officials as a result of not being included in coordination and decision-making. 6. QITABI’s activities that involved building CERD and DOPS capacity are likely to be sustained. CERD can now continue conducting EGR and ICT training without outside support. CERD’s interest in scaling up EGRA and adopting the reading benchmarks is a significant step toward institutionalization. Full scale-up, however, will require strong support from MEHE, which is subject to external support for capacity-building and resources. Following the O&M plan is critical for MEHE to continue to operate and maintain the buses once the QITABI project ends. RECOMMENDATIONS RECOMMENDATIONS TO IMPROVE QITABI’S IMPLEMENTATION 1. To obtain MEHE’s endorsement of the reading benchmarks and EGRA expansion, QITABI should develop an advocacy campaign with CERD input. In addition, QITABI should consider providing capacity-building to MEHE and CERD to implement EGRA and analyze the assessment data. QITABI should also consider providing refresher EGR training to teachers during the final year of the project 2. To allow adequate time for schools to benefit from technology, QITABI should prioritize and expedite the provision of e-content as well as complementary ICT materials (CDs). 3. QITABI should consider providing DOPS with capacity development to build a core group of senior ICT coaches. Similarly, the development and handing over of the online training platform (Moodle) to CERD should include training of CERD officials in its operation, maintenance, and expansion. 4. QITABI should consider conducting an assessment of enrollment and attendance in the selected schools to establish a baseline before delivering the buses to schools. A similar assessment later XI | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION during the academic year will provide data to compare with baseline to measure increases both in enrollment and attendance. This assessment will be subject to project extension. 5. To help achieve the expected outcomes for Component 3 by the end of the project, QITABI should develop a detailed and actionable plan, with input from USAID, MEHE, and CERD, with clear activity descriptions, implementation strategies, and pathways. To increase commitment and support, QITABI should educate MEHE and CERD officials about the component in order to increase their understanding of its objectives, activities, and expectations. 6. QITABI should work closely with MEHE—and if necessary request USAID assistance—obtain its support to: a. Pilot EWS and PRCs as early as the first quarter of the coming academic year (2017- 2018). b. Advocate for and support MEHE to direct schools to allow access to classroom libraries and ICT equipment to Shift 2 students. c. Ensure that the ALCs function effectively during and after the project d. Complete the development of the M&E system for the basic education directorate to ensure its use after QITABI ends. 7. QITABI should consider providing technical assistance to MEHE so that the Ministry can determine allocation of its own resources and/or mobilize external resources to maintain classroom libraries and ICT equipment in QITABI schools to ensure that children have continued access to these materials and benefit from them. 8. QITABI should closely work with CERD to develop a clear strategy outlining how CERD plans to offer EGR and ICT training to schools after QITABI ends and identify technical support (further training, technical consultants) for CERD in order to implement these plans RECOMMENDATION FOR FUTURE EDUCATION PROGRAMMING 1. Extend QITABI for at least one more year. With this extra time monitor and assess Component 2 activities that have not yet fully matured. New activities should not be introduced during the extension period. 2. USAID should advise QITABI to conduct a gender assessment of the content employed by the project (e.g., classroom library books, classroom observation tools). USAID should use findings from the ongoing gender assessment of the education sector to inform the design of the next education project. 3. Future project design plans (PDP) in the education sector should consider activities aimed at continuing improvement in Arabic reading outcomes, based on the findings of this evaluation. Specific future activities to consider are: a. Funding the continuation of initiatives to support Arabic reading skills; b. Short term capacity-building for CERD and MEHE in analyzing standardized assessment data; c. Supporting DOPS and CERD in implementation of EGRA as an assessment and institutional and learning tool. 4. Because the evaluation found that the training of trainers (TOT) model is effective and sustainable, future activities that include teacher training would benefit from incorporating the TOT model in their design to promote CERD and DOPS capacity-building, ownership, and sustainability. 1 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION INTRODUCTION PROJECT DESCRIPTION The public education sector in Lebanon has struggled to increase the quality of public education since the end of the civil war in 1990. Basic public education in particular faces multiple barriers to providing quality education and access for all children. Challenges include low learning outcomes, lack of qualified teachers, an outdated early-grade curriculum, limited professional development opportunities for teachers, and weak monitoring systems. In addition, schools have poorly resourced facilities, such as a shortage of information communication technology (ICT) and teaching and learning materials (TLM), including library books. The absence of a standardized measure of quality in the early grades—i.e., student learning outcomes and a monitoring system to assess student progress further hinder the development of effective strategies to improve quality education.2 To complicate matters, the recent influx of Syrian refugees into the country since 2011—resulting in an increase in student numbers—has further strained the public school system and introduced other challenges to making quality education accessible to all children. Lebanon currently hosts approximately 1.5 million Syrian refugees, more than half (54%) of whom are children. Only 40% of these children are enrolled.3 This strain has been felt not only by students, but also by teachers, principals, parents. and the Ministry for Higher Education and Education (MEHE). The emphasis of MEHE and donors has shifted from access to education alone to a focus on quality education, which is intended to be not only more inclusive of vulnerable children but that also provides superior instruction in language and literacy, particularly reading skills improvement.4 Schools should aim to provide professional development and training opportunities to teachers to improve their curriculum quality and to equip them to manage their classrooms and their student learning outcomes more effectively. To address these issues of quality education and access in Lebanese primary schools, the United States Agency for International Development in Lebanon (USAID/Lebanon) awarded the Quality Instruction Towards Access and Basic Education Improvement (QITABI, or “My Book” in Arabic) project to World Learning and its partners. QITABI is a four-year, $41.2 million education initiative, with a period of performance of September 30, 2014 through September 29, 2018. QITABI aims to improve reading outcomes for public primary school students, expand access to quality education for vulnerable children, and strengthen management within the public education system. The project supports MEHE in developing the professional capacity of teachers to provide comprehensive and quality instruction, improve Arabic reading skills in early grades (1–4), and improve the quality of learning environments (e.g., through the provision of ICT equipment and classroom libraries) for both enrolled public school students and out-of-school vulnerable children, including refugees. QITABI also addresses barriers to accessing education and builds MEHE’s capacity to sustain these reforms. Within USAID/Lebanon’s Country Development Cooperation Strategy (CDCS) 2014–2019, QITABI promotes the achievement of Development Objective (DO) 1 (Improved capacity of the public sector in providing transparent, quality services across Lebanon) and of the corresponding Intermediate Result 1.1 (Expanded equitable access and improved learning outcomes for vulnerable students in Lebanese public schools). The project is also responsive to Goal 1 (Improved reading skills for 100 million children in primary grades by 2015) and Goal 3 (Increased equitable access for 15 million learners in environments affected by crises and conflict by 2015) of USAID’s Education Strategy 2011–2015. 2 USAID (2014). Request for Application – Improved Basic Education Services Project (IBESP) 3 Ministry of Education and Higher Education (2016). Reaching All Children with Education: RACE II (2017–2021) Government of Lebanon and the United Nations (2017). Lebanon Crisis Response Plan (2017–2020) UNHCR (2016). UNHCR Reports Crisis in Refugee Education. Retrieved from UNHCR website: http://www.unhcr.org/afr/news/press/2016/9/57d7d6f34/unhcr-reports-crisis-refugee-education.html 4 Ibid. 2 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION To meet these objectives, the project provides capacity-building, technical assistance, transport facilities, ICT equipment, and TLM within the context of three project components with associated activities. Each component contributes to a sub-Intermediate Result (sub-IR) under the CDCS Development Objective 1: • Component 1 (sub-IR 1.1.1): Improving reading outcomes for primary-level public school students • Component 2 (sub-IR 1.1.2): Expanding access to safe and relevant education for vulnerable public school students • Component 3 (sub-IR 1.1.3): Strengthening management (resilience) in the education system to better direct and monitor education World Learning is the lead implementing agency and overall cooperative agreement grantee for QITABI. It manages coordination with and among MEHE, including the General Director for Basic Education, DOPS, and CERD. World Learning provides management of and leadership for all three project components, with support from three other partners: 1. Management Systems International (MSI): Under Component 1, MSI leads EGRA, teacher surveys, and establishes Arabic-language reading benchmarks for Grades 2 and 3. 2. Ana Aqra’a: Under Component 1, Ana Aqra’a supports EGR training and coaching for teachers; training for CERD and DOPS; EWS; and parental awareness. 3. America-Mideast Educational and Training Services, Inc. (AMIDEAST): Under Component 2, AMIDEAST provides professional development to support implementation of ICT in schools and through educational activities. DEVELOPMENT HYPOTHESIS QITABI’s development hypothesis is threefold: (1) providing its intervention package (assessments, teacher training, coaching, classroom materials) will improve learning outcomes, particularly reading at the primary school level; (2) providing improved facilities (ICT resource rooms) and services (free transport, improved teaching) will increase access to quality education; and (3) improving government skills in education management and standardized assessment will increase MEHE’s capacity to oversee services, ensuring sustainability of QITABI’s results. (See Annex 4 for a list of objectives, outcomes, and proposed indicators). CRITICAL ASSUMPTIONS The key underlying assumptions for the successful implementation of QITABI are as follows: 1. Strong support from MEHE is crucial to the successful and timely implementation of the project activities and achievement of targets. Two of the major challenges for QITABI were buy-in from MEHE and obtaining grant approval from the Government of Lebanon (GOL). The signing of the Memorandum of Understanding (MoU), which only took place on March 31, 2016, is critically linked to the GOL issuing approval of the grant; hence, the implementation of the activities of the project at full capacity was delayed until the signing of the MoU with MEHE and grant approval issuance by GOL. 2. There must be a measure of stability in the operating environment. If security deteriorates or conflict flares up, then activities in the affected areas would be suspended and possibly modified/canceled to respond to changing needs. 3. Timely support and activity approvals from USAID are essential for ensuring timely implementation of planned activities. 3 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION EVALUATION PURPOSE AND QUESTIONS The purpose of the QITABI midterm performance evaluation is to assess the relevance, effectiveness, and sustainability of the project and its expected results. The evaluation provides pertinent information about project results thus far, lessons learned, and practical recommendations, including corrective actions (if any) required for maximizing the benefits of the project. The evaluation findings will assist USAID, project implementing partners, and CERD/MEHE in decision￾making to (a) understand the immediate results manifesting in the targeted schools, in MEHE, and in CERD; (b) enhance QITABI’s interventions in the remaining life of the project; and (c) inform future USAID basic education programs. The evaluation covers the period between September 2014 and May 2017. The primary audience for the evaluation includes (i) the USAID/Lebanon Mission, particularly the Education, Democracy, and Governance Office, and (ii) QITABI implementing partners. Secondary users include (i) MEHE, (ii) CERD, and (iii) other donors and organizations working on basic education in Lebanon. The evaluation answers the following questions as identified by USAID/Lebanon. EVALUATION QUESTION 1: RELEVANCE How relevant is the QITABI project to the needs of the public basic education sector in Lebanon? This question focuses on exploring ways in which the project supports: a. The emerging needs of MEHE and the increased demand for access to quality education and educational systems as the result of the Syrian refugee influx in the country. MEHE Education Sector Development Plan (ESDP 2010–2015) and Reaching All Children with Education (RACE II 2017 and 2021)5 will be used as references to help guide the answer to this question. b. USAID’s 2011–2015 Education Strategy and USAID/Lebanon’s CDCS 2014–2019 EVALUATION QUESTION 2: EFFECTIVENESS To what extent is QITABI on track to achieve its objectives—i.e., improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools? The answer to the question should explore: a. What have been the concrete contributions of the project interventions thus far in achieving the objectives of the project? b. What has worked and how, what has not worked as anticipated, and what are any unintended outcomes? c. What factors have been crucial for the project’s achievement (or lack thereof) of planned results thus far? d. What gender considerations have been taken into account by different project interventions and what are the most visible discrepancies, if any? 5 RACE II is MEHE’s multi-year, multi-stakeholder national strategy, which aims to deliver education to all children in Lebanon, including refugees and vulnerable Lebanese children. The Executive Committee—an advisory body that includes donors and UN agencies—manages the overall progress of the RACE implementation. RACE focuses in improving access to education opportunities, increasing the quality of education services, and strengthening the capacity of the education system. 4 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION EVALUATION QUESTION 3: SUSTAINABILITY To what extent are the project activities and results likely to continue after the project ends? This question focuses on exploring: a. What are the prospects for the sustainability of the end results of the project, what results appear to be less sustainable, and why? b. What is QITABI’s sustainability plan, and what is required of QITABI, MEHE, and USAID for the sustainability of the results? 5 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION METHODS AND LIMITATIONS The evaluation design consisted of a primarily qualitative approach using key informant interviews (KIIs) and focus group discussions (FGDs). The KII respondents included QITABI project staff, USAID/Lebanon’s Education, Democracy, and Governance Office staff, MEHE, and CERD. The FGD participants consisted of the direct project beneficiaries—teachers, principals, CERD trainers, and DOPS coaches. No primary quantitative data were collected, though the evaluation used project￾monitoring data to triangulate findings where relevant. Data collection occurred from May 10 to June 2, 2017. A comprehensive evaluation design matrix with the evaluation questions, data sources, data collection methods, and analysis methods can be found in Annex 1. DESK REVIEW The evaluation team conducted a systematic review of relevant internal and external documents, including the original contract agreement, contract modifications, EGRA baseline reading benchmarks reports, project work plans, monitoring and evaluation (M&E) plans, progress reports (annual/ quarterly), memoranda of understandings signed with MEHE, USAID’s Education Strategy, the RACE II strategy document, and all other project documents relevant to answering the evaluation questions. In addition, the evaluation team reviewed relevant publicly available MEHE documents, which provided insights into MEHE’s strategy and education sector priorities and which may lend additional context to the project’s relevance, effectiveness, and sustainability. Annex 2 contains the list of all documents reviewed. QUALITATIVE DATA COLLECTION KEY INFORMANT INTERVIEWS (KII): The evaluation team6 conducted 29 KIIs (26 of which were with women) with USAID key staff members, World Learning, Ana Aqra’a, AMIDEST, MSI, MEHE (General Director (GD) for Basic Education, DOPS), and CERD. The KIIs focused on project activities and their relevance to MEHE needs, factors that contributed toward achievement of the project results to date, sustainability of the anticipated project results, and implementation challenges. Key informants were purposively selected because of their intimate knowledge of and perspective on different project components. Key informants consisted of a diverse group of stakeholders, including decision-makers within partner organizations, MEHE, activity managers, points of contact, and M&E officials (see Figure 3). The interview guides and the list of interviewees can be found in Annexes 3 and 4, respectively. The evaluation team conducted six follow-up interviews with QITABI leadership, and CERD and MEHE officials in late June for clarification and additional information about the preliminary findings. The team’s local education expert, a former senior CERD official, led these interviews. FOCUS GROUP DISCUSSIONS (FGDS): With enumerators from its data collection partner Information International, the Social Impact (SI) evaluation team facilitated a total of 38 FGDs with project beneficiaries. Twenty-six FGDs included school-level beneficiaries—primary school teachers and principals. The remaining 12 included CERD trainers, DOPS coaches, and QITABI EGR facilitators and trainers. The FGD questions for school-level beneficiaries focused on the efficacy of trainings and other project activities as well as on changes that teachers and principals observed in their practices, classrooms, students, and schools following participation in QITABI activities. FGD sampling was conducted purposively to ensure that schools, principals, and teachers from the different regions (governorates and cazas) were proportionately represented and to cluster FGDs 6 The evaluation team consisted of six core members—one international consultant who led the team, one SI staff member from headquarters (Washington, D.C.), two local SI staff members, and two local consultants—and an additional SI team member was added as needed. In the interest of time, the team divided into three pairs to conduct the KIIs and note-taking. 6 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION together geographically for ease of access for participants. It took into account participant mobility, geographic spread of schools, FGD venue proximity to FGD participants, and total number of QITABI beneficiaries in each category. The evaluation team also considered representation of both rural and urban primary schools and the inclusion of both men and women in the FGDs where possible. The FGDs with teachers mainly focused on Component 1 activities (EGR intervention, school libraries, classroom ICT, EGR coaching), and the FGDs with principals focused on Component 2 activities (ICT resource rooms, professional development in ICT, transportation/buses). The number of FGDs per task was determined based on how wide the coverage of the task was at the time of the evaluation and the relative sensitivity or importance of the findings to the evaluation purpose. Table 1: FGD Sample The evaluation included four FGDs with CERD and five FGDs with DOPS-level beneficiaries who participated in QITABI trainings of trainers (TOTs). These FGD questions focused on the relevance and effectiveness of the training, the effectiveness and sustainability of DOPS coaching support to schools, and the rollout trainings that CERD conducted. The CERD centers were chosen to represent different regions in Lebanon. Members of the evaluation team also conducted three focus groups with EGR facilitators and trainers to triangulate their feedback with those of trainees and get a more complete picture of the effectiveness and challenges of trainings. A total of 344 respondents participated in FGDs; 79% were women. A two-member team of enumerators that consisted of a moderator and a note-taker conducted each FGD. While the moderator’s job was to make sure all questions included in the FGD guides were adequately addressed, the note-takers audiotaped the discussion and took detailed notes for each question. Table 1 illustrates the number of FGDs, schools, and respondents for each stakeholder group. DATA ANALYSIS: The evaluation team began the study with a systematic review of all available program documents to trace the process that QITABI used to implement the project. After the team completed qualitative data collection and received the focus group notes, it reconvened in Beirut to analyze the data and develop findings, conclusions, and recommendations. The team used a structured and systematic approach to analyze the qualitative data and, where possible, triangulated methods and sources to ensure the reliability and validity of results. Key steps included: • Summarizing notes from individual interviews and FGDs • Coding KII and FGD data according to themes relevant to the evaluation questions • Preparing an evaluation data analysis matrix with the themes that emerged in the KIIs and FGDs to identify key study findings as they related to the evaluation questions • Preparing a detailed outline summarizing key findings based on the data analysis, conclusions for each study question, and overall recommendations Project documents allowed the team to verify claims that emerged from interviews—for example, when interviewees noted that project interventions were relevant to public education needs, the evaluation team checked the education sector plan to see if this was in fact the case. Quantitative monitoring data offered statistical information (e.g., number of resource rooms, number of male/female teachers trained) that allowed the evaluation team to determine to what extent the project achieved its targets. LIMITATIONS SELECTION BIAS: Because qualitative data collection is not intended to produce statistically representative samples, it often relies on purposive sampling, meaning that respondents are selected for their particular roles, responsibilities, and knowledge. As a result, although these respondents 7 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION typically represent a variety of views on a topic, they may not represent all relevant views, and selection bias can emerge. RESPONSE BIAS: The scope of the study did not include assessing student outcomes. This makes the evaluation highly reliant on teachers and principals’ perceptions and self-reports of the effect of services and equipment on teacher practices in the classroom and student learning outcomes. It is possible that teachers overestimate effects on student learning, either because they are saying what they think the evaluation team wants to hear (response bias/social desirability bias) or because their view of the effects of their changed teaching methods and/or improved learning environment on student outcomes is more optimistic than the actual student outcomes. That said, the evaluation team considers teachers and principals’ perceptions of improved teaching practices and learning outcomes to be an important data source. GENDER: The evaluation addresses gender and social dimensions to the extent possible given sample and data limitations. The majority of the public primary school teachers in Lebanon are female (approximately 87%). 7 It was not possible to hold separate FGDs for male and female teachers. As a result, findings from the FGDs cannot be disaggregated by gender. However, the report includes gender-disaggregated project data by project achievements where possible. For example, the protocols included asking interview and FGD respondents about gender considerations and perceptions of any gender differences in student or teacher classroom behaviors. TIMING: Since this is a midterm evaluation and QITABI closes in September 2018, QITABI partners and USAID will have ample time to benefit from the findings. However, the timing of data collection—between May 10 and June 2, 2017—is a particularly busy time for MEHE, teachers, and principals, who conduct annual examinations and grade papers. This affected the expected turnout (i.e., between 7 and 10 participants per FGD) in some of the FGDs, where the actual participation averaged about five participants each. Participation in the majority of the FGDs was within the expected range. OBSERVATIONS: Site visits to observe project activities were not possible because the schools were not in session and most QITABI training activities had already ended for the academic year. 7 UNESCO: Percentage of Female Teachers by Teaching Level of Education. http://data.uis.unesco.org/index.aspx?queryid=178 (Retrieved on May 10, 2017). Most recent data from 2014. 8 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION FINDINGS AND CONCLUSIONS The findings in this section of the report are based on three distinct sources: (I) key informant interviews conducted with representatives of QITABI implementing partners, USAID, MEHE, and CERD officials, (2) focus groups discussions with primary beneficiaries and implementing partners’ trainers, and (3) project documents and M&E data. Counts from the key informant interviews and focus group discussions are cited next to their corresponding findings below. The counts include the number of KIIs and/or FGDs in which the finding was present out of the total number of relevant KIIs and FGDs—for example, KII 8/13; FGD 9/16. QUESTION 1: RELEVANCE “How relevant is the QITABI project to the needs of the public basic education sector in Lebanon?” FINDINGS COMPONENT 1: IMPROVING READING OUTCOMES FOR PRIMARY-LEVEL PUBLIC SCHOOL STUDENTS Under Component 1, QITABI introduced the EGR model by developing and implementing EGRA baseline, developing reading benchmarks and EGR teacher training manuals based on the Balanced Literacy Approach (BLA), training DOPS in EGR coaching, providing EGR TOTs to CERD trainers, and supporting the EGR training rollout to Arabic-language teachers. In addition, QITABI provided classroom libraries and classroom ICT equipment in 260 schools. The results generated by these activities are intended to contribute toward achieving the Sub-IR 1.1.1 (Improving reading outcomes for primary-level public school students). Early Grade Reading (EGR) The EGR model that QITABI developed under Component 1, which primarily focuses on improving early-grade reading in Arabic, aligns with USAID’s CDCS and Education Strategy (KII 6/7). Improvement in early reading in Arabic language was previously ignored and is considered a necessity by MEHE, USAID, and QITABI. In addition, it contributes toward improving quality in public schools in Lebanon, which is supported both by MEHE’s ESDP and RACE II (KII 12/15). EGRA During Year 1 of the project, QITABI developed the EGRA baseline tool in collaboration with CERD and MEHE and implemented it in 120 Cohort 1 schools in the beginning of Year 2 and in 120 Cohort 2 schools a few months later. The EGRA tool was piloted in 20 schools before full implementation in 240 schools. The EGRA baseline implementation was followed by the development of reading benchmarks. Data from KIIs and document review indicate that EGRA responds to USAID Education Strategy as a key intervention for assessing reading outcomes (KII 6/9). Lebanon does not have a national-level standardized assessment of student achievement; MEHE ESDP recommends it and RACE II identifies it as a gap (KII 5/9). QITABI perceives national-level assessments and establishing benchmarks for achievement necessary for improving learning outcomes in Arabic language (KII 8/14). While MEHE sees their importance and relevance, it considers EGRA/national-level assessments costly and believes that MEHE does not have the capacity to implement them. CERD, on the other hand, has formally expressed an interest in scaling up the EGR activities, including EGRA, and adopting the reading benchmarks (KII 3/5). 9 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION EGR Training Arabic language (or any language) is not mentioned in ESDP, but training in teaching Arabic responds to ESDP’s component of professional development for public school teachers and improving the quality of education—a priority for MEHE (KII 8/14). QITABI developed the EGR training in Year 1 and trained DOPS counselors in EGR coaching and CERD trainers through a trainer of trainers (TOT) approach in the beginning of Year 2. When developing the EGR modules, QITABI collaborated with CERD to address feedback on EGR training content. The training addresses the capacity development needs of teachers, DOPS, and CERD in innovative teaching methodologies—e.g., differentiated learning, child-centeredness—in Arabic￾language teaching (KII 9/14). Classroom Libraries QITABI provided classroom libraries to Grades 1 to 4 in Component 1 schools. The classroom libraries include leveled books for students with varying reading abilities and read-aloud books. These libraries aim to enhance student reading attainment and facilitate BLA in classrooms. Teachers appreciate classroom libraries, as they complement the EGR model for improving the quality of Arabic-language teaching. MEHE considers them a need for primary schools (FGD 12/16; KII 4/7). Classroom ICT Equipment By the second quarter of Year 3, QITABI completed the provision of the classroom ICT equipment to Component 1 schools. Classroom ICT equipment is designed to support instruction and learning in the Arabic-language classroom. It addresses school-level ICT needs and is relevant to MEHE’s National Education Technology Strategic Plan (NETSP) and ICT Action Plan. Like classroom libraries, classroom ICT equipment increases the relevance of Component 1 interventions to MEHE’s needs. (KII 9/14). The classroom ICT equipment was installed in Grades 1 through 6 and includes a secured cabinet, an uninterruptible power supply (UPS), speakers, an LCD projector, a projection screen, and a laptop. COMPONENT 2: EXPANDING ACCESS TO SAFE AND RELEVANT EDUCATION FOR VULNERABLE PUBLIC SCHOOL STUDENTS QITABI activities under Component 2 include installing ICT resource rooms in 300 schools, training teachers on the use of ICT in education, building the capacity of DOPS coaches in providing ICT support to teachers, providing ICT training to CERD trainers, and procuring and distributing buses to address barriers to access to education. Results from Component 2 interventions are intended to support Sub-IR 1.1.2 (Expanding access to safe and relevant education for vulnerable public school students). Redevelopment of Component 2 Displaced Syrians, vulnerable Lebanese, and Palestinian refugees number over three million in Lebanon. Nearly half of them are children and adolescents growing up at risk and deprived of basic services, including education.8 QITABI’s initial plan for Component 2 mainly focused on expanding access to education for vulnerable children through non-formal education (NFE). MEHE did not deem NFE a priority for development interventions, as it does not have an NFE framework, without which it cannot oversee and regulate NFE interventions in Lebanon. As a result, MEHE restricted development programs—including QITABI—from focusing on NFE. This contributed to the almost 18-month delay in the implementation of Component 2 activities (KII 5/8). After extensively coordinating and clarifying the education development priorities of the Government of Lebanon, USAID signed an MoU with MEHE, and QITABI replaced NFE with ICT resource rooms, ICT professional development, and transportation/school buses (KII 5/7). These interventions are in line with the QITABI award (e.g., transportation solutions and professional 8 Government of Lebanon and the United Nations (2017). Lebanon Crisis Response Plan (2017–2020) 10 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION development), support MEHE’s needs, and respond to USAID’s CDCS and Education Strategy for expanding access to education for vulnerable children in Lebanon (KII 5/7). They also respond to teachers and principals’ needs for ICT and the improvement of the school learning environment (FGD 6/8). ICT Resource Rooms QITABI provided 400 ICT resource rooms to 300 Component 2 schools. World Learning started the equipment procurement and distribution process for installing ICT resource rooms in schools in the last quarter of Year 2 and completed it in the second quarter of Year 3. The ICT resource rooms include a laptop, an interactive board, Active Inspire software, and Active Expression interactive clickers (wireless response devices) for students. Stakeholders report that MEHE has a high regard for the provision of ICT resource rooms as they respond to MEHE’s needs highlighted in the NETSP and ICT Action Plans (KII 8/10). At the school level, they respond to teachers and students’ needs and enhance the school learning environment and the public school image (FGD 3/8; KII 8/10). Several school principals stated that their schools previously had limited access to ICT equipment but that it was not used before QITABI due to being outdated or broken or because teachers lacked lesson plans that incorporated it (FGD 3/8). Professional Development in ICT The professional development in ICT activity focuses on building DOPS and CERDs capacity in providing ICT coaching and training support to teachers. The professional development of teachers in ICT responds to MEHE’s priorities of including ICT in education in public schools. (KII 6/10). QITABI is in the process of developing the capacity of 61 DOPS coaches who recently received a five-day ICT training as a part of the 20-day total of ICT training that the DOPS coaches will undergo through Component 2 professional development activities. The ICT training for DOPS aims to build the capacity of the coaches in different domains, including the use of active board technology, Microsoft Office, and integration of ICT tools in the teaching and learning process at school. The ICT training also intends to prepare these coaches to sit for Microsoft Office Specialist (MOS) and Microsoft Certified Educator (MCE) certification exams. (Arrangement for taking the certification exams is MEHE’s responsibility.) MEHE and DOPS coaches appreciate the professional development training in ICT because it responds to MEHE’s priorities as referenced in the ICT Action Plan (FGD 1/1; KII 6/10). QITABI provided ICT TOT to CERD trainers in the use of active board technology as well as ICT resource room equipment to eight CERD regional centers. Data from KIIs and CERD FGDs indicate that the provision of the ICT training and ICT resource room equipment to CERD helps it provide quality continuous training in ICT (FGD 1/1; KII 7/10). Transportation/School Buses Under Component 2, QITABI is in the process of purchasing 100 buses to be delivered to 100 schools. School buses were included in Component 2 activities at the request of MEHE to address barriers (transportation) to access to education for (5,000) vulnerable children, which is MEHE’s priority (KII 5/8). COMPONENT 3: STRENGTHENING MANAGEMENT (RESILIENCE) IN THE EDUCATION SYSTEM TO BETTER DIRECT AND MONITOR EDUCATION Component 3 provides support to MEHE and CERD to develop policies/guidelines through desk research, policy workshops, and technical support. In addition, it seeks to strengthen data collection, monitoring, management, and sensitization of MEHE units for informed decision-making. Strengthening management in the education system aligns with MEHE’s needs as highlighted in its ESDP and RACE II. It also supports USAID’s CDCS. In addition, it supports the institutionalization of Component 1 and 2 interventions (KII 6/8). 11 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION OVERALL FINDING KII data from key stakeholders—MEHE, CERD, USAID, and QITABI—and existing MEHE and QITABI documents indicate that QITABI’s interventions were designed and adapted in cooperation with MEHE and address MEHE’s education sector priorities. CONCLUSIONS Component 1 • The elements in the first component of QITABI are aligned with national education strategies to directly respond to the priorities identified by the government and USAID. • The government (MEHE and CERD) recognizes EGRA, EGR, classroom libraries, and the early-grade reading benchmarks as necessary initiatives and believes they respond to an unmet need for a child-centered, individualized approach to Arabic-language teaching. Yet MEHE is concerned that they lack the capacity and resources to expand EGRA. • EGR training is relevant to the needs of administrators, teachers, schools, and students in improving Arabic-language proficiency. Component 2 • QITABI’s initial plan to focus on NFE did not address MEHE’s priorities for development projects. However, QITABI’s flexibility in addressing MEHE’s priorities under Component 2 interventions resulted in an amicable replacement of NFE with the provision of ICT resource rooms, professional development in ICT, and transportation solutions to improve school access and address school-level ICT shortcomings. MEHE considers these interventions highly significant for achieving overall national education priorities. Component 3 • There is a clear link between ESDP, USAID’s CDCS, and RACE II in Component 3 activities to strengthen management by and the capacity of public education sector actors. The relevance of Component 3 activities is also established through their support to the institutionalization of Components 1 and 2 activities. QUESTION 2: EFFECTIVENESS “To what extent is QITABI on track to achieve its objectives—i.e., improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools?” FINDINGS COMPONENT 1: IMPROVING READING OUTCOMES FOR PRIMARY-LEVEL PUBLIC SCHOOL STUDENTS EGRA QITABI designed and implemented EGRA baseline in 240 Cohort 1 and 2 schools, exceeding by 25% its target of assessing 3,840 Grade 2 and 3 students in Arabic reading by the end of Year 2 (see Table 2). Approximately 50% of the total students assessed were girls (KII 13/15; QITABI quarterly reports). The EGRA baseline corresponds to Indicator 1a (Number of students understanding EGRA) and contributes to the IR 1.1.1 (Improved reading outcomes for primary-level public school students). The life of the project (LOP) target is 7,680 students. Table 2: EGRA Baseline, Number of Students Assessed 12 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION The results from the baseline indicate weaknesses in students’ fundamental reading skills in both Grades 2 and 3—approximately 3% of students are proficient in reading fluency in both grades combined. Girls outperform boys by a small but significant margin. 9 The results from the EGRA baseline and endline (scheduled for the next academic year) will provide information about student reading performance and demonstrate the effects of QITABI’s early-grade reading interventions on student reading outcomes over time. To develop national standards for early-grade reading in Arabic language, QITABI designed reading benchmarks. The project collaborated with MEHE, DOPS, and CERD on the EGRA tool as well as on benchmarks development and adapted them to the Lebanese context through meetings and workshops (KII 9/13). According to QITABI quarterly reports, 27 participants attended the EGRA tool development workshop (23 women and 4 men), while 25 participated in the benchmark development workshop (22 women, 3 men). Participants included representatives, technical staff, Arabic-language specialists and trainers from MEHE, DOPS, and CERD as well as Arabic-language teachers from public and private schools. Selection was based on organization affiliation, expertise, and geography. The benchmarks classify student EGRA scores in categories—e.g., beginner, intermediate, proficient, and advanced. Reporting assessment scores by categories or benchmarks aims to allow MEHE to use assessment data, develop reading performance targets in early grades, and track progress toward those targets. QITABI provided training support to DOPS and CERD staff/statisticians to build their capacity in professional analysis software (SPSS) and EGRA data analysis as preparation for MEHE’s deeper participation in analysis of EGRA data. QITABI provided three training workshops to DOPS and CERD (7 women, 4 men). The first focused on using the data analysis software, while the second and third focused on EGRA data analysis. The CERD representatives reported that the SPSS training was a refresher for them (as they have the capacity to use the software) and overall improved their knowledge of evaluation and analysis methods. For DOPS, while the training improved their knowledge of SPSS and EGRA analysis, they still lack the capacity to perform EGRA data analysis without extensive training. Both DOPS and CERD have expressed an interest in additional training and involvement in the next round of data collection and analysis (FGD 2/2; KII 3/3). EGR Training QITABI developed the EGR training modules in collaboration with CERD. The modules focus on the BLA adopted by QITABI to improve reading achievement in Arabic in the 260 project schools. BLA is a set of techniques for teachers to work with students in three key areas: reading, writing, and word work. Implementation was slightly delayed as MEHE and CERD took time to review, provide their input, and approve (KII 9/13). Twenty CERD trainers (16 women, 4 men) received EGR training of trainers (TOT) and, in turn, successfully rolled out the training over three rounds to 1,025 Arabic-language teachers—96% women (KII 8/13) (see Table 3). QITABI has overachieved its Year 2 target for teachers trained (achieved, 639; target, 294) and is progressing to achieve its Year 3 target of training 1,295 teachers. The EGR training corresponds to Indicator 1c (Number of primary school educators who complete professional development training on implementing evidence-based reading instruction with USG [U.S. Government] assistance) and contributes to IR 1.1.1 (Improved reading outcomes for primary-level public school students). Table 3: EGR Training Beneficiaries The QTIABI EGR facilitators, and 38 DOPS coaches who received the EGR training, provide in￾school coaching support to teachers (KII 7/10) with the BLA. The DOPS coaching support includes 9 QITABI (2017). QITABI Cohorts 1 and 2 Combined EGRA Baseline Report. 13 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION classroom observation and feedback for teachers. Many FGDs report that DOPS coaching motivates teachers and provides useful support (FGD 16/24); however, some say that they received limited support (FGD 8/24). DOPS has only 38 counselors for Arabic-language coaching, nine of whom were hired on the recommendation of QITABI. DOPS coaches are stretched to cover over 950 public schools and provide additional support to QITABI schools (KII 6/10; FGD 2/3). Coaching from QITABI EGR facilitators is widely appreciated for providing support to teachers in assessment, lesson planning, preparation, and follow-up (FGD 23/27). In addition, teachers share experiences and resources with fellow teachers in social media circles (Facebook and WhatsApp groups) and seek support from EGR facilitators (FGD 14/18). Teachers appreciate and generally apply BLA methods. They report that the training increased their ability as educators, enhanced their assessment knowledge and skills, and improved their techniques in Arabic-language teaching (FGD 16/18). Both KII and FGD data show that teachers appreciate the child-centered and differentiated learning approaches of BLA and report that they are more innovative in their work and are able to engage on a more individual student–needs basis (KII 8/12; FGD 13/18). Interviews with stakeholders and FGDs with teachers and principals indicate that BLA methods are well received in classrooms (KII 9/12; FGD 16/26). Both teachers and principals during FGDs noted that students have made noticeable learning gains. For example, they are more enthusiastic about Arabic language (FGD 17/26); their vocabulary, spelling, visual, and reading skills have improved, including reading alone or out loud (FGD19/26) and reading comprehension (FGD 11/26). Their motivation to read has also improved (FGD 21/26). In addition, teachers also report that BLA addresses classroom needs, and as a result, students are more engaged, participate more in classroom activities, exhibit improved classroom behavior (e.g., they help each other more) and are more disciplined, particularly boys (FGD 17/18). QITABI recently conducted a midline assessment in Cohort 2 schools to measure if students’ Arabic-language skills began to show improvements after one year of EGR intervention (KII 4/5). The results from the assessment were pending at the time of the evaluation. Although teachers consider student assessment important, some noted that implementing regular assessments consumes too much classroom instruction time (FGD 19/26). They report feeling overwhelmed and find time management challenging due to the workload, overcrowded classrooms (FGD 22/26), and lack of space in the classroom, which constrain full application of BLA in some schools. For example, overcrowded classrooms restrict teachers’ ability to organize students in circles for reading— “circle time” is a BLA activity. (KII 8; FGD 4/26). In addition, teachers and principals report a lack of parental involvement for student follow-up at home (FGD 8/26). Evidence from FGDs with teachers indicates that teachers would like additional training in a range of areas, including additional professional support to address the needs of students with differing learning difficulties (FGD 11/18) as well as additional training for EGR. Many report an interest in watching a “practice class” or sample lessons (FGD 13/18). The EGR intervention also included building teachers’ capacity to assess students’ basic Arabic￾language skills and identify student literacy levels in Grades 1 through 4. The Universal Screening tool developed for this purpose complements BLA and guides teachers to differentiate instructions, develop learning baselines for students, track individual progress and guide responses to student learning needs. Teachers assess student Arabic-language skills in the beginning (pretest) and end of the academic year (posttest). QITABI implemented the pretest in Cohort 1 and 2 schools in the beginning of the current academic year. Teachers assessed a total of 9,321 students from Grades 1 to 4 with support from the EGR facilitators. The posttest was being implemented at the time of this evaluation (KII 7/12). In collaboration with CERD, QITABI developed an early warning system (EWS) for the teachers to identify students at risk of developing reading difficulties and address student needs with support from the school and community. QITABI has not yet been able to implement EWS or create 10 14 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION parent reading circles (PRCs) as planned due to the delay in receiving the list of schools and approval from MEHE. MEHE would like QITABI to pilot EWS and PRCs first before considering scaling up. EWS will be piloted in the beginning of the next academic year (KII 9/12). QITABI’s Year 3 and LOP target for PCRs is five. Formation of PRCs corresponds to Indicator 1d (Number of parent teacher associations [PTAs] or community governance structures engaged in primary or secondary education supported with USG assistance) and contributes to IR 1.1.1 (Improved reading outcomes for primary-level public school students). Classroom Libraries Evidence from KIIs and FGDs indicate that the provision of classroom libraries to Component 1 schools improved the school learning environment and increased student interest in reading and books (KII 6/12; FGD 16/26). Teachers and principals note regular use of libraries in Shift 1 by both Lebanese and non-Lebanese students (FGD 20/26). Students in Shift 2 are also benefiting from these opportunities in some schools, but Shift 2 students do not have access to the libraries in all schools (FGD 4/26). QITABI only targets Shift 1 in anticipation that students in Shift 2 benefit from services and equipment provided through the project. Finally, the leveled books also generate reading interest in students and are helpful to teachers in differentiating instructions according to students’ varying reading abilities (KII 5/12; FGD 12/18). QITABI distributed 163,881 textbooks and other teaching and learning materials to targeted schools, exceeding its Year 2 targets of 114,607 TLM. The provision of libraries corresponds to one of the five key performance indicators (KPIs), KPI 5 (Number of primary or secondary textbooks and other teaching and learning materials provided with USG assistance) and contributes to IR 1.1 (QITABI purpose) (Expanded equitable access and improved learning outcomes for vulnerable students in Lebanese public schools). According to project quarterly reports, QITABI initially had planned to deliver these libraries to all 260 schools during Year 2, but due to delays in getting approvals from MEHE and CERD for the library books, delivery to the 140 schools (Cohort 1 and pilot schools) was completed toward the end of Year 2, while 120 Cohort 2 schools received libraries in the beginning of Year 3. Classroom ICT Equipment KIIs with QITABI and MEHE positively report that the provision of classroom ICT equipment complements BLA, encourages innovation, and contributes to improving the school learning environment and desirability (KII 7/13). Teachers and principals report that ICT has facilitated students’ understanding of teachers’ lessons (FGD 16/26). QITABI provided brief orientation sessions on the use of the equipment to Arabic-language teachers at the time of equipment installation, yet more training is needed. Teachers and principals report that the ICT equipment is not used satisfactorily because of limited training (FGD 12/26). QITABI recently began ICT training for teachers, and it will continue in Year 4. Despite the positive reports overall regarding ICT use, some schools lack ICT equipment and libraries for safety and for maintaining materials, constraining access for Shift 2 schools (FGD 9/26). In addition, some teachers report that they lack support from principals for acquiring materials (FGD 4/26). In general, teachers and principals believe that schools need more teaching and learning materials (FGD 14/26) as well as complementary materials for ICT equipment—e.g., CDs (FGD 10/26). To complement BLA and support the use of ICT in education, QITABI is developing e-content for Arabic-language teaching (under development at the time of this evaluation). The provision of e￾content will include digital versions of classroom library books. QITABI provided classroom ICT equipment to 260 schools, thus meeting its Year 3 and LOP targets. It responds to Indicator 1b (Number of schools/learning spaces that receive in-classroom ICT equipment to improve reading outcomes) and contributes to IR 1.1.1 (Improved reading outcomes for primary-level public school students). In addition, Component 1 interventions correspond to KPI 2 (Number of schools targeted with USG assistance [Assistance to Improve Reading Outcomes]) and 15 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION contributes to IR1.1 (QITABI purpose) (Expanded equitable access and improved learning outcomes for vulnerable students in Lebanese public schools). The LOP target is 477 for Components 1 and 2, following the removal of 83 duplicate schools. COMPONENT 2: EXPANDING ACCESS TO SAFE AND RELEVANT EDUCATION FOR VULNERABLE PUBLIC SCHOOL STUDENTS ICT Resource Room Evidence from KIIs and principal FGDs indicate that students, teachers, and schools appreciate the provision of ICT materials through interactive resource rooms, which offer technology that is not normally accessible to teachers and students in or outside of schools. These rooms are beneficial to the wider school population; offer a variety of subjects, including Arabic language; and provide equal access to girls and boys (KII 6/10; FGD 6/8). QITABI provided 400 ICT resource rooms to 300 Component 2 schools, achieving its Year 3 and LOP target. It corresponds to Indicator 2c (Number of schools/learning spaces that are equipped with ICT resource rooms to improve learning environment) and contributes to IR 1.1.2 (Expanded access to safe and relevant education for vulnerable public school students). Principals and some KII respondents report that school learning environments and reputations are improving as a result of the addition of resource rooms. These rooms encourage and facilitate more interactive and creative teaching methods that benefit both students and teachers. Principals report improvement in students’ engagement in the classroom with the incorporation of active boards (FGD 7/8). Students show overall increased interest in learning through the use of ICT and improved overall performance (FGD 4/8). As noted above, principals report that teachers need training to be able to most effectively use ICT resource rooms (FGD 7/8). On Indicator 2.2, which measures the number of schools showing improved learning environments as a result of ICT-in￾education interventions, QITABI will report by the end of Year 4. Some KII respondents reported that assessing the effects of resource rooms on the improvement of school learning environments may be challenging because the remaining project time is limited (KII 5/13). Some schools have weak physical infrastructure, in which the application, installation, and short- and long-term use of ICT materials are challenging—e.g., lack of electricity, Internet, and space (KII 3/13). School principals also have maintenance and security concerns about the ICT equipment relating to its use by students in Shift 2 (FGD 6/8). Others report overcrowded conditions, where overworked teachers manage a high number of students and equipment can easily be overused—teachers fear that frequent use among a higher number of students will wear out the equipment earlier than would happen otherwise (FGD 5/8). Professional Development in ICT Sixty-one DOPS coaches (35 women, 26 men) received a five-day ICT training (part of 20 days of training total), including in Microsoft applications—e.g., Word, PowerPoint. MEHE selected these coaches for Component 2 ICT intervention based on the assessment of their ICT knowledge and skills (KII 4/7). DOPS coaches report that the training increased their knowledge of ICT. They also expressed support for ICT resource rooms and for providing continuous coaching and follow-up for teachers’ use, yet they request additional training to be able to provide ICT coaching support to schools effectively (KII 5/12; FGD 1/1). DOPS will receive further ICT training during the final year of the project. Senior MEHE officials report keen interest in DOPS capacity-building in ICT. They would like intensive training for a small group of DOPS coaches (e.g., five coaches) to develop a core senior ICT team within DOPS that can provide ICT training and engage in peer-to-peer capacity development in ICT for DOPS (KII 2/7). Fifty CERD trainers (27 women, 23 men) have received training on the ICT software, Active Inspire. Although the trainers had prior knowledge of ICT, including active boards, they report that the training provided useful information (FGD 1/2). CERD has not yet trained any teachers in ICT under the QITABI project; they will start from the coming 16 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION academic year (KII 4/7). QITABI is also collaborating with CERD on the development of an online platform, Moodle, that will serve as a repository of all training materials developed under QITABI, including EGR and ICT training, for CERD’s future use and adaptation (KII 5/7). CERD reports increased commitment after the receipt of ICT resource room equipment at eight regional CERD centers. KIIs with CERD and QITABI indicate that having the ICT resource room equipment at the training centers enables CERD to provide practical and comprehensive training on equipment with which they now have firsthand experience (KII 5/7). Providing ICT-in-education training/coaching to teachers corresponds to Indicator 2b (Number of teachers receiving ICT-in-education training/coaching to improve learning environment) and contributes to IR 1.1.2 (Expanded access to safe and relevant education for vulnerable public school students). Teacher training in ICT is ongoing, and data on the number of teachers trained was not available at the time of this evaluation. According to QITABI’s approved M&E plan (Year 3), progress on this indicator will be reported at the end of Year 3. QITABI’s target for Year 3 and LOP is 840 teachers. Transportation/School Buses The key intervention under Component 2 to address barriers to accessing education for vulnerable children is the provision of school buses to 100 public schools. To purchase the 100 buses, QITABI is awaiting a waiver from USAID for non-US manufactured vehicles, after which it will initiate full procurement. MEHE selected 100 schools located in remote rural areas; over 50% of the selected schools operate in two shifts. This is in contrast to the 70% requested by QITABI (KII 7/10). Providing transportation (buses) to schools is new for MEHE; no school transport regulations existed. QITABI worked with MEHE to develop and issue regulations to set basic guidelines for MEHE with regard to owning and operating buses. In addition, QITABI developed the operation and maintenance (O&M) plan in coordination with MEHE to highlight necessary steps and name those accountable for carrying them out in order to ensure operations, maintenance, and safety during the life of the project. QITABI will use the O&M plan as a reference to develop an operating manual/guide in Arabic, which will be supplied to the relevant schools in the induction program before delivery of the buses (KII 5/10; O&M Plan). The procurement process was delayed for almost a year due to selecting schools, developing the regulations for the buses and an O&M plan, realigning the QITABI budget, modifying language for transportation methods, and obtaining a value-added tax (VAT) exemption. Data from KIIs with QITABI, USAID, and MEHE indicate that completing the procurement and delivery of buses will likely take another year (KII 7/10). After schools receive the buses, limited project time will remain for QITABI to assess their use and adherence to the O&M plan in order to determine the effectiveness in improving access to education for vulnerable children (KII 5/10). According to school principals, transportation solutions/buses support school access for in school vulnerable children who cannot attend with regularity due to lack of free transportation, and out-of￾school vulnerable children who are unable to enroll (FGDs 7/8). KIIs with MEHE officials and QITABI report that buses will benefit approximately 5,000 students per academic year in QITABI schools who live in rural areas or have physical disabilities (FGD 7/8; KII 4/7). Non-Formal Education (NFE) NFE was removed as a direct activity under QITABI at the request of MEHE. However, the QITABI partner, Ana Aqra’a, does use mobile classrooms in Akkar governorate as a cost-sharing activity. This activity addresses the needs of out-of-school students near refugee camps to prepare them for re-entering the formal education system (KII 3/6). Providing buses to schools corresponds to Indicator 2.1 (Number of vulnerable students gaining access to education) and contributes to IR 1.1.2 (Expanded access to safe and relevant education for vulnerable public school students). QITABI’s Year 4 target is 5,060 and the LOP target is 5,180. QITABI is reporting on 182 for Year 2 mainly resulting from the restricted NFE activity. In addition, Component 2 interventions support KPI 2 (Number of schools targeted with USG assistance [Assistance 17 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION to Expand Access]) and contributes to IR1.1 (QITABI purpose) (Expanded equitable access and improved learning outcomes for vulnerable students in Lebanese public schools). The LOP target is 477 for Components 1 and 2, following the removal of 83 duplicate schools. COMPONENT 3: STRENGTHENING MANAGEMENT (RESILIENCE) IN EDUCATION SYSTEM TO BETTER DIRECT AND MONITOR EDUCATION Component 3 is a culmination of the lessons learned and achievements to date of Components 1 and 2 and focuses on developing a model of institutional support to MEHE and CERD for the sustainability of key project interventions. Under Component 3, QITABI is providing technical support to MEHE and CERD both directly and by seconding consultants to these entities. The consultants support MEHE and CERD in reviewing project material, facilitating input on all project materials and tools from MEHE and CERD, and synthesizing project reports, such as EGRA. They also support QITABI in desk reviews of CERD and MEHE systems and bylaws. In addition, Component 3 works with MEHE and CERD by advocating for QITABI interventions and the tools (assessment, logbooks), guides, manuals, and reports/data and by promoting ways in which the tools and other materials can contribute to improving MEHE education systems. Since Component 3 capitalizes on Component 1 and 2 outputs, work under this component accelerated in Year 2 (KII 3/7). Although some MEHE and CERD officials generally have positive impressions of the technical support through consultants, reporting that assistance is useful to help review and provide feedback on QITABI reports and tools, others perceive that the assistance (e.g., writing a circular) is more administrative than technical (KII 4/7). The General Director of MEHE requested to be and eventually became the main focal point for coordination between QITABI and MEHE for Component 3. Based on QITABI recommendations, the General Director reinitiated the Arabic Language Committees (ALCs), consisting of principals, headmasters, and Arabic teachers. These committees are required to support improvement in Arabic-language teaching in their respective schools. QITABI plans to mobilize ALCs’ support for BLA in Year 4 (KII 4/7). A number of tools have been created under QITABI including EGRA, Universal Screening, classroom observation, and ICT logbooks to monitor classroom ICT use. Currently, QITABI is working with the Director of Basic Education at MEHE to support developing a basic M&E system for the directorate to collect data from the schools about QITABI interventions. Data generated by the system will help QITABI present evidence-based policy recommendations to MEHE as well as inform decision-making at MEHE (KII 5/7). The result from this activity will contribute to Indicator 3.1 (Number of new management and monitoring systems, procedures, and tools recommended and/or introduced to [MEHE/DOPS and CERD] as a result of USG assistance). Finally, not all reports were positive regarding these management and monitoring initiatives. Several key stakeholders (from MEHE and CERD) expressed concerns that they have limited knowledge, clarity, or understanding of Component 3 activities or of the scope and purpose of this component. They report that they do not have a clear idea what the structure, vision, or expected outputs are for Component 3 (KII 3/7). QITABI trained 52 DOPS and CERD representatives, overachieving its target of 35. The target for Year 3 is 77 and 91 for the LOP; QITABI has already trained nine toward its Year 3 target. Similarly, the target for implementing standardized learning assessments is two for Year 2, which the project achieved. These Component 3 outputs are covered under Components 1 and 2 and correspond, respectively, to Indicators 3.3 (Number of education administrators and officials who complete professional development activities with USG assistance) and 3c (Number of standardized learning assessments supported by USG). 18 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Cross-Cutting Themes The project has not done any gender assessment of the content of the books included in the classroom libraries. However, KII and FGD participants did not see gender as a concern for project activities. They report that boys and girls both equally benefit from project activities, including BLA, classroom libraries, and ICT equipment (KII 14/21; FGD 22/27). Similarly, both Lebanese and Syrian students equally benefit from the EGR interventions, classroom resources, and ICT resource rooms during Shift 1 (QITABI does not focus on Shift 2) (KII 4/13; FGD 15/27). Overall, MEHE and CERD appreciate QITABI’s approach to working collaboratively and the project’s contributions to the public education sector. They admire QITABI’s leadership and management and have a favorable impression of QITABI’s team (KII 5/7). CONCLUSIONS • Overall, QITABI’s leadership, strong project management, and approach to working collaboratively with MEHE and CERD have contributed to the project’s progress thus far and to building strong relationships with key stakeholders. The results indicate that the project is on track to achieve its objectives—i.e., improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools. This is especially true for Component 1 and 3 interventions. While QITABI is on track to achieve Component 2 results associated with providing the ICT resource rooms and professional development in ICT, the significant delay in the delivery of the school buses will likely hinder the project’s ability to assess the effects of the buses within the current project life. Component 1 • QITABI successfully designed and implemented EGRA for the first time in Lebanon, introducing a framework for reading assessment and providing key information on student early-grade reading skills as a baseline. The reading benchmarks, if adopted by MEHE, will classify student progress and help trace improvements in learning outcomes over time. QITABI’s approach to collaboration with its government counterparts allowed it to successfully implement the EGRA baseline and develop reading benchmarks. • The professional analysis software and EGRA analysis training were beneficial to both DOPS and CERD; however, they lack the capacity to analyze EGRA data without external support. DOPS in particular needs extensive training to be able to analyze assessment data independently. • The BLA is a replicable model that, with ongoing coaching and teacher monitoring, contributes to improved outcomes for both students and teachers in early-grade Arabic teaching and reading. • The EGR coaching model is critical to support teachers and is key to effective implementation of BLA in the classroom. The EGR training was effective at improving the teaching skills of Arabic-language teachers in innovation, child-centeredness, and differentiated instruction for students of varying reading skills. However, on-the-ground challenges such as overcrowded classrooms can limit the efficacy of BLA. In addition, teachers need further training (refreshers) in the practical application of BLA in classroom settings, and DOPS faces human resources constraints that will limit its support. • The classroom-level effects of EGR training are positive—students’ fundamental skills in Arabic-language reading, level of interest, and relationships with teachers and other students have improved. The reported improvement in student reading achievement could not be verified with the midline assessment results as those were pending at the time of this evaluation. • Classroom libraries and ICT are welcome additions to the school environment and contribute positively to teacher innovation and student engagement, especially for students with varying levels of reading ability and for those with limited access to ICT. However, access is often at the discretion of principals and limited almost exclusively to Shift 1 students, regardless of nationality. 19 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Component 2 • The ICT resource rooms have a positive effect on teaching, student learning, engagement in classroom activities, and the school learning environment, but infrastructure shortfalls challenge overall efficacy in some schools. Providing ICT training to DOPS and CERD ensures their professional development and continued ability to serve as mentors to teachers for ICT use. • Component 2 activities started significantly late owing to the time that MEHE took to sign the MoU. The delay in procuring and delivering buses has further delayed QITABI’s plans to expand access to education for vulnerable children. The buses will likely be delivered to schools next year, only months before the end of the project. This will make it challenging for QITABI to provide post-delivery support to schools and to MEHE on the effective use and operation of the buses, monitoring progress, and measuring and reporting effects. • The fact that MEHE issued regulations for the buses and that QITABI developed the O&M plan is a significant achievement. The O&M plan guides effective use, operation, and maintenance of the buses. Component 3 • Key stakeholders have a limited understanding of the scope of Component 3, its activities, and/or how they contribute to QITABI’s objectives. While coordinating directly with the General Director for Component 3 can potentially generate swift decision-making and implementation, it also likely adds to the lack of understanding of the component for key MEHE officials. • The response from MEHE regarding the technical support is mixed. The single tangible output of Component 3 (independent of Component 1 and 2 outputs) thus far is a circular issued by the General Director to reactivate the ALCs. While reactivating the ALCs is important, follow-up from MEHE and capacity-building of these committees will be necessary to draw on their support to QITABI activities. • QITABI’s current engagement under Component 3, with the Director of Basic Education to develop a basic M&E system to monitor QITABI, has the potential to bear positive results. This will allow MEHE not only to use the tools that QITABI has developed, but also to appreciate their usefulness in improving learning and monitoring progress. Cross-Cutting • The evaluation team could neither speak to students about nor observe the use of classroom libraries, thus the team is unable to assess whether the content of the books takes into account gender considerations and what effect the inclusion/exclusion of gender considerations may have had on the intended results. QUESTION 3: SUSTAINABILITY “To what extent are the project activities and results likely to continue after the project ends?” FINDINGS COMPONENT 1: IMPROVING READING OUTCOMES FOR PRIMARY-LEVEL PUBLIC SCHOOL STUDENTS EGRA MEHE and CERD have the EGRA tool, benchmarks, and EGR modules to continue implementing (KII 10/13). CERD has formally expressed interest in scaling up EGRA and adopting the reading benchmarks. MEHE recognizes their value but is not ready to support EGRA scale-up because of a lack of capacity and resources (KII 3/3). DOPS and CERD need further capacity-building to 20 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION implement EGRA. They have shown interest in more SPSS and EGRA analysis training and involvement in the next round of research (KII 5/9; FGD 2/2). EGR Training Both DOPS and CERD have been trained in EGR. CERD appreciates the BLA and its value to improving the teaching of Arabic language and has already included EGR training in its future plans (KII 6/9). Teachers are willing to continue implementing BLA in the classroom with support from DOPS and EGR facilitators (FGD 14/18). However, overcrowded classrooms, limited classroom space, and limited TLM are challenges to the effective implementation and sustainability of BLA activities (FGD 11/18). Classroom Libraries Given the positive effects of the classroom libraries on teaching, teachers expressed strong interest in continuing to use them. Many report, however, that overcrowded classrooms constrain continued maintenance because of overuse—i.e., the frequency with which a book is used in a class increases significantly if the class is overcrowded, thus books will likely wear out sooner than expected (FGD 12/18). Classroom ICT Equipment Although the ICT equipment is under warranty for up to three years, reports from teacher FGDs indicate that without training, the use of ICT in the continued implementation of BLA will be limited. QITABI plans to provide ICT training to teachers in Year 4 (KII 3/5; FGD 17/18). COMPONENT 2: EXPANDING ACCESS TO SAFE AND RELEVANT EDUCATION FOR VULNERABLE PUBLIC SCHOOL STUDENTS ICT Resource Rooms According to the FGDs with principals, schools value the provision of the ICT resource rooms and express a willingness to continue using them (FGD 6/8). All ICT equipment provided to schools was selected on the basis of its low cost, low likelihood of becoming obsolete in the near future, and ease of use over time. In addition to the warranty, most schools have an ICT point person who serves as a key resource for troubleshooting and maintenance of equipment. However, this person’s longevity as a school resource depends on allocation of funds from the school budget (KII 3/5). Professional Development in ICT QITABI provides ICT training to enable DOPS to coach teachers. DOPs coaches report that they will need the remaining ICT training to be able to provide effective coaching to schools during and after the life of the project, though they need more capacity-building support (KII 7/9). In contrast, CERD training centers have ICT resource room equipment, their trainers have been trained, and QITABI plans to provide an online training repository. CERD plans to offer ICT training as part of its regular program during and after the project ends (KII 7/10). Transportation and School Buses QITABI successfully developed the O&M plan for the school buses, thus providing a structured framework for MEHE to oversee the buses and ensure they remain fully functional after the project ends. (The plan includes the school transportation regulation issued by MEHE.) However, insufficient time will remain after the delivery of the buses to schools and before the end of the project for QITABI to support MEHE to implement the plan (KII 7/10). 21 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION COMPONONT 3: STRENGTHENING MANAGEMENT (RESILIENCE) IN EDUCATION SYSTEMS TO BETTER DIRECT AND MONITOR EDUCATION. The Arabic Language Committees have been reactivated under Component 3 to provide support to Arabic-language teaching and EGR interventions in schools. Some stakeholders expressed a lack of clarity about ALCs’ usefulness—e.g., how the project will mobilize their support for the project activities—and about the prospect for their steady operations in QITABI and non-QITABI schools once the project ends (KII 3/5). Note: Standardized assessments (EGRA) and professional development of MEHE and CERD officials are Component 3 outputs but produced under Components 1 and 2. Thus the prospects of their sustainability are discussed above under those components. CONCLUSIONS Overall, many program elements in Components 1 and 2 are likely to continue beyond the life of the project. For Component 1, these include EGRA and its reading benchmarks, CERD EGR trainings, and the improvements in Arabic reading skills. However, sustainability is highly dependent on strong support from MEHE, capacity-building for DOPS, and increased financial resources. In addition, overcrowding and limited TLM threaten the achievements made in reading skills and the use of classroom libraries. For Component 2, program elements that are likely to continue include the maintenance and use of ICT rooms, the ICT training that CERD offers, and MEHE’s operation and maintenance of the school buses. However, the sustainability of these initiatives is dependent, respectively, on DOPS capacity to provide continued support to ICT and on having sufficient time after the delivery of the buses to ensure effective implementation of the plan. For Component 3, sustaining QITABI’s support to management systems is subject to the successful institutionalization of its key interventions—e.g., standardized assessments, EGR and ICT training, and management and monitoring systems. Sustaining effective functioning of the ALCs is subject to building their capacity and mobilizing their support for QITABI activities. 22 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION RECOMMENDATIONS FOR FUTURE EDUCATION PROGRAMMING 1. Extend QITABI for at least one more year. With this extra time monitor and assess Component 2 activities that have not yet fully matured. Determine from this monitoring which Component 2 activities should be supported that maximize improved reading outcomes. New activities should not be introduced during the extension period. 2. USAID should advise QITABI to conduct a gender assessment of the content employed by the project (e.g., classroom library books, classroom observation tools). USAID should use findings from the ongoing gender assessment of the education sector to inform the design of the next education project and require implementing partners to more explicitly address gender considerations in project interventions. This should include any content and tools that the future project develops. 3. There is a continuing need to improve Arabic reading outcomes in Lebanon. Future project design plans (PDP) in the education sector should consider activities aimed at continuing improvement in Arabic reading outcomes, based on the findings of this evaluation. Specific future activities to consider are: d. Funding the continuation of initiatives to support Arabic reading skills; e. Short term capacity-building for CERD and MEHE in analyzing standardized assessment data; f. Supporting DOPS and CERD in implementation of EGRA as an assessment and institutional and learning tool. 4. Because the evaluation found that the training of trainers (TOT) model is effective and sustainable, future activities that include teacher training would benefit from incorporating the TOT model in their design to promote CERD and DOPS capacity-building, ownership, and sustainability. QITABI 1. To obtain MEHE’s endorsement of the reading benchmarks and EGRA expansion, QITABI should develop an advocacy campaign with CERD input. This may include a cost-benefit analysis of EGRA and a needs assessment for CERD and MEHE to identify capacity gaps and required resources as well as to facilitate decision-making. In addition, QITABI should consider providing capacity-building to MEHE and CERD to implement EGRA and analyze the assessment data. QITABI should also consider providing refresher EGR training to teachers during the final year of the project that include practical applications of the BLA, allowing trainees to observe sample lessons in overcrowded classroom settings. 2. To allow adequate time for schools to benefit from technology, QITABI should prioritize and expedite the provision of e-content as well as complementary ICT materials (CDs). Adding the ongoing training of teachers in ICT to these program elements can potentially bolster the use of the equipment and maximize the benefits of QITABI. 3. Capitalizing on the expressed interest of DOPS in ICT and the critical role coaching could play in its work, QITABI should consider providing DOPS with capacity development to build a core group of senior ICT coaches. Similarly, the development and handing over of the online training platform (Moodle) to CERD should include training of CERD officials in its operation, maintenance, and expansion. To complement DOPS support and improve ICT use in schools, the project should encourage schools to participate in CERD ICT training beginning in the coming academic year. 23 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 4. To measure the effects of school buses on access, QITABI should consider conducting an assessment of enrollment and attendance in the selected schools to establish a baseline before delivering the buses to schools. A similar assessment later during the academic year will provide data to compare with baseline to measure increases both in enrollment and attendance. This assessment will be subject to project extension. 5. To help achieve the expected outcomes for Component 3 by the end of the project, QITABI should develop a detailed and actionable plan, with input from USAID, MEHE, and CERD, with clear activity descriptions, implementation strategies, and pathways. The plan should delineate how the component will work toward achieving these targets, what resources it will need, and the kind of support it can expect from MEHE, CERD, and USAID. To increase commitment and support, QITABI should educate MEHE and CERD officials about the component in order to increase their understanding of its objectives, activities, and expectations. 6. QITABI should work closely with MEHE—and if necessary request USAID assistance—obtain its support to: e. Pilot EWS and PRCs as early as the first quarter of the coming academic year (2017- 2018). f. Advocate for and support MEHE to direct schools to allow access to classroom libraries and ICT equipment to Shift 2 students. g. Ensure that the ALCs function effectively during and after the project h. Complete the development of the M&E system for the basic education directorate to ensure its use after QITABI ends. 7. Classroom libraries eventually wear out and replacements are necessary. Similarly, ICT equipment provided to schools will need maintenance and/or replacement once the warranty runs out. QITABI should consider providing technical assistance to MEHE so that the Ministry can determine allocation of its own resources and/or mobilize external resources to maintain classroom libraries and ICT equipment in QITABI schools to ensure that children have continued access to these materials and benefit from them. These schools should also be directed to maintain and safeguard learning materials and equipment. 8. Given CERD’s expressed interest in scaling up EGR, QITABI should closely work with CERD to develop a clear strategy outlining how CERD plans to offer EGR and ICT training to schools after QITABI ends and identify technical support (further training, technical consultants) for CERD in order to implement these plans. Furthermore, the strategy may include ways in which CERD could support scaling up EGR, including EGRA, in the future and the kind of resources it might require to do so. 24 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ANNEXES Annex 1: Inception Report Annex 2: Documents Reviewed Annex 3: Data Collection Instruments Annex 4: QITABI Intended Results/Scope of Work 25 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ANNEX I: INCEPTION REPORT QUALITY INSTRUCTION TOWARDS ACCESS AND BASIC EDUCATION IMPROVEMENT (QITABI) Mid-term Performance Evaluation Inception Report May 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Social Impact, Inc. 26 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION QUALITY INSTRUCTION TOWARDS ACCESS AND BASIC EDUCATION IMPROVEMENT (QITABI) Mid-term Performance Evaluation Inception Report This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from: DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 27 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract/Order No. AID-268-C-15-00001 28 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ACRONYMS ADS Automated Directives System AMIDEAST American-Mideast Educational and Training Services, Inc. CDCS Country Development Cooperation Strategy CERD Center for Educational and Research COP Chief of Party COR Contract Officer Representative DEC Development Experience Clearinghouse DOPS Department d’Observation Pedagogique Scolaire/ Guidance and Counseling Department EQUI Evaluation Quality Use and Impact ET Evaluation Team EGRA Early Grade Reading Assessment EWS Early Warning System FGD Focus Group Discussion GOL Government of Lebanon ICT Information Communication Technology IPs Implementing Partners IR Intermediate Result LOE Level of Effort KII Key Informant Interview M&E Monitoring & Evaluation MEHE Ministry of Education and Higher Education MSI Management Systems International NFE Non Formal Education PMP Performance Management Plan PMPS Performance Management and Support Program for Lebanon SOW Statement of Work TOT Training of Trainers USAID United States Agency for International Development WL World Learning 29 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION TABLE OF CONTENTS EXECUTIVE SUMMARY................................................................................................ 30 INTRODUCTION........................................................................................................... 30 PROJECT BACKGROUND............................................................................................ 31 DEVELOPMENT HYPOTHESIS ................................................................................... 33 PURPOSE AND INTENDED USE ................................................................................ 34 INTENDED AUDIENCE ................................................................................................ 34 EVALUATION DESIGN AND METHODS.................................................................. 35 EVALUATION MANAGEMENT ................................................................................... 43 30 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION EXECUTIVE SUMMARY The United States Agency for International Development (USAID) awarded the nationally focused Quality Instruction Towards Access and Basic Education Improvement (QITABI or “My Book” in Arabic) activity to World Learning and its partners on September 30, 2014. QITABI, implemented through September 29, 2018 by World Learning and its partners, is an education project that aims to improve reading outcomes for public primary school students, expand access to quality education for vulnerable children, and strengthen management and resilience within the public education system. QITABI is also addressing barriers to accessing education and building the capacity of the Ministry of Education and Higher Education (MEHE) to sustain reforms. This mid-term performance evaluation will assess QITABI activities to-date for their relevance, effectiveness, and sustainability. The evaluation will provide pertinent information, lessons learned and recommendations that will assist the primary intended users—USAID and QITABI partners—to implement current course corrections where necessary and inform future education programming in Lebanon. Social Impact, as the prime implementer of USAID’s Performance Monitoring Support Project for USAID Lebanon, will be conducting this mid-term performance evaluation. The evaluation team has worked with USAID to agree on the following evaluation questions: 1) Relevance: How relevant is the QITABI project to the needs of the public basic education sector in Lebanon? 2) Effectiveness: To what extent is QITABI on track to achieve its objectives—i.e., improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools? 3) Sustainability: To what extent are the project activities and results likely to continue after the project ends? The evaluation will mainly draw on data collected through key informant interviews (KIIs) and focus groups discussions (FGDs). Other sources of data will include the project documents and secondary documents identified during the desk review, KIIs and FGDs. Most of the data will be qualitative, though where possible, the evaluation team will also draw on the project’s quantitative monitoring and evaluation data as well as that received or available from MEHE. Key informants will primarily be project, USAID and MEHE staff, while focus groups will consist mainly of direct beneficiaries – principals and teachers – but also trainers and coaches from Center for Education Research and Development (CERD) and MEHE’s Guidance and Counseling Department (DOPS). Following fieldwork, the evaluation team will systematically code and analyze the data to answer the evaluation questions, prepare a presentation for USAID sharing preliminary findings, and produce a draft and final report with findings and recommendations. INTRODUCTION The USAID-funded, nationally focused Quality Instruction Towards Access and Basic Education Improvement (QITABI or “My Book” in Arabic) activity was awarded to World Learning and its partners on September 30, 2014, and is implemented through September 29, 2018. This $41.2 million education initiative aims to improve reading outcomes for public primary school students, expand access to quality education for vulnerable children, and strengthen 31 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION management and resilience within the public education system. QITABI has already conducted the first Early Grade Reading Assessment (EGRA) in Lebanon to measure the reading fluency of young students in Arabic. Assessment results will inform subsequent activities, which will also focus on the public schools that are most affected by the Syrian refugee crisis. In addition, QITABI is addressing barriers to accessing education and building the capacity of the Ministry of Education and Higher Education (MEHE) to sustain reforms. As QITABI evolves past its halfway point, USAID requested a mid-term performance evaluation to assess QITABI activities for their relevance, effectiveness, and sustainability. USAID and QITABI partners will be the primary users of the evaluation. The findings will inform future USAID and national education programming in Lebanon and help determine how to better implement quality programming more quickly. World Learning will use the findings and recommendations while planning for the project activities in the remaining life of the project. Secondary users of the evaluation include MEHE, CERD and DOPS, as the findings will provide them feedback on the effectiveness of the activities and highlight how they can more effectively support public primary schools. Social Impact, as the prime implementer of the Performance Monitoring Support Project for USAID Lebanon, will be conducting the mid-term performance evaluation. This inception report lays out the plan that the evaluation team will follow to answer evaluation questions. The report first describes the project’s background, next the evaluation’s purpose and intended use and users, the evaluation questions, and finally the evaluation methodology. The evaluation will be conducted within the context of SI’s Evaluation Quality Use and Impact (EQUI™) approach, processes, and protocols. EQUI processes are designed to achieve a quality evaluation that incorporates learning and utilization. The evaluation will highlight lessons learned and provide recommendations, taking into consideration programming of future basic education projects in Lebanon. Future efforts will improve student learning outcomes in public primary schools, building upon the successes and lessons learned from USAID’s body of prior and current work in the sector. To support the take up of findings, this evaluation will include a post-evaluation action plan. Aligned with this mission, SI will continue to actively engage USAID, implementing partners (IPs), and other stakeholders in the evaluation process. PROJECT BACKGROUND The USAID-Funded, nationally focused QITABI aims to expand equitable access and improve learning outcomes for vulnerable students in Lebanese public schools. Basic education initiatives have recently taken on increased importance as the public education system has been strained by the continuing inflow of Syrian refugee children into Lebanese public schools. This strain has been felt not only by students, but also teachers, principals, parents and the Ministry for Higher Education and Education (MEHE). Emphasis has shifted from access to education alone to a focus on quality education that is not only more inclusive of vulnerable children, but also provides superior instruction in language and literacy, namely reading skill improvement. Schools should aim to provide professional development and training opportunities to teachers to improve their curriculum quality and to equip them to manage their classrooms and their student learning outcomes more effectively. QITABI’s overarching goal is to support MEHE to improve the quality of learning environments for public school students as well as vulnerable and out of school children including refugees, and develop the professional capacity of teachers to provide comprehensive instruction. It addresses the underlying needs for teacher professional development, reading skills and 32 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION curriculum quality. QITABI promotes achievement of the Development Objective “Improved capacity of the public sector in providing transparent, quality services across Lebanon,” and of the corresponding Intermediate Result 1.1 “Expanded equitable access and improved learning outcomes for vulnerable students in Lebanese public schools.” The project is also responsive to USAID’s Education Strategy 2011-2015 Goals 1 “Improved reading skills for 100 million children in primary grades by 2015” and 3 “Increased equitable access for 15 million learners in environments affected by crises and conflict by 2015.” The project provides technical assistance and capacity-building within the context of three project components (sub-Intermediate Results [sub-IRs] in the USAID Mission’s draft Country Development Cooperation Strategy (CDCS 2014-2019) focused on: - Sub-IR 1.1.1: Improving reading outcomes for primary level public school students o Component 1: 1. Institute the use of an early grade assessment to evaluate the reading skills of young learners in 260 primary schools 2. Identify struggling students through an early warning system (EWS) 3. Develop teacher training modules and classroom materials, train CERD and DOPS master trainers, and improve early grade reading (EGR) implemented in targeted schools 4. Provide technical assistance to MEHE and CERD in Improving EGR curriculum, providing teaching methods and materials including EGR training modules and supplementary early grade educational materials. - Sub-IR 1.1.2: Expanding access to safe and relevant education for vulnerable public school students o Component 2: 1. Provision of evidence-based policy research, analysis and recommendations for improving access to quality education 2. Equip up to 300 schools with ICT resource rooms to improve the learning environments in the schools 3. Conduct training on ICT strategy and professional development for DOPS staff and the schools. 4. Address the transportation barrier to school attendance, often faced by the vulnerable student population, through procurement of up to 100 school buses. - Sub-IR 1.1.3: Strengthening management (resilience) in education system to better direct and monitor education o Component 3: 1. Reach out of school youth with a bus or mobile bus program to conduct basic literacy and numeracy classes and perform outreach in rural and remote areas 2. Collect and analyze data at the school-level and sensitization of MEHE units and departments on evidence-based decision-making. 3. Strengthen management in the education system to better directly monitor education by engaging stakeholders at central, regional and school levels to use data for informed policy, and conduct policy workshops. 33 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION World Learning is the lead implementing agency and overall cooperative agreement grantee for QITABI. World Learning manages the coordination among MEHE, CERD, DOPS and other stakeholders. In addition, it regularly communicates and collaborates with QITABI partners and service providers to ensure smooth implementation. Under project activities, along with providing management and leadership of all project activities, World Learning also leads components 1, 2 and 3, oversees the procurement of ICT equipment to support classroom instruction, and supports the setup of ICT resource rooms and the provision of school buses. World Learning works with three other partners to implement project activities: 1) Management Systems International (MSI) Under Component 1, MSI leads the EGRA (student assessments), teacher surveys, establishes Arabic-language reading benchmarks for Grades 1&2, and supports capacity building for MEHE representatives. 2) Ana Aqra’a’ Under Component 1, Ana Aqra’a supports classroom observation, EWS, teacher training on reading instruction, material development, teacher coaching in EGR, training for CERD and DOPS, parental awareness and non-formal education (NFE) through the mobile bus initiative. 3) American-Mideast Educational and Training Services, Inc. (AMIDEAST) Under Component 2, AMIDEAST provides professional development to support implementation of ICT in schools and through educational activities. DEVELOPMENT HYPOTHESIS QITABI’s design emphasizes improving both access and learning outcomes, and institutionalizing upgrades across the sector including at the teacher, administrative, and ministerial levels. As a result of project interventions—assessments, teacher training, coaching, classroom materials— learning outcomes will improve, particularly in reading at the primary level. Better management and higher instructional standards will be institutionalized and sustainable. Improved facilities, services, and teaching – along with more effectively engaged communities and school councils – are expected to improve learning but also increase the chances of youth remaining in the formal education system. Special efforts, particularly under the “learning environment” component, focus on issues that have arisen as a result of the influx of Syrian refugee children into the public education system, e.g., over-crowding, psychosocial challenges, and teacher overload. The QITABI approach engages stakeholders both at school and central level. At the school level, QITABI’s student-centered model is expected to improve access to education and improve early grade reading through data-driven decision-making. At the central level the approach focuses on increasing communication, promoting best practices in management, and building a cadre of experts to make evidence-based decisions with the ultimate aim of improving educational outcomes.. To sustain positive change both at the school and central levels, there must be effective systems in place to identify what works and to institutionalize those practices. By sharing data, working together with MEHE to identify key areas of collaboration, and jointly providing school directors and teachers with more relevant and timely support, the approach will likely lead to an increase in access to education and improved classroom teaching and school management 34 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION PURPOSE AND INTENDED USE The purpose of this mid-term evaluation of the QITABI project is to assess the relevance, effectiveness, and sustainability of progress toward reaching the project’s expected results. The evaluation will provide pertinent information about project’s results thus far, lessons learned, and practical recommendations including corrective actions (if any) required maximizing the benefits of the project. The evaluation findings will assist USAID, project implementing partners, and CERD/MEHE in decision making to (a) understand the immediate results manifesting in the targeted schools, in MEHE and in CERD, (b) enhance QITABI’s interventions in the remaining life of the project, and (c) inform future USAID basic education programs. The evaluation covers the period between September 2017 and May 2017. INTENDED AUDIENCE The primary audience for the evaluation includes: (i) the USAID/Lebanon Mission, particularly the Education, Democracy and Governance Office, and QITABI implementing partners. The secondary users include (i) MEHE, (ii) CERD, and (iii) other donors and organizations working on basic education in Lebanon. The final evaluation report will become publicly available on the Development Experience Clearinghouse (DEC). EVALUATION QUESTIONS 1. Relevance How relevant is the QITABI project to the needs of the public basic education sector in Lebanon? Explanation: This question addresses the relevance of the project to MEHE’s long and short￾term needs. It will focus on exploring ways in which the project supports: a. The emerging needs of MEHE and the increased demand for access to quality education and educational systems as the result of the Syrian refuge influx in the country. MEHE Education Sector Development Plan (2010-2015) and Reaching All Children with Education (RACE II 2017 and 2021) will be used as references to help guide the answer to this question. b. USAID’s 2011-2015 Education Strategy and USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019. 2. Effectiveness To what extent is QITABI on track to achieve its objectives—i.e., improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools? Explanation: This question addresses the effectiveness of the project components. The answer to the question should explore: a. What have been the concrete contributions of the project interventions thus far in achieving the objective of the project? b. What has worked and how, what has not worked as anticipated, and what are any unintended outcomes? c. What factors have been crucial for the achievement or project’s inability to achieve planned results thus far? d. What gender considerations have been taken into account by different project interventions and what are the most visible discrepancies, if any? 35 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Answers should contribute to identifying corrective actions (if any) to maximize the benefits of the project and make recommendations for the enhancement of the results in the remaining project life. 3. Sustainability To what extent are the project activities and results likely to continue after the project ends? Explanation: This question will focus on exploring: a. What are the prospects for the sustainability of the end results of the project and what results appear to be less sustainable and why? b. What is QITABI’s sustainability plan and what is required of QITABI, MEHE, and USAID for the sustainability of the results? The question will generate recommendations about (a) what steps and resources will be necessary between now and when the project ends to sustain project results after the project is concluded, and (b) what type of support MEHE and CERD will need after the project ends to continue project achievements beyond the life of the project. EVALUATION DESIGN AND METHODS The evaluation team will use a primarily qualitative approach to answer the evaluation questions, conducting interviews with key stakeholders and focus group discussions with project beneficiaries. The KII respondents will include QITABI project staff, USAID/Lebanon’s Education Office staff, MEHE, and CERD. The FGD participants will consist of the direct project beneficiaries—teachers and principals. This will be supplemented with quantitative data available from project monitoring, but no primary quantitative data will be collected. The evaluation questions as mapped against data sources, data collection methods, and analysis methods are listed in Table 1. Data Collection Data Collection will begin on May 8th and finish by May 29th. The evaluation will employ the following data collection methods: 1. Desk Review: The evaluation team will conduct a systematic literature review of relevant internal and external documents These include the contract agreement, contract modifications, EGRA baseline report, project work plans, annual and quarterly reports, memoranda of understandings signed with MEHE, USAID’s Education Strategy, RACE II strategy document and all other project documents relevant to answering the evaluation questions. In addition, the evaluation team will review relevant external studies of Lebanon’s education sector that may lend additional context to the project’s relevance, effectiveness, and sustainability. 2. Key Informant Interviews (KII): The qualitative interview protocols will focus on exploring the relevance of the QITABI project to MEHE’s long and short-term needs, how effective the project has been so far, and how sustainable the anticipated results will be. The team will conduct KIIs with USAID key staff members, World Learning, Ana Aqra’a’, MSI, and AMIDEAST along with key officials from MEHE and 36 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION CERD. The KIIs will provide an in-depth understanding of project achievements to date, factors that influenced the effectiveness of project intervention, and implementation of project interventions. See Annex 3 for data collection instruments. Annex I contains the list of key informants. The list includes the main points of contact, decision-makers and activity managers from each organization for the project as identified by USAID and World Learning, supplemented with stakeholders identified during desk review. The evaluation team (ET) will update this list with any additional key informants identified during the course of initial interviews and a full list of key informants consulted will be included as an annex in the final evaluation report. The ET anticipates conducting up to 25 KIIs. 3. Focus Group Discussions: Social Impact’s data collection resource partner in Lebanon, Information International, will conduct focus group discussions with selected project beneficiaries, primary teachers, principals and CERD and DOPs trainers/coaches. FGD moderators and note-takers from Information International will undergo a day￾long refresher training with SI on performing effective FGDs including skills training in facilitation, communication and note-taking. The sampling for the FGDs, described in the following section, is determined based on a purposive sample of the different schools, principals and teachers involved in the project across Lebanon so that both high- and low-uptake participants will be included in the sample, as well as urban, and rural schools (See Annex 2 for the list of FGD participants). The FGD questions will focus on how participation in QITABI activities (e.g. training) might have affected their practices, classrooms and schools. This information from FGDs will help the evaluation answer the effectiveness and, to some extent, sustainability questions. The ET will also conduct FGDs with CERD and MEHE’s DOPS staff who participated in QITABI trainings. These FGDs will focus on the relevance and effectiveness of the training, the effectiveness and sustainability of the rollout trainings CERD conducts, and the effectiveness and sustainability of DOPS’s school coaching based on QITABI trainings. In addition, the ET expects to conduct FGDs with a sample of trainers from each implementing partner (IP), and of EGR facilitators, in order to triangulate their feedback with those of trainees and get a more complete picture of the effectiveness and challenges of trainings. For the FGD sample, see Table 1 in the following section. FGD Sampling by Component Task and Beneficiary Type To identify a representative set of direct beneficiaries for interviews that can give diverse points of view and ultimately allow for documenting best practices and lessons learned, the ET has determined a purposive sample of QITABI’s direct beneficiaries for the FGDs. The final sample size balances time available against interviewing a broad set of participants for each component and sub-task. The sample also ensures that different geographic areas in Lebanon are represented. 37 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION The evaluation team plans to conduct 32 FGDs with direct beneficiaries. This includes 18 with teachers attending EGR training and coaching, eight with principals of schools that benefited from components 1 and 2, and six with CERD and DOPS staff who participated in QITABI trainings. The total number of FGDs will be 37 when including those FGDs with IP trainers and EGR facilitators. The ET estimates that this will cover approximately 208 schools and 299 individuals, although these numbers may vary depending on the ability of the invited participants to attend and MEHE’s approval of the schedule. The evaluation team categorized FGD participants according to component task. Furthermore, the categorization process took into account participant mobility, geographic spread of schools, FGD venue proximity to FGD participants, and total number of QITABI beneficiaries in each category. It should be noted that the host school intervention is of special interest to USAID and QITABI for its focus on alleviating pressure on public schools that are hosting high numbers of Syrian refugee students. Table1 shows the breakdown of the FGDs by task and beneficiary type. Table 1: FGD Sample Stakeholder title # FGDs # Schools represented (est.) # FGD Respondents (est.) Task EGR Training 1 Teachers 8 64 64 Task EGR Training 2 Teachers 8 64 64 Task EGR Coaching Teachers 2 16 16 Task Classroom Libraries Principals 8 64 64 Task Classroom ICT Task Resource Room ICT Task School Buses10 Total of FGDs with school beneficiaries 26 208 208 Government Trainees CERD trainers (EGR TOT, EGR analysis and SPSS training) 3 26 DOPS coaches (EGR CB, EGR analysis and SPSS training) 3 23 Project Trainers 10 Since QITABI hasn’t yet provided buses to the selected schools, the FGD questions will only explore respondents’ perception of the association between transportation provided by school and increased access to education for vulnerable children. 38 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Ana Aqra’a – EGR TOT and CB 1 5 Ana Aqra’a – EGR coaching 1 10 Ana Aqra’a- EGR Training 1 7 EGR Facilitators 2 20 FGD Grand Total 37 208 299 In addition, the evaluation team considered the following factors when selecting the school FGD participants: • Geographic representation: To ensure that different geographic areas in Lebanon are represented and facilitate teachers’ and principals’ ease of attending FGDs, the ET has stratified FGD selection by governorate, and within governorate, clustered FGDs in Cazas with high participation (or, for smaller Cazas where travel is easier, grouped 2-3 together). FGDs will be held in Beqaa, Akkar, North Lebanon, South Lebanon, Mt. Lebanon, Nabatieh, Baalbek and Beirut. • Urban vs rural areas: Schools situated in both rural and urban areas will be represented in all the different categories of FGD targeting teachers and principals. • School representation: There will be maximum representation of targeted schools in the FGDs. Out of the 513 schools targeted by the project, the FGD will cover an estimated 217 schools. • Gender consideration: To the extent possible, an equal representation of men and women will be taken into consideration when selecting the FGD participants. As there are considerably higher numbers of female teachers, care will be taken to purposively select male participants for inclusion in the FGD. For the relatively smaller trainer populations—e.g., trainings with only 5-7 trainers—the ET will include all trainers available in the FGD. For trainer populations that exceed the reasonable size for a focus group, the ET will select 8-10 trainers based on availability to participate in FGDs (convenience sampling). Because the ET expects that the primary value of these FGDs will be to triangulate responses with those of the direct beneficiaries to verify their feedback, the ET will conduct only one FGD with each trainer category. Data Analysis Methods Once the ET completes data collection and receives all the focus group notes, it will reassemble at the Social Impact Beirut Office for data analysis and development of findings, conclusions and recommendations. The study team will use a structured and systematic approach to analyzing the qualitative data and will triangulate across multiple methods and data sources to ensure the reliability and validity of results. The study team will analyze the qualitative data as follows: • Summarize notes from individual interviews and FGDs; 39 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION • Code KII and FGD data according to themes relevant to the evaluation questions; • Prepare an evaluation data analysis matrix identifying the themes that emerge in the KIIs and FGDs to facilitate systematic and rigorous data analysis aimed at identifying key study findings as they relate to the evaluation questions; and • Prepare a detailed outline summarizing key findings based on all the data analysis for each study question and overall recommendations. Social and Gender Analysis The evaluation team recognizes that gender and other social factors play a role in how direct and indirect beneficiaries are affected by and participate in project activities. The team anticipates that beneficiary gender and the representation of women among the overall group of QITABI direct beneficiaries may have implications for project relevance, effectiveness, and sustainability. To the extent possible, the ET will also explore how the trainings may have affected male and female students’ learning environments and classroom performance differently (namely in resource allocation and accessibility, participation and reading skills). This will be based on the data from teachers and principals FGDs only. Lebanese public schools are under immense pressure as a result of the increasing number of non-Lebanese primary school students. The evaluation team will explore how refugee children may have benefitted from project interventions, whether these benefits were differently apportioned, how targeting of host schools directly or indirectly influenced project effectiveness and relevance. Limitations Since this is a mid-term evaluation and QITABI closes in September 2018, QITABI partners and USAID will have ample time to benefit from the evaluation findings. However, the timing of the evaluation has implications for data collection. The data for the evaluation will be collected between May 8 and May 29. This is a particularly busy time for teachers and principals who will be conducting examinations and grading papers, which may affect the FGD participants’ turn out. In order to maximize turnout, the evaluation team will invite up to 14 participants for each FGD (where possible). It is expected that that average number of participants per FGD will be between seven and 10. The majority of the public primary school teachers in Lebanon are female (approximately 87%).11 It was not possible to hold separate FGDs for male and female teachers. As a result findings from the FGDs cannot be disaggregated by gender. 11 UNESCO: Percentage of Female Teachers by Teaching Level of Education. http://data.uis.unesco.org/index.aspx?queryid=178 (Retrieved on May 10, 2017) 40 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION However, the evaluation team plans to review project data and disaggregate findings according to gender where possible. We are also reviewing the way the project activities take male/female students’ needs into account. Improving students’ reading is the goal of the project. Yet the self reported data we collect will give limited information about whether students’ reading has actually improved. In addition, it will not be possible to collect accurate data about the project’s contribution to students’ reading at this stage given that this is a formative evaluation (the project is ongoing) rather than a summative evaluation. That said, the evaluation team will review and report all relevant QITABI data on students, including the EGRA baseline data and teachers subsequent assessments. 41 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Table 1: Evaluation Design Matrix Evaluation Criteria and Question Sub-topic Data Sources Data Collection Methods Data Analysis Plan Relevance: How relevant is the QITABI project to the needs of the public education sector in Lebanon? MEHE’s long- and short￾term needs as highlighted in the ESDP - RACE/RACE II; RACE and other official MEHE documents Desk review Summarize notes from KII Interview coding of context from MEHE officials and background documents; data analysis matrix - USAID Education Strategy and CDCS - MEHE staff - IP staff KIIs - Lebanon education sector background documents Effectiveness: To what extent is QITABI on track to achieve its objective- i.e. improving reading skills and expanding equitable access to education for vulnerable students in Lebanese public schools? Effectiveness of the project components: -Teachers participating in EGR coaching/training Desk review Component 1: Improved reading outcomes for primary level public school students -Principals of schools with ICT procurement KIIs Interview coding; data analysis matrix -IP staff -MEHE staff FGDs Comparison of teacher training groups and teacher coaching groups -CERD and DOPS Gender analysis -QITABI trainers of CERD/DOPS (EGR/CB/Analytics) -World Learning staff -Project documents on QITABI M&E, USAID education strategy, MEHE data where available Component 2: Expanded access to safe and relevant education for vulnerable public school students -Teachers participating in EGR coaching/training - Teachers at schools with ICT equipment - Principals of schools with ICT procurement -QITABI trainers of CERD/DOPS (EGR/CB/Analytics) - Project documents on QITABI M&E, USAID education strategy, MEHE policies Desk Review FGDs KIIs Gender analysis Interview coding; data analysis matrix -World Learning staff 42 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Component 3: Strengthening management (resilience) in the education system to better direct and monitor education - Project documents on QITABI M&E, USAID education strategy, -MEHE policies -World Learning staff FGDs KIIs Interview coding - Project M&E data, reports Desk Review Sustainability: To what extent are the project activities and results likely to continue after the project ends? School (principal and teacher) continuation of activities or retention of training School ability to manage ICT resources -Principals - Ministry officials FGDs KIIs Interview coding -Teachers - CERD and DOPS Desk Review - Project reports/MEHE data -QITABI staff 43 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION EVALUATION MANAGEMENT Deliverables, Timeline and Dissemination Plan As set out in the initial SOW, the ET will deliver the following to USAID: Deliverable Expected Delivery Date Scope of Work: PMSPL II team will prepare the SOW including methodologies, tools, and evaluation work plan and time schedule. The SOW will be submitted to the COR at USAID/Lebanon for approval Completed and approved on April 7, 2016 Inception Report: PMSPL II team and Evaluation team will conduct a systematic literature search, assembling key documents, and meeting with possible sub-contractors. Both teams will also prepare a draft methodology plan including sampling methodology, beneficiary matrix, primary research questions, interview protocols, focus group questions and a list of proposed individuals to be interviewed. The methodology plan, interview schedules, and interview protocol will be presented to USAID staff prior the start of the evaluation. Submitted to USAID May 12, 2017 Briefings: The evaluation team will provide a briefing to USAID prior to the start of the evaluation and at the end of the evaluation. The initial briefing will focus on presenting the evaluation team plan for conducting the evaluation. Discussions with the Mission on the proposed methodologies for the evaluation will result in an Inception Report (a deliverable) that states the agreed methodologies. Toward the end of the field work, prior to the development of the Draft Report, the team will carry out a preliminary findings presentation at USAID. In addition to this, a final briefing meeting will be done after the final report is written. Initial in-briefing with USAID conducted May 4, 2017 Weekly email updates from the ET to the QITABI and PMSPL II CORs, with meeting as necessary Preliminary Findings presentation on June 12, 2017 Draft Report and outline of the Final Report: A draft report of the findings and recommendations will be submitted to USAID COR clearly describing findings, conclusions, and recommendations, within two weeks of completing the data collection. USAID will provide comments on the draft report within one week of submission. The evaluation team will consider USAID comments and revise the draft report accordingly and as appropriate for an independent evaluation. Where differences of opinion exist these will be presented in a Statement of Differences Annex. Expected delivery June 23 with USAID comments to be provided by July 3, 2017 Final Report: The final evaluation report will be provided to USAID in MS Word and Adobe PDF within 15 calendar days following receipt of final comments from USAID. The report will include all of the components outlined in the “Structure of the Evaluation Report”, but not necessarily in the order specified below. The report will not exceed 25 pages (excluding Expected delivery of final report on July 17 subject to receipt of comments from USAID by July 3. 44 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION appendices). Appendices should at a minimum include the scope of work for the evaluation; a list of individuals interviewed; a complete description of the methodology used for the evaluation; and any survey or questionnaires used. The report will be submitted in English, electronically in MS Word format and compliant with USAID Graphic Standards. All deadlines assume timely approval from MEHE for fieldwork and USAID’s ability to give comments on the inception report and draft evaluation report on the timelines discussed during the initial kickoff meeting. These deadlines and other evaluation activities are illustrated in the work plan found on the following two pages. PMPSL will upload the final approved evaluation report to the DEC for public access and distributed to WL, QITABI and other stakeholders at the discretion of USAID/Lebanon. USAID will share the report with other stakeholders at its discretion. The report will follow USAID branding requirements. Comments received after the budgeted end date for this evaluation may be addressed in a Statement of Differences Annex at the discretion of USAID/Lebanon. 45 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION WORKPLAN 47 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION TEAM STRUCTURE The evaluation team will consist of a Team Leader, Local Education expert, Qualitative Research Analyst, Performance Evaluation Specialist, technical support from PMSPL II evaluation team, and data collection from Information International, PMPSL’s data collection sub-contractor. SI’s head office will support evaluation management and quality. 1) Evaluation Team Leader, Dr. Jehanzaib Khan: Dr. Khan is responsible for coordinating and directing the overall evaluation, including preparation and submission of the draft and final evaluation reports. (45 days LOE) 2) Lebanon Education Expert, will contribute expert technical inputs into designing the evaluation methodology, creating FGD and KII instruments, conducting FGDs and KIIs, participating in data analysis, and contributing to report writing. (30 days LOE) 3) Qualitative Research Analyst, Zeina Helou will contribute her significant experience in qualitative data collection, management and analysis during field work with beneficiaries as well as contributing to the draft report input (30 days LOE). 4) Performance Evaluation Specialist, Leah Ghoston: Ms. Ghoston contributes quality assurance to ensure that the evaluation follows SI’s EQUI® process. She additionally serves as a team member to contribute to evaluation methodology and instrument design, facilitate FGDs and KIIs, and contribute to data analysis and reporting. (48 days LOE) 5) Information International: Given the limited time available for fieldwork and complexity of the QITABI project, SI will retain the PMPSL data collection partner, Information International, to conduct the bulk of the FGDs. A team of approximately 10 Information International facilitators will conduct the FGDs with direct beneficiaries over a period of one week. They will create detailed summary sheets for each focus group daily and share these with the ET as they are created. PMSPL II staff work alongside the other team members as part of the evaluation team, providing logistical and technical support. PMSPL II resident staff and HO staff participates in drafting, reviewing and editing the draft final report before circulating to the stakeholders for comment and review. Preparation of the evaluation report is the responsibility of the team leader/technical expert. Final editorial responsibility and quality control for the report is with Social Impact. Comments received are to be address by the technical expert supported by the PMSPL II staff. The draft evaluation report is to be reviewed by USAID/Lebanon and, at the request of USAID/Lebanon, a broad range of stakeholders. PMSPL II will certify that there is no conflict of interest or potential conflict of interest with respect to the performance of this evaluation on the part of the contractor and the contractor’s team members. Social Impact will guarantee that substitutions will not be made for individuals proposed as team members without the approval of USAID 48 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ANNEX 2: DOCUMENTS REVIEWED List of Key Documents Reviewed 1 QITABI Contract Agreement 2 Project Modifications Documents 3 Quarterly Report (10) 4 Work Plans (Three) 5 M&E Plans 6 Reaching All Children with Education (RACE II) 7 ICT Action Plan 8 MEHE’s Education Sector Development Plan (2010-2015) 9 MHE’s National Educational Technology Strategic Plan 10 Country Development Cooperation Strategy (CDCS) 2014-2019 11 Early Warning System Guide 12 EGR Training Modules 13 School Buses O&M plan 14 Universal Screening guide (grades 1, 2 ,3 ,4) 15 EGRA Baseline Report 16 Reading Benchmarks Report 17 Teachers Classroom Observation Tool 18 Teacher Questionnaire 19 Activeinspire User Guide 20 Resource Room Logbook 21 Project Database (list of schools, list of trainees, list of trainers, and others) 49 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ANNEX 3: DATA COLLECTION INSTRUMENTS It should be noted that these protocols are general guides with questions to ask across respondents/groups from each category of stakeholders, such that certain results can be compared across these groups. Depending on the stakeholder, particular questions may be omitted if they are not relevant. Ultimately, protocols will be specifically tailored for each respondent and group as more relevant details become available. The protocols listed here are the ones that were finalized during the inception report development phase. KII Guides Quality Instruction Towards Access and Basic Education Improvement (QITABI) Mid-Term Evaluation Qualitative Interview Protocol (WL COP-DCOP) Introduction: We work for Social Impact, a USAID contractor that provides performance monitoring and evaluation support for various development projects in Lebanon. The purpose of this interview/focus group discussion is to gain further knowledge and insight into the work of the USAID-sponsored Quality Instruction Towards Access and Basic Education Improvement (QITABI). As a knowledgeable person within your organization we value your input to our evaluation. We anticipate that your responses will help us gain further knowledge of QITABI’s contributions to basic education sector in Lebanon, understand challenges faced during implementation, and develop recommendations for future programming. Your participation is completely voluntary. Your answers will be used for data analysis purposes only and will be kept strictly confidential. No Interview respondents will be identified by name in the report. Before the interview begins, I would like to ask your permission to take notes to capture your responses and to tape record the interview so that we can further enhance our notes for data analysis. Organization: Date: Respondent’s Name: Venue: Official Title: Interviewee: Note-taker: Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – Ensuring timely implementation of project activities – Coordination with implementation partners – Coordination with MEHE and CERD – Coordination with USAID RELEVANCE: 50 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 2. What is your understanding of the needs of public education in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how each component addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school students C2: Expand access to safe and relevant education for vulnerable public school students C3: Strengthened management to better direct and monitor education – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 4. In your opinion, are the project interventions making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? Examples of some the key interventions? Component 1 • EGRA baseline • Assessment of teachers’ knowledge of reading instruction • EGR training and coaching (CERD, DOPS, teachers) • Development of reading benchmarks Component II • Provision of resources/equipment to schools • Professional development • What else? Component III • What are C3 achievements thus far 5. Please tell us about some of the major achievements of QITABI thus far a. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of project activities? b. Please describe how QITABI is addressing those challenges? 7. Has the implementation of all project activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – Parent teacher associations/Parents circles meetings – Professional development/ICT trainings – Procurement of buses – Non-formal education (NFA) 51 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION – Psychosocial support to schools? – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? a. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? b. How would you describe coordination between QITABI, and MEHE and CERD? Successes? Challenges? – 9. Please tell us about some of the key accomplishments of QITABI that benefited/will benefit MEHE? a. What strategic plans, policies, manuals, curricula, and institutional support model (under C3) developed by QITABI are/will be adopted and used by MEHE. Please give details b. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks – Early Warning System (ESW) – Institutional support model (C3) 10. To what extent are the planned interventions of QITABI sufficient for achieving the desired results? (For example, can provision of ICT equipment help expanding access to education?) a. Please describe how these results will contribute toward achieving the project objective? Examples? - Improve reading outcomes - Expand access to safe and relevant education - Strengthen management 11. Did the project design/implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefit from the project activities? c. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities? Examples of activities: – EGR activities – Access to resource rooms – Access to classroom libraries – Access to ICT equipment 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? a. Please describe how QITABI coordinates with those projects? b. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of QITABI results? c. Please describe factors that might make QITABI results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 52 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION WL – Component 1 LEAD 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that key project activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, is MEHE committed to institutionalizing the EGR model, including reading benchmarks EWS? c. What about the maintenance of ICT equipment after the project ends? In your understanding, is MEHE committed and/or does it have the resources to provide ICT maintenance support to schools after the project ends? d. What about the maintenance of school buses that QITABI plans to provide? e. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – Coordination other implementation partners – Coordination with MEHE and CERD – Coordination with USAID RELEVANCE: 2. What is your understanding of the needs of public education in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? b. Please describe how Component I addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 53 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 4. In your opinion, are the project interventions, particularly Component 1 activities, making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? b. Please tell us what QITABI’s Balanced Literacy Approach (BLA) entails? c. Is it the right approach in the current Lebanese context? If yes, please tell how? d. Please describe how BLA complements/supports the implementation of EGRA in Lebanon? Examples of some the key interventions? Component 1 • EGRA baseline • EGRA-midline (a.k.a. universal screening in EGR) • EGR training and coaching (CERD, DOPS, teachers) • Assessment of teachers’ knowledge of reading instruction • Development of reading benchmarks • EWS • Parents reading circles • What else? 5. Please tell us about some of the major achievements of QITABI thus far b. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of C1 activities? b. Please describe how QITABI is addressing those challenges? 7. Has the implementation of Component 1 activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – Parent teacher associations/Parents circles meetings – Reading Benchmarks – EWS – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? a. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? b. How would you describe coordination between QITABI, and MEHE and CERD? Successes? Challenges? – 9. Please tell us about some of the key accomplishments of QITABI (Component 1) that benefited/will benefit MEHE? c. What strategic plans, policies, manuals, and curricula developed by QITABI are/will be adopted and used by MEHE and CERD. Please give details d. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks – Early Warning System (ESW) 54 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION WL – Component I1 LEAD 10. To what extent are the planned Component 1 interventions of sufficient for achieving the desired results? a. Please describe how these results will contribute toward achieving the project objective? Examples? - Improve reading outcomes 11. Did the project design/implementation include a specific strategy for including women? d. What steps were/are being taken to ensure participation of women? Explain? e. What steps were/are being taken to ensure male and female students equally benefit from the project activities? f. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities? Examples of activities: – EGR activities – Access to resource rooms – Access to classroom libraries – Access to ICT equipment 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? c. Please describe how QITABI coordinates with those projects? d. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of Component 1 results? c. Please describe factors that might make Component 1 results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – CERD and DOPS training – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that Component 1 activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, are MEHE and CERD committed to institutionalizing the EGR model, including reading benchmarks, EWS? c. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? c. How long have you served in this position? a. How long have you been involved with QITABI and in what – Responsibilities – Ensuring timely implementation of project activities – Coordination with implementation partners 55 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ways? – Coordination with MEHE and CERD – Coordination with USAID RELEVANCE: 2. What is your understanding of the needs of public education in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how Component 2 addresses MEHE’s needs: C2: Expand access to safe and relevant education for vulnerable public school students – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 4. In your opinion, are the project interventions, particularly Component 2 activities, making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? b. Please tell us QITABI’s approach to building CERD and DOPS capacity through C2? c. Please describe ways in which providing ICT equipment and school buses may increase access to education for vulnerable children? d. Please tell us how NFE activities contribute to expanding access to education? Examples of some the key interventions? Component 1 • Transportation (100 buses) • ICT equipment for 300 schools • Professional Development in ICT and Psychosocial support for DOPS • ICT training/TOT for CERD • Non-formal education through school in a bus • What else? 5. Please tell us about some of the major achievements of QITABI thus far a. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID c. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of C2 activities? d. Please describe how QITABI is addressing those challenges? 56 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 7. Has the implementation of Component 2 activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – School buses – ICT equipment – ICT training (DOPS and CERD) – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Overburden MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? a. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? b. How would you describe coordination between QITABI, and MEHE and CERD? Successes? Challenges? – 9. Please tell us about some of the key accomplishments of QITABI (Component 2) that benefited/might benefit MEHE? a. What strategic plans, policies, manuals, and curricula developed by QITABI are/will be adopted and used by MEHE and CERD. Please give details b. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. – ICT equipment – School buses – NFE – What else? 10. To what extent are the planned Component 2 interventions of sufficient for achieving the desired results? (For example, can provision of ICT equipment to school help expanding access to education?) b. Please describe how these results will contribute toward achieving the project objective? Examples? a. Expand access to safe and relevant education 11. Did the project design/implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefit from the project activities? c. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities? Examples of activities: – Equal access to school buses – ICT equipment – Psychosocial support (girls’ needs Vs boys’ needs) – Non-formal education – Access to resource rooms – Access to classroom libraries 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? a. Please describe how QITABI coordinates with those projects? b. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 57 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION WL Country Representative 13. To what extent does the project design/implementation specifically address sustainability and how? d. In your opinion, are the project results sustainable? Which ones? a. If yes, please describe the factors that contribute/will contribute to the sustainability of Component 2 results? b. Please describe factors that might make Component 2 results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – CERD and DOPS ICT training – Transportation (school buses) – ICT equipment – Non-formal education 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that Component 2 activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, are MEHE and CERD committed to institutionalizing the ICT in education and ICT professional development models? c. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? d. How long have you served in this position? e. How long have you been involved with QITABI and in what ways? f. What kind of support to you provide to QITABI? – Responsibilities – Providing support to of QITABI activities and other WL projects – Coordination with implementation partners – Coordination with MEHE and CERD – Coordination with USAID RELEVANCE: 2. What is your understanding of the needs of public education in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? c. Please describe how each component addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school students C2: Expand access to safe and relevant education for vulnerable public school students C3: Strengthened management to better direct and monitor education – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 58 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION EFFECTIVENESS: 4. In your opinion, are the project interventions making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? Examples of some the key interventions? Component 1 • EGRA baseline • Assessment of teachers’ knowledge of reading instruction • EGR training and coaching (CERD, DOPS, teachers) • Development of reading benchmarks Component II • Provision of resources/equipment to schools • Professional development • What else? Component III • What are C3 achievements thus far 5. Please tell us about some of the major achievements of QITABI thus far a. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of project activities? b. Please describe how QITABI is addressing those challenges? 7. Has the implementation of all project activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – Parent teacher associations/Parents circles meetings – Professional development/ICT trainings – Procurement of buses – Non-formal education (NFA) – Psychosocial support to schools? – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? c. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? d. How would you describe coordination between QITABI, and MEHE and CERD? Successes? Challenges? – 9. Please tell us about some of the key accomplishments of QITABI that benefited/will benefit MEHE? a. What strategic plans, policies, manuals, curricula, and institutional Explore the process through which these were developed: – Policies, Strategy documents, professional 59 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION support model (under C3) developed by QITABI are/will be adopted and used by MEHE. Please give details b. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? development manuals/curricula developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks – Early Warning System (ESW) – Institutional support model (C3) 10. To what extent are the planned interventions of QITABI sufficient for achieving the desired results? c. Please describe how these results will contribute toward achieving the project objective? Examples? - Improve reading outcomes - Expand access to safe and relevant education - Strengthen management 11. Did the project design/implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefited from the project activities? Examples of activities: – EGR activities – Access to resource rooms – Access to classroom libraries – Access to ICT equipment 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? a. Please describe how QITABI coordinates with those projects? b. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of QITABI results? c. Please describe factors that might make QITABI results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that key project activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, is MEHE committed to institutionalizing the EGR model, including reading benchmarks EWS? c. What about the maintenance of ICT equipment after the project ends? In your understanding, is MEHE committed and/or does it have the resources to provide ICT maintenance support to schools after the project ends? d. What about the maintenance of school buses that QITABI plans to provide? e. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 60 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION WL M&E Manager 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served i this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – Tracking progress – Progress report and M&E plans – Coordination with implementation partners RELEVANCE: 2. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how each component addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school students C2: Expand access to safe and relevant education for vulnerable public school students C3: Strengthened management to better direct and monitor education – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 3. In your opinion, are the project interventions making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? Examples of some the key interventions? Component 1 • EGRA baseline • Assessment of teachers’ knowledge of reading instruction • EGR training and coaching (CERD, DOPS, teachers) • Development of reading benchmarks Component II • Provision of resources/equipment to schools • Professional development • What else? Component III • What are C3 achievements thus far 4. Please tell us about the QITABI’s M&E mechanism. a. Please tell us how QITABI monitors project activities? b. Please describe how QITABI tracks progress? c. What kind of M&E support do you provide to Implementing partner? d. What support do you receive from the implementing partner in tracking progress/monitoring project activities? 5. Please tell us about some of the major achievements of QITABI thus far a. In your opinion, what are the factors that played key roles in - QITABI leadership? - Role of QITABI partners - Role of USAID 61 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION accomplishing those achievements? - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. Please describe how QITABI is addressing those challenges? 7. Has the implementation of all project activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – Parent teacher associations/Parents circles meetings – Professional development/ICT trainings – Procurement of buses – Non-formal education (NFA) – Psychosocial support to schools? – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? a. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? b. How would you describe coordination between QITABI, and MEHE and CERD? Successes? Challenges? – 9. Please tell us about some of the key accomplishments of QITABI that benefited/will benefit MEHE? a. What strategic plans, policies, manuals, curricula, and institutional support model (under C3) developed by QITABI are/will be adopted and used by MEHE. Please give details b. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks – Early Warning System (ESW) – Institutional support model (C3) 10. To what extent are the planned interventions of QITABI sufficient for achieving the desired results? d. Please describe how these results will contribute toward achieving the project objective? Examples? - Improve reading outcomes - Expand access to safe and relevant education - Strengthen management 11. Did the project design/implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefited from the project activities? Examples of activities: – EGR activities – Access to resource rooms – Access to classroom libraries – Access to ICT equipment 12. Do you have knowledge of other education programs - The World Bank (RACE II) 62 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION USAID/Education Office implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? a. Please describe how QITABI coordinates with those projects? b. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - UNICEF - GIZ - DFID SUSTAINABILITY 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of QITABI results? c. Please describe factors that might make QITABI results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that key project activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, is MEHE committed to institutionalizing the EGR model, including reading benchmarks EWS? c. What about the maintenance of ICT equipment after the project ends? In your understanding, is MEHE committed and/or does it have the resources to provide ICT maintenance support to schools after the project ends? d. What about the maintenance of school buses that QITABI plans to provide? e. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – Ensuring timely implementation of activities – Coordination with and providing support to QITABI – Coordination with MEHE and CERD RELEVANCE: 2. What is your understanding of the needs of public education sector needs in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting the provision of public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 63 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how each component addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school students C2: Expand access to safe and relevant education for vulnerable public school students C3: Strengthened management to better direct and monitor education b. Was MEHE consulted with when designing the QITABI project? If yes, please describe the process through which USAID took MEHE on board. If no, why not? – MEHE’s emerging and long term needs/priorities given the Syrian refuge crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 4. In your opinion, are the project interventions making adequate progress toward achieving the planned results? a. If yes, please give us examples of the interventions and how they are contributing to achieving the project’s planned results. If no, please tell us why not? Examples of some the key interventions? Component 1 • EGRA baseline • Assessment of teachers’ knowledge of reading instruction • EGR training and coaching (CERD, DOPS, teachers) • Development of reading benchmarks Component II • Provision of resources/equipment to schools • Professional development • What else? Component III • What are C3 achievements thus far 5. In you opinion, what are some of the major achievements of QITABI thus far a. What are the factors that played key role in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE and CERD - Working QITABI partners - What else? a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of project activities? b. Please share your understanding of how QITABI is addressing those challenges. c. Please describe how USAID is supporting QITABI in addressing those challenges. 7. Has the implementation of all project activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – Parent teacher associations/Parents circles meetings – Professional development/ICT trainings – Procurement of buses – Non-formal education (NFA) – Psychosocial support to schools? – What else 64 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? a. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take those steps? b. How would you describe coordination between QITABI and USAID, and QITABI and MEHE? Successes? Challenges? 9. Please tell us about some of the key accomplishments of QITABI that benefited/will benefit MEHE? a. What strategic plans, policies, manuals, curricula and institutional support model (under C3) developed by QITABI are/will be adopted and used by MEHE. Please give details b. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? – Policies, Strategy documents, professional development manuals/curricula developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks 10. To what extent are the planned interventions of QITABI sufficient for achieving the desired results? e. Please describe how these results will contribute toward achieving the project objective? Examples? - Improve reading outcomes - Expand access to safe and relevant education - Strengthen management 11. Did the project design/implementation include a specific strategy for including women? a. What steps is USAID taking to ensure participation of women? Explain? b. What steps is USAID taking to ensure male and female students equally benefit from the project activities? c. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities? Examples of activities: – EGR activities – Access to resource rooms – Access to classroom libraries – Access to ICT equipment 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? a. Please describe how USAID coordinates with those projects? b. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? b. If yes, please describe the factors that contribute/will contribute to the sustainability of QITABI results? c. Please describe factors that might make QITABI results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks 65 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION MSI Component 1 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that key project activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, is MEHE committed to institutionalizing EGR model including reading benchmarks and EWS? c. In your understanding, is MEHE committed and/or does it have the resources to provide ICT maintenance support to schools after the project ends? d. What about the maintenance of school buses that QITABI plans to provide? e. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? f. What are USAID’s plans between now and end of the project for making key project achievements sustainable? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – EGRA baseline – EGR intervention – Coordination with other implementation partners RELEVANCE: 2. What is your understanding of the needs of public education in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how Component I addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 4. Please describe MSI’s support/contribution to the implementation of QITABI. 4.1. Please tell us about the EGRA baseline. a. How was it implemented and who was involved in the implementation—sample selection, enumerator training, data – EGRA baseline – Teacher survey – Establishing benchmarks for Grades 1 and 2 – EGRA end-line – Support capacity building of MEHE 66 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION collection and analysis and reporting? b. How do MSI and QITABI plan to use the findings from the baseline c. When is the EGRA end-line expected and what are QITABI’s/MSI’s plans for using the findings from the end-line? 4.2. Please tell us about the purpose of the teacher survey? a. How was it implemented? b. How are the findings from the survey guiding C1 activities? 4.3. Could you describe the process through which national reading benchmarks were established? a. Please explain their purpose and whether they are contributing to improving reading in Lebanon? 4.4. Please tell us what other forms of support is MSI providing to MEHE/CERD. 4.5. To what extent were MEHE and CERD involved in the design and implementation of these C1 activities. Describe the process. – Universal screening (a.k.a. EGRA mid-line) 5. In your opinion, are the project interventions, particularly Component 1 activities, making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? Examples of some the key interventions? Component 1 – EGRA baseline – EGRA-midline (a.k.a. universal screening in EGR) – EGR training (CERD, DOPS, teachers) – Assessment of teachers’ knowledge of reading instruction – Development of reading benchmarks 6. Please tell us about some of the major achievements of QITABI, particularly Component 1, thus far a. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of MSI - Role of USAID - Role MEHE - Role CERD - What else? 7. What kinds of challenges did/does QITABI/MSI encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of C1 activities? b. Please describe how QITABI is addressing those challenges? 8. Has the implementation of Component 1 activities been timely? d. If yes, please tell us about the factors that contribute/d to the timely implementation e. If no, please tell us about the activities whose implementation did not take place according to the project work plans. 67 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION f. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 9. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? a. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? b. How would you describe coordination between MSI and other QITABI partners c. Please give us example of coordination and collaboration between MSI and other QITABI partners. Successes? Challenges? – 10. Please tell us about some of the key accomplishments of QITABI (Component 1) that benefited/will benefit MEHE? a. What strategic plans, policies, manuals, data, and curricula produced by MSI benefit/ will be adopted and used by MEHE and CERD. Please give details b. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks – Early Warning System (ESW) 11. Did the CI implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefit from the project activities? c. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities? Examples of activities: – EGR activities 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes in Lebanon? e. Please describe how MSI coordinates with those projects? f. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of Component 1 results? c. Please describe factors that might make Component 1 results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – CERD and DOPS training – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that Component 1 activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, are MEHE and CERD committed to institutionalizing the EGR model, including reading benchmarks, EWS? c. In your opinion, what kind of support will MEHE and CERD 68 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION MEHE need to continue project achievements beyond the life of the project? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? g. How long have you served in this position? h. How long have you been involved with QITABI and in what ways? – Responsibilities – Coordination with and providing support to QITABI – Coordination with USAID RELEVANCE: 2. What is your understanding of the public education sector needs in Lebanon? c. Please tell us ways in which the Syrian refugee crisis is affecting the provision of public education services in the country. d. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. Please tell us about MEHE’s current education sector plan/policy? When was it developed and who developed it? a. Could you describe ways in which the plan/policy takes into consideration the current demand for education? If MEHE’s current policy is guided by RACE II and ESDP (2010-2015), please ask when MEHE will develop the next education sector plan? 4. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? c. Please describe how each component addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school students C2: Expand access to safe and relevant education for vulnerable public school students C3: Strengthened management to better direct and monitor education – MEHE’s emerging and long term needs/priorities given the Syrian refuge crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 d. Was MEHE consulted with when USAID was designing the QITABI project? If yes, please describe the nature of MEHE’s involvement in the design of the project. e. If no, please tell us what effects, if any, it has had on QITABI’s implementation? – EFFECTIVENESS: 69 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 5. In your opinion, are the project interventions making adequate progress toward achieving the project’s results? b. If yes, please give us examples of the interventions and how they are contributing to achieving the project’s results. If no, please tell us why not? c. Please describe ways in which QITABI activities are supporting public schools in Lebanon? Examples of some the key interventions? Component 1 • EGRA baseline • Assessment of teachers’ knowledge of reading instruction • EGR training and coaching (CERD, DOPS, teachers) • Development of reading benchmarks Component II • Provision of resources/equipment to schools • Professional development • What else? Component III • What are C3 achievements thus far 6. In you opinion, what are some of the major achievements of QITABI thus far c. What are the factors that played key role in accomplishing those achievements? d. Please describe the kind of support MEHE provides to QITABI to achieve its planned results? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 7. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE and CERD - Working QITABI partners e. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of project activities? f. Please share your understanding of how QITABI is addressing those challenges. 8. Tell us about your experience of working QITABI. a. How would you describe coordination between QITABI and MEHE? b. Does MEHE face any challenges? If yes, what kind of challenges does MEHE face in supporting QITABI implementation? – Overburden and overstretched MEHE – Working with WL – Working with other implementing partners/service providers – Working with USAID 9. In you opinion, has the implementation of all project activities been timely? g. If yes, please tell us about the factors that contribute/d to the timely implementation h. If not, please tell us about the activities whose implementation did not take place according project work plans. A few examples of delayed activities: – Parent teacher associations/Parents circles meetings – Professional development/ICT trainings – Procurement of buses – Non-formal education (NFA) – Psychosocial support to schools? – What else i. Please describe the factors that led to the delayed implementation. – QITABI’s approach to getting MEHE on board – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Overwhelmed/overburdened MEHE – What else? 10. Please tell us about some of the key accomplishments of QITABI that benefited/will benefit MEHE? e. What strategic plans, policies, manuals, curricula developed by QITABI are/will be adopted and used by MEHE. Please give details f. Please describe ways in which public schools that are overwhelmed – Policies, Strategy documents, professional development manuals, curricula and institutional support model developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks 70 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION by non-Lebanese students may benefit from the project? – EWS 11. Did the project design/implementation include a specific strategy for including women? g. What steps is MEHE taking to ensure participation of women? Explain? h. What steps is MEHE taking to ensure male and female students equally benefit from the project activities? i. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities? Examples of activities: – EGR activities – Access to resource rooms – Access to classroom libraries – Access to ICT equipment 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? g. Please describe how MEHE coordinates with those projects? h. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 13. In your understanding, to what extent does the project design/implementation specifically address sustainability and how? e. In your opinion, are the project results sustainable? f. If yes, please describe the factors that contribute/will contribute to the sustainability of QITABI results? g. Please describe factors that might make QITABI results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that key project activities and results are sustainable? f. How do you see the roles of MEHE and CERD in the sustainability of the results? g. In you opinion, is MEHE committed to institutionalizing EGR and assessment model including reading benchmarks and EWS? h. In your understanding, is MEHE committed and/or does it have the resources to provide ICT maintenance support to schools after the project ends? i. What about the maintenance of school buses that QITABI plans to provide? j. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? k. What are MEHE’s plans between now and end of the project for making key project achievements sustainable? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 71 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – Coordination with and providing support to QITABI – Coordination with USAID – Coordination with MEHE RELEVANCE: 2. What is your understanding of the needs of public education sector needs in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s and CERD’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s and CERD’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. Please tell us about MEHE’s current education sector plan/policy and CERD’s role in its implementation. b. Could you describe ways in which the plan/policy takes into consideration the current demand for education? If MEHE’s current policy is guided by RACE II and ESDP (2010-2015), please ask reference both RACE II and ESDP in your questions? 4. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how each component addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school students C2: Expand access to safe and relevant education for vulnerable public school students C3: Strengthened management to better direct and monitor education – MEHE’s and CERD’s emerging and long term needs/priorities given the Syrian refuge crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 a. Was CERD consulted with when USAID was designing the QITABI project? If yes, please describe the nature of CERD’s involvement in the design of the project. b. If no, please tell us what effects, if any, it has had on QITABI’s implementation? EFFECTIVENESS: 5. In your opinion, are the project interventions making adequate progress toward achieving the project’s results? a. If yes, please give us examples of the interventions and how they are contributing to achieving the project’s results. If no, please tell us why not? b. Please describe ways in which QITABI activities are supporting public schools in Lebanon? Examples of some the key interventions? Component 1 • EGRA baseline • Assessment of teachers’ knowledge of reading instruction • EGR training and coaching (CERD, DOPS, teachers) • Development of reading benchmarks Component II • Provision of resources/equipment to schools • Professional development • What else? Component III • What are C3 achievements thus far 6. In you opinion, what are some of the major achievements of QITABI thus far e. What are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE 72 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION f. Please describe the kind of support CERD provides to QITABI to achieve its planned results? - Role CERD - What else? 7. What kinds of challenges did/does QITABI in working toward those those achievements? - Working with MEHE and CERD - Working QITABI partners a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of project activities? b. Please share your understanding of how QITABI is addressing those challenges. Tell us about your experience of working QITABI. c. How would you describe coordination between QITABI and CERD? d. Does CERD face any challenges? If yes, what kind of challenges does MEHE face in supporting QITABI implementation? – Working with WL – Working with other implementing partners/service providers – Working with USAID – Overburden and overstretched MEH 8. In you opinion, has the implementation of project activities related to CERD such training been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If not, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – Professional development/ICT trainings – Psychosocial support to schools? – What else c. Please describe the factors that led to the delayed implementation. – QITABI’s approach to getting MEHE/CERD on board – Lack of/inadequate support from MEHE – Overwhelmed/overburdened MEHE – What else? 9. Please tell us about some of the key accomplishments of QITABI that benefited/will benefit CERD? g. What strategic plans, policies, manuals, curricula developed by QITABI are/will be adopted and used by CERD. Please give details h. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? – Policies, Strategy documents, professional development manuals, curricula and institutional support model developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 10. Did the project design/implementation include a specific strategy for including women? a. What steps is CERd taking to ensure participation of women? Explain? b. What steps is CERD taking to ensure male and female students equally benefited from the project activities? Examples of activities: – EGR activities – ICT training – Access to resource rooms, libraries, ICT equipment 11. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? i. Please describe how CERD coordinates with those projects? j. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 73 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ANA AQRA’A – Component 1 12. In your understanding, to what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? b. If yes, please describe the factors that contribute/will contribute to the sustainability of QITABI results? c. Please describe factors that might make QITABI results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks 13. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that key project activities and results are sustainable? a. How do you see the roles of CERD in the sustainability of the results? b. In you opinion, is CERD committed to institutionalizing EGR and assessment model including reading benchmarks and EWS? c. In your understanding, is CERD committed and/or does it have the resources to provide ICT support/training to public school teachers after the project ends? d. In your opinion, what kind of support will CERD need to continue project achievements beyond the life of the project? e. What are CERD’s plans between now and end of the project for making key project achievements sustainable? 14. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Implementation of EGR activities – Coordination with other implementation partners – Coordination with MEHE and CERD – Coordination with USAID RELEVANCE: 2. What is your understanding of the needs of public education in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 74 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how Component I addresses MEHE’s needs: C1: Improved reading outcomes for primary level public school – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 4. Please describe Ana Aqra’a’s support/contribution to the implementation of QITABI. a. Please tell us about the EGR teacher training modules. How were they developed and who was involved in their development— approach, content etc.? b. Please describe Ana Aqra’a’ role in training CERD and DOPS—e.g., selection of trainees, number of training sessions for each cohort, roll-out. c. Please tell about Ana Aqra’a’s contribution to the development of materials for in-class activities? How were these materials developed and who was involved in their development? d. What about classroom observation and the development of EWS? In your opinion how do they contribute to the improvement of reading achievement? e. Please describe the process through which Ana Aqra’a provides coaching to teachers in EGR instruction. f. Could tell us about the implementation of NFE through mobile buses. How do you select who receives NFE through this activity? How does it contribute to the project results? g. Has Ana Aqra’a conducted any parental awareness activities? If yes, please give us details. If no, tell us why not? h. To what extent were MEHE and CERD involved in the design and implementation of these C1 activities. Describe the process. 5. In your opinion, are the project interventions, particularly Component 1 activities, making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? Examples of some the key interventions? Component 1 • EGR training manuals • EGR training and coaching (CERD, DOPS, teachers) • EWS • Parental awareness activities • What else? 6. Please tell us about some of the major achievements of QITABI, particularly Component 1, thus far g. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 75 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 7. What kinds of challenges did/does QITABI/MSI encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of C1 activities? b. Please describe how QITABI is addressing those challenges? 8. Has the implementation of Component 1 activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of delayed activities: – ESW – Parent teacher associations/Parents circles meetings – Reading Benchmarks – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 9. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? e. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? 10. How would you describe coordination between Ana Aqra’a and other QITABI partners? a. Please give us example of coordination and collaboration between Ana Aqra’a and other QITABI partners (particularly between Ana Aqra’a and MSI). 11. Please tell us about some of the key accomplishments of QITABI (Component 1) that benefited/will benefit MEHE? i. What strategic plans, policies, manuals, data and curricula developed by QITABI are/will be adopted and used by MEHE and CERD. Please give details j. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. – Balanced Literacy Approach (BLA) – National EGR benchmarks – Early Warning System (ESW) 12. To what extent are the planned Component 1 interventions of sufficient for achieving the desired results? f. Please describe how these results will contribute toward achieving the project objective? Examples? - Improve reading outcomes 13. Did the project design/implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefit from the project activities? c. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C1 activities? Examples of activities: – EGR activities – Classroom observations – Access to resource rooms – Access to classroom libraries – Access to ICT equipment 14. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes in Lebanon? - The World Bank (RACE II) - UNICEF - GIZ 76 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION k. Please describe how Ana Aqra’a coordinates with those projects? l. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - DFID SUSTAINABILITY 15. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of Component 1 results? c. Please describe factors that might make Component 1 results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – CERD and DOPS training – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 16. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that Component 1 activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, are MEHE and CERD committed to institutionalizing the EGR model, including EWS? c. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 17. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming 77 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION AMIDEAST – Professional Development Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – Ensuring timely implementation of project activities – Coordination with WL and other implementation partners – Coordination with MEHE and CERD – Coordination with USAID RELEVANCE: 2. What is your understanding of the needs of public education in Lebanon? a. Please tell us ways in which the Syrian refugee crisis is affecting public education services in the country. b. In your opinion, has the refugee crisis had an effect on MEHE’s priorities? If yes, in what ways. If no, tell us why not? – MEHE’s needs in relation to provision of public education services to (a) Lebanese population and (b) Syrian refugee children 3. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how Component 2 addresses MEHE’s needs: C2: Expand access to safe and relevant education for vulnerable public school students – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 4. In your opinion, are the project interventions, particularly Component 2 activities, making adequate progress toward achieving the planned results? a. If yes, please tell us about the C2 interventions that AMIDEST is implementing and how are they contributing to achieving the project’s planned results. If no, please tell us why not? b. Please tell us QITABI’s approach to building CERD and DOPS capacity through C2? c. Please describe ways in which providing ICT equipment and professional development activities may increase access to education for vulnerable children? Examples of some the key interventions? Component 1 • ICT equipment for 300 schools • Professional Development in ICT and Psychosocial support for DOPS • ICT training/TOT for CERD • Working with school leadership to strengthen management capacity • What else? 5. Please tell us about some of the major achievements of QITABI thus far a. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of QITABI partners - Role of USAID - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does QITABI encounter in working toward those achievements? - Working with MEHE/CERD - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. What kinds of challenges does the prevalence of refugee children in primary schools pose for the implementation of C2 activities? b. Please describe how QITABI is addressing those challenges? 78 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 7. Has the implementation of Component 2 activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of activities: – ICT equipment – ICT training (DOPS and CERD) – Psychosocial support – Strengthening management capacity at school level – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Overburdened MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? f. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? g. How would you describe coordination between QITABI, and MEHE and CERD? Successes? Challenges? – 9. Please tell us about some of the key accomplishments of QITABI (Component 2- professional development) that benefited/might benefit MEHE? a. What strategic plans, policies, manuals, and curricula developed by QITABI are/will be adopted and used by MEHE and CERD. Please give details b. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. – ICT equipment – Professional development training and manuals, framework. – What else? 10. To what extent are the planned Component 2 interventions of sufficient for achieving the desired results? (For example, can provision of ICT equipment to school help expanding access to education?) a. Please describe how these results will contribute toward achieving the project objective? Examples? – Expand access to safe and relevant education 11. Does the component design/implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefit from the C2 activities? c. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities Examples of activities: – ICT equipment – Psychosocial support (girls’ needs Vs boys’ needs) – Access to resource rooms 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? a. Please describe how QITABI coordinates with those projects? b. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 79 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION AMIDEAST MIS Specialist 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of Component 2 results? c. Please describe factors that might make Component 2 results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – CERD and DOPS ICT training – Transportation (school buses) – ICT equipment – Non-formal education 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that Component 2 activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, are MEHE and CERD committed to institutionalizing the ICT in education and ICT professional development models? c. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming Questions Prompts 1. Could you describe your role within your organization and your involvement in QITABI? a. How long have you served in this position? b. How long have you been involved with QITABI and in what ways? – Responsibilities – Data collection and management – Tracking progress – Progress report and M&E plans – Coordination with implementation partners RELEVANCE: 2. What is your understanding of the purpose/objective of QITABI and ways in which the project interventions address MEHE’s needs? a. Please describe how Component 2 addresses MEHE’s needs: C2: Expand access to safe and relevant education for vulnerable public school students – MEHE’s emerging and long term needs/priorities given the Syrian refugee crisis – RACE II & MEHE’s Education Sector Plan (2010-2015) – USAID’s 2011-2015 Education Strategy – USAID’s Country Development Cooperation Strategy (CDCS) 2014-2019 EFFECTIVENESS: 80 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 3. In your opinion, are the C2 interventions making adequate progress toward achieving the planned results? a. If yes, please tell us about those interventions and how are they contributing to achieving the project’s planned results. If no, please tell us why not? Examples of some the key interventions? Component 1 • ICT equipment for 300 schools • Professional Development in ICT and Psychosocial support for DOPS • ICT training/TOT for CERD • Working with school leadership to strengthen management capacity • What else 4. Please tell us about the AMIDEAST’s M&E and data management mechanisms. a. Please tell us how QITABI monitors project activities? b. Please describe how QITABI tracks progress? c. What kind of M&E support do you provide to implementing partner? d. What support do you receive from the implementing partner in tracking progress/monitoring project activities? 5. Please tell us about some of the major achievements of Component 2 thus far h. In your opinion, what are the factors that played key roles in accomplishing those achievements? - QITABI leadership? - Role of AMIDEAST - Role of USAID - Role MEHE - Role CERD - What else? 6. What kinds of challenges did/does AMIDEAST encounter in working toward those achievements? - Working with MEHE - Working with other implementing partners/service providers (explore coordination Vs collaboration) - Working with USAID a. Please describe how AMIDEAST is addressing those challenges? 7. Has the implementation of all Component 2 activities been timely? a. If yes, please tell us about the factors that contribute/d to the timely implementation b. If no, please tell us about the activities whose implementation did not take place according to the project work plans. A few examples of activities: – Professional development/ICT trainings – ICT equipment – Psychosocial support to schools? – What else c. Please describe the factors that led to the delayed implementation. – Lack of/inadequate support from MEHE – Lack of/inadequate support from CERD – Strained/Overwhelmed MEHE – What else? 8. In your opinion, will the implementation of the delayed activities be completed before the project ends next year? If yes, how? a. What steps, if taken, will help QITABI to efficiently and effectively implement those activities? Who should take these steps? b. How would you describe coordination between QITABI, and MEHE and CERD? Successes? Challenges? – 9. Please tell us about some of the key accomplishments of Component 2 that benefited/will benefit MEHE? k. What strategic plans, policies, manuals, and curricula developed by QITABI are/will be adopted and used by MEHE. Please give details Explore the process through which these were developed: – Policies, Strategy documents, professional development manuals/curricula developed. 81 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Focus Group Discussion Protocols l. Please describe ways in which public schools that are overwhelmed by non-Lebanese students may benefit from the project? – CERD and DOPS capacity building in ICT (strategy and manuals) – Provision of ICT equipment to schools 10. To what extent are the planned interventions of QITABI sufficient for achieving the desired results? (For example, can provision of ICT equipment to school help expanding access to education?) g. Please describe how these results will contribute toward achieving the project objective? Examples? - Expand access to safe and relevant education 11. Did the project design/implementation include a specific strategy for including women? a. What steps were/are being taken to ensure participation of women? Explain? b. What steps were/are being taken to ensure male and female students equally benefit from the project activities? c. What steps were/are taken to ensure Lebanese and Non-Lebanese students equally benefit from C2 activities Examples of activities: – Access to ICT equipment – Psychosocial support (girls’ needs Vs boys’ needs) – Access to resource rooms 12. Do you have knowledge of other education programs implemented since 2014 that support improvement in learning outcomes and equitable access to education in Lebanon? m. Please describe how AMIDEAST coordinates with those projects? n. Please explain ways in they might be contributing directly or indirectly toward QITABI’s achievements? - The World Bank (RACE II) - UNICEF - GIZ - DFID SUSTAINABILITY 13. To what extent does the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Which ones? b. If yes, please describe the factors that contribute/will contribute to the sustainability of Component 2 results? c. Please describe factors that might make Component 2 results less sustainable. – Policies, Strategy documents, professional development manuals/curricula developed. – EGR/Balanced Literacy Approach (BLA) – National EGR benchmarks – EWS 14. Between now and the end of the project, what steps must QITABI take and what resources should it mobilize to ensure that Component 2 activities and results are sustainable? a. How do you see the roles of MEHE and CERD in the sustainability of the results? b. In you opinion, are MEHE and CERD committed to institutionalizing the ICT in education and ICT professional development models? c. In your opinion, what kind of support will MEHE and CERD need to continue project achievements beyond the life of the project? 15. If you were to redo/redesign the project, what improvements would you introduce? – Suggestions/recommendations for improving future programming 82 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION Focus group discussions with public school TEACHERS who received EGR Training (1 or II or both) 1. How long have you been teaching in public schools? Which subjects? What grades? 2. Under QITABI project, how much training have you received? a. How long was the training and when was it held? b. Please describe what the training focused on (contents)—Assessment? Reading? Early warning system (EWS)? What else? 3. Could you briefly describe how the [EGR] training addresses the needs of your work/teaching (relevance)? 4. Before you received the training, how familiar were you with the topic/content of the training? a. Oral reading assessment techniques? b. Teaching reading? 5. Did the training increase your knowledge of assessment and teaching reading following this training? a. If yes, how? If no, why not? 6. What did you gain as a result of your participation in this training? b. How has it affected your teaching? 7. Can you give examples how you are applying the knowledge and skills from the training in the classroom? a. Could you describe how you assess students’ reading skills? b. How often do you assess your students’ reading skills? c. Could you describe ways in which you teach reading? d. How often do you use the EGR TLM material? 8. Could you describe how you identify students who are struggling in reading through EWS? 9. What do you do once you identify struggling students? Please describe the steps/process. 10. What kind of support do you receive from (Please describe and give examples) a. EGR facilitators? b. DOPs? 11. Are you member of any social media-based teacher network/learning circle? If yes, what kind of support do you receive from the network/circle support? If not a member, why not? 12. Do you see any change in your students’ reading skills as a result of EGR training? If yes, please give details/examples. If no, why not? 13. Did the training equally address the needs of a. Female and male students? If yes, how do you ensure that boys’ and girls’ reading shortcomings are equally attended to/addressed? If no, explain why not? b. Lebanese and non-Lebanese children? If yes how do you ensure that the reading shortcomings of Lebanese and non-Lebanese children are equally attended to/addressed? If no, explain why not? 14. In the future, how do you anticipate using the knowledge and skills that you gained as a result of your participation in the training? Please describe. 15. What are the key challenges in applying in the classroom what you learned through the training? a. Lack of EGR (TLM) material b. Inadequate training received c. Lack of cooperation from school administration/principal d. Overcrowded school/classroom 83 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 16. What suggestions do you have for improving this program/activity in the future? 17. What other support has your school received from QITABI? a. Classroom libraries? i. Please tell us how you use them? If no, why not? b. ICT Equipment for classrooms? i. Please tell us how you use them? If no, why not? c. ICT resource rooms? i. Please tell us how you use them? If no, why not? 18. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with public school TEACHERS who received EGR Coaching only (No EGR Training) 1. How long have you been teaching in public schools? Which subjects? What grades? 2. Under QITABI project, how much coaching support have you received? a. How long did the coaching support last—when did start and end? Who coached you? b. Please describe what the coaching focused on—Assessment? Reading? Early warning system (EWS)? What else? 3. Could you briefly describe how the EGR coaching addresses the needs of your work/teaching (relevance)? 4. Before you received the coaching support, how familiar were you with the topic/content of the training? a. Oral reading assessment techniques? b. Teaching reading? 5. Did the training increase your knowledge of assessment and teaching reading? c. If yes, how? If no, why not? 6. Did coaching affect your teaching? If yes, please describe how. If no, why not? 7. Can you give examples of how you are applying the knowledge and skills from the EGR coaching in the classroom? a. Could you describe how you assess students’ reading skill? b. How often do you assess your students’ reading skills? c. Could you describe ways in which you teach reading? d. How often do you use the EGR TLM material? 8. Could you describe how you identify students who are struggling in reading through EWS? 9. What do you do once you identify struggling students? Please describe the steps/process. 10. What kind of support do you receive from (please describe and give examples) a. EGR facilitators? b. DOPs? 11. Are you member of any social media-based teacher network/learning circle? If yes, what kind of support do you receive from the network/circle support? If not a member, why not? 12. Do you see any change in your students’ reading skills as a result of EGR coaching? If yes, please give details/examples. If no, why not? 84 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 13. Did the training equally address the needs of a. Female and male students? If yes, how do you ensure that boys’ and girls’ reading shortcomings are equally attended to/addressed? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how do you ensure that the reading shortcomings of Lebanese and non-Lebanese children are equally attended to/addressed? If no, explain why not? 14. In the future, how do you anticipate using the knowledge and skills that you gained as a result of the training in the EGR coaching? Please describe. 15. What are the key challenges in applying in the classroom what you learned through the coaching? a. Lack of EGR (TLM) material b. Inadequate coaching received c. Lack of cooperation from school administration/principal d. Overcrowded school/classroom 16. What suggestions do you have for improving this program/activity in the future? 17. What other support has your school received from QITABI? a. Classroom libraries? ii. Please tell how you use them? If no, why not? b. ICT Equipment for classrooms? ii. Please tell us how you use them? If no, why not? c. ICT resource rooms? ii. Please tell us how you use them? If no, why not? 18. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with public school PRINCIPALS whose schools received classroom libraries and ICT equipment, and resource rooms. 1. How long have you been a principal? Which school? 2. Under QITABI project, what kind of support has your school been receiving? a. Classroom ICT equipment? If yes, please tell us what equipment your school received. b. ICT resource room? If yes, please tell us what resources/equipment are in the resource room that was provided to your school. c. Classroom libraries? If yes, please tell us what a classroom library entails. d. EGR and ICT training? (if the response to any of the above is no, please probe why not?) 3. Could you briefly describe how the following resources provided by QITABI address the needs of your school? (Please encourage the FGD participant to talk about each of the following resources). a. Classroom ICT equipment b. ICT resource room c. Classroom libraries d. Training (ICT and EGR) 4. Before you received these resources from QITABI, please tell how ICT was used in your school? 5. What changes have you observed in your school since the school started receiving support (classroom ICT equipment and ICT resource room) from QITABI? Changes in: i. School environment? ii. Student enrollment? iii. Student attendance? 85 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION iv. Managing and accommodating overcrowded classroom? v. Student learning achievement? vi. Teaching practices? vii. What else? 6. Could you give examples that show how the classroom ICT equipment and resource room are used in your school? a. How are your teachers and students benefiting from these resources? 7. Could you tell us if you notice changes in your school as a result of the classroom libraries that QITABI provided? If yes, please describe those changes. a. Classroom environment? b. Student attendance? c. Student learning achievement? d. Teaching practices? e. Managing and accommodating overcrowded classroom? f. What else? 8. Could you give examples of how your teachers and students are benefiting from the classroom libraries? 9. Do you see any change in your students’ reading skills as a result of EGR training and EGR TLM? If yes, please give details/examples. If no, why not? 10. Do male and female children have equal access to the classroom ICT equipment, resource room and classroom libraries? If, yes please tell us how? 11. Do Lebanese and non-Lebanese children have equal access to these resources? If yes, how? If no, explain why not? 12. In relation to QITABI interventions, what kind of support does your school receive from (please describe and give examples) a. EGR facilitators? b. DOPs? 13. How do you anticipate using the resources provided by QITABI in the future in your school? Please describe. 14. What are the key challenges in using these resources in your school (Please give details)? a. Lack of EGR (TLM) material? b. Lack of or inadequate ICT training? c. Lack of cooperation from MEHE and/or CERD? d. Overcrowded school/classroom? 15. Please tell us if the inflow of non-Lebanese population (refugees) has any effect on your school? a. If yes, please tell us in what ways it has affected it? b. How are you addressing the challenges (that resulted from overcrowded classrooms)? c. In what ways have QITABI interventions in your school addressed some of these challenges? 16. What suggestions do you have for addressing the challenges you highlighted? a. Additional training? b. Additional equipment? c. Parental engagement? d. Transportation for vulnerable (refugee and Lebanese) children? e. Type of support from QITABI? MEHE? CERD? 17. What other support has your school received from QITABI? Please describe. 86 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 18. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with QITABI Trainers (EGR TOT and CB) 1. Please tell us about your work. a. How long have you been working with QITABI and in what capacity? b. What does you work entail? 2. Please describe the EGR TOT/Capacity building training you provided the trainers. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what the training focused on: i. Teaching early grade reading ii. Oral reading assessment techniques? iii. Early warning system (EWS)? iv. Classroom observation v. What else? 3. Please tell us if you have ever administered an early grade reading assessment (EGRA) to students (during or after the coaching/training). If yes, please give us details—when? How many? How long on average does an EGRA last? 4. Could you briefly describe how the EGR TOT/capacity building training addresses the needs of a. The teachers/schools? b. CERD and DOPS? 5. Did the training influence trainers’ knowledge of EGR, and EGR assessment? a. If yes, please describe ways in which the training enabled to train CERD and DOPS? b. If no, tell us why not? 6. Tell us about the kind of feedback you receive from from these trainers regarding how they apply in their training what they learned in the training you provided? Examples? 7. Could you tell us what challenges they face applying the knowledge and skills they gained from the training in providing training to CERD and DOPS? a. Lack of EGR reference material? b. Lack of cooperation from CERD/DOPS c. What else? 8. In your opinion, did EGR training equally address the needs of a. Female and male students? If yes, how does the training help teachers to equally address boys’ and girls’ reading shortcomings? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how does the training help teachers to equally address the reading shortcomings of Lebanese and non-Lebanese children? If no, explain why not? 9. What challenges did you face in providing the training to these trainers? 10. In the future, how do you anticipate CERD trainers/DOPS coaches using the knowledge and skills that they gained as a result of the EGR training/coaching? Please describe. a. In your opinion, do they need any kind of support to continue using the knowledge and skills you gain in your work? If yes, please tell us what kind of support and who you expect to provide them that support—QITABI? CERD? MEHE b. What suggestions do you have for improving the training activity in the future? 87 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 11. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with QITABI Trainers (EGR Training CERD/EGR Coaching DOPS) 1. Please tell us about your work. a. How long have you been working with QITABI as a trainer? b. What does your work entail? c. How long have you been providing EGR training/coaching support to CERD/DOPS? 2. Please describe the EGR training/coaching you conducted. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what the training focused on: i. Teaching early grade reading ii. Oral reading assessment techniques? iii. Early warning system (EWS)? iv. Classroom observation v. What else? 3. Please tell us if you have ever administered an early grade reading assessment (EGRA) to students (during or after the coaching/training). If yes, please give us details—when? How many? How long on average does an EGRA last? 4. Could you briefly describe how the EGR training/coaching addresses the needs of the teachers/schools? 5. Did the training influence CERD/DOPS representatives’ knowledge of EGR, and EGR assessment? a. If yes, please describe ways in which the training enabled them to train and coach teachers? b. If no, tell us why not? 6. Describe to us the kind of support you receive from QITABI’s EGR facilitators (For DOPS participants only). 7. Tell us about the kind of feedback you receive from CERD/DOPS regarding how they apply in their training/coaching what they learned in the ERG training/coaching you provided? Examples? 8. Could you tell us what challenges CERD/DOPS face applying the knowledge and skills they gained from EGR training/coaching in training teachers? a. Lack of EGR reference material? b. Inadequate EGR training they received? c. Lack of cooperation from school administration/principal? d. Lack of support from DOPS/CERD/QITABI e. What else? 9. In your opinion, did EGR training equally address the needs of a. Female and male students? If yes, how does the training help teachers to equally address boys’ and girls’ reading shortcomings? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how does the training help teachers to equally address the reading shortcomings of Lebanese and non-Lebanese children? If no, explain why not? 10. What challenges did you face in providing EGR training to CERD/DOPS? 11. In the future, how do you anticipate CERD trainers/DOPS coaches using the knowledge and skills that they gained as a result of the EGR training? Please describe. a. In your opinion, do they need any kind of support to continue using the knowledge and skills you gain in your work? If yes, please tell us what kind of support and who you expect to provide them that support—QITABI? CERD? 88 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION b. What suggestions do you have for improving the training activity in the future? 12. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with QITABI EGR Facilitators 1. Please tell us about your work. a. How long have you been working with QITABI as an EGR facilitator? b. How many schools do you cover and how much time on average do you dedicate to each school per week or month? c. What does you work entail? d. How long have you been providing EGR support to DOPS and teachers? 2. Please describe the EGR support/coaching you provide to DOPS and teachers? a. Please describe what the EGR support that you provide typically focuses on: i. Teaching early grade reading? ii. Oral reading assessment techniques? iii. Early warning system (EWS)? iv. Classroom observation? v. Use of ICT in classrooms? vi. What else? 3. Please tell us if you have ever administered an early grade reading assessment (EGRA) to students (during or after the coaching/training). If yes, please give us details—when? How many? How long on average does an EGRA last? 4. Could you briefly describe how the EGR support/coaching addresses the needs of the teachers/schools? 5. Did the coaching influence teachers’ knowledge of EGR, and EGR assessment? a. If yes, please describe ways in which the EGR coaching enabled them to teach reading. b. Please describe ways in which the EGR coaching enabled them to assess students’ reading skills. c. If no, tell us why not? 6. How would you describe working/collaborating with DOPS? a. Please tell us about the support you provide EGR support to DOPS coaches. Describe the process. 7. Tell us about the kind of feedback you receive from DOPS regarding how they apply in their work (i.e., coaching teachers) what they learned from receiving EGR support from EGR facilitators? 8. Could you tell us what challenges DOPS face applying the knowledge and skills they gained from EGR support that you provide? 9. Could you tell us what challenges teachers face applying the knowledge and skills they gained from EGR support that you and DOPS provide? 10. In your opinion, did EGR support you provide equally address the needs of a. Female and male students? If yes, how does the it help teachers to equally address boys’ and girls’ reading shortcomings? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how does the training help teachers to equally address the reading shortcomings of Lebanese and non-Lebanese children? If no, explain why not? 11. What challenges do you face in providing EGR support to DOPS and teachers? a. Lack of EGR (TLM) material b. Inadequate training received 89 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION c. Lack of cooperation from school administration/principal d. Overcrowded school/classroom e. What else? 12. In the future, how do you anticipate teachers and DOPS coaches using the knowledge and skills that they gained from the EGR support that you provide in their work? Please describe. a. In your opinion, do they need any kind of support to continue using the knowledge and skills in their work? If yes, please tell us what kind of support and who you expect to provide them that support— QITABI? MEHE? CERD? b. What suggestions do you have for improving the training activity in the future? 13. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with DOPS (Professional Development -ICT Training) 1. Please tell us about your work. a. How long have you been working as a DOPS counselor/monitor? Which region/cazas do you focus on? b. What does you work entail? 2. Please describe the ICT training you received under QITABI. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what topics the training focused on: 3. Before you received the ICT training, how familiar were you with these topics? 4. Did the ICT training increase your knowledge of use of ICT in teaching? If yes, in what ways? If no, why not? 5. Did the TOT affect ways in which you provide ICT coaching/support to teachers? If yes, please give examples of how you are applying or plan to apply the knowledge and skills from the ICT training in training teachers? If no, 6. Could you describe how you coach/plan to coach teachers? Please describe the process. a. How many coaching sessions have you provided so far/expect to provide? b. How long does a typical ICT coaching session last? 7. In your opinion, does the ICT coaching have/will have any effect on teachers’ teaching practices? If yes, how. If no, why not? 8. Tell us about the kind of feedback you receive from teachers/schools regarding (please skip this question, if DOPS hasn’t yet started providing ICT coaching support to schools) a. How teachers apply in classroom what they learned ICT coaching? Examples? b. Changes in how teachers teach reading and assess students? Please give us examples. c. Changes in students’ learning achievement? Please describe the kind of changes teachers notice in students’ reading achievement. 9. In your opinion, did ICT training equally address the needs of a. Female and male students? If yes, how do you ensure that teachers equally address boys’ and girls’ ICT needs/shortcomings? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how do you ensure that teachers equally address the ICT needs/shortcomings of Lebanese and non-Lebanese children? If no, explain why not? 10. What challenges do you face in providing ICT coaching to teachers? 90 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 11. In the future, how do you anticipate using the knowledge and skills that you gained as a result of the ICT Training in coaching teachers? Please describe. a. Do you need any kind of support to continue using the knowledge and skills you gain in your work? If yes, please tell us what kind of support and who you expect to provide you that support—QITABI? MEHE? b. What suggestions do you have for improving the training activity in the future? 12. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with DOPS Counselors/Coaches (EGR CB) 1. Please tell us about your work. a. How long have you been working as a DOPS counselor/monitor? Which region/cazas do you focus on? How many schools do you cover/provide support to? b. What does you work entail? c. How long have you been providing EGR coaching/support to schools? 2. Please describe the capacity building/coaching training you received under QITABI. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what the training focused on: i. Coaching techniques ii. Early grade reading and assessment? iii. Early warning system (EWS)? iv. Classroom observation v. What else? 3. Please tell us if you ever administered an early grade reading assessment (EGRA) to students (during or after the coaching training). If yes, please give us details—when? how many? How long on average does an EGRA last? 4. Before you received the training, how familiar were you with these topics? 5. Did the training increase your knowledge of teaching EGR, and EGR assessment? If yes, in what ways? If no, why not? 6. Did the training affect ways in which you provide EGR support to teachers? If yes, please give examples of how you are applying the knowledge and skills from the EGR capacity building training in coaching teachers? If no, why not. 7. Could you describe how you observe teachers in classroom—describe the process? How often? 8. In your opinion, does EGR training have any effect on teachers’ teaching practices? If yes, how. If no, why not? a. Are teachers generally able to apply what they learned in the ERG training? Examples? b. Do you notice any change in how teachers teach reading and assess students? If yes please describe. If no, tell us why not. c. Do you observe any change in students’ reading achievement? If yes, please describe the kind of changes you notice in their reading achievement. If no, why not? 9. Do you see any difference between teaching practices of the teachers who received EGR training and those who only receive EGR coaching? If yes, please give us examples. 10. Could you tell us what challenges teachers face applying the knowledge and skills they gained from EGR training in their classrooms? 91 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION a. Lack of EGR (TLM) material b. Inadequate coaching received c. Lack of cooperation from school administration/principal d. Overcrowded school/classroom 11. Please describe what steps you take after observing teachers. a. Feedback to teachers? QITABI? MEHE? b. How do you assist/guide teachers who face challenges teaching reading and/or assessing student reading fluency? c. How long does a typical coaching session last? 12. Could you describe the nature of EGR assistance teachers/schools request from you? Please give examples? 13. In your opinion how often have you been able to successfully provide EGR coaching support to teachers? a. What challenges do you face in providing EGR coaching support to schools? 14. How do you describe working with EGR facilitators? a. How do you collaborate with them? How often? 15. Did EGR coaching equally address the needs of a. Female and male students? If yes, how do you ensure that teachers equally address boys’ and girls’ reading shortcomings? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how do you ensure that teachers equally address the reading shortcomings of Lebanese and non-Lebanese children? If no, explain why not? 16. In the future, how do you anticipate using the knowledge and skills that you gained as a result of the EGR capacity building training? Please describe. a. Do you need any kind of support to continue using the knowledge and skills you gain in your work? If yes, please tell us why kind of support and who you expect to provide you that support—QITABI? MEHE? b. What suggestions do you have for improving the coaching activity in the future? 17. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with CERD /DOPS (EGR Analysis and SPSS Capacity Building) 1. Please tell us about your work. a. What’s your professional title (e.g., CERD statistician, Arabic language specialist) and how long have you been working at CERD/DOPS? b. What does you work entail? 2. Please describe the SPSS Training Workshop you attended under QITABI. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what the training focused on: i. Descriptive statistics ii. Inferential statistics iii. Anything else? 3. Please describe the EGRA Analysis Training Workshop you attended. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what the training focused on: i. Assessing reliability and validity (psychometric properties) of EGRA instruments? ii. Running statistical analysis used for EGRA reporting? iii. Interpretations of the results? iv. What else? 92 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION 4. Could you briefly describe how the SPSS and EGR analysis training workshops addressed the needs of you work? 5. Before you received the SPSS and EGRA analysis training, how familiar were you with these topics? 6. Did these workshops increase your capacity to a. Use SPSS for data analysis? If yes, in what ways? If no, why not? b. Analyze EGRA data and interpret results? If yes, please tell us how? If no, why not? 7. Could you give examples of how you are applying the knowledge and skills from these workshops in your work? a. Could you tell if you can analyze EGRA data without substantial support from others? If yes, tell us if you have analyzed EGRA data independently, and when and how many times? 8. Could you tell us what challenges you face applying the knowledge and skills gained from SPSS and EGR analysis-training workshops? a. Lack of SPSS reference/EGRA analysis material b. Inadequate training c. Lack of cooperation from DOPS/CERD d. What else? 9. In the future, how do you anticipate using the knowledge and skills that you gained as a result of your participation in these workshops? Please describe. a. Do you need any kind of support to continue using the knowledge and skills you gain in your work? If yes, please tell us why kind of support and who you expect to provide you that support—QITABI? CERD? b. What suggestions do you have for improving the training activity in the future? 10. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with CERD Trainers (EGR TOT) 1. Please tell us about your work. a. How long have you been working as a CERD trainer? Which region/cazas do you focus on? b. What does you work entail? c. How long have you been providing EGR training to teachers? 2. Please describe the EGR training/TOT you received under QITABI. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what the training focused on: i. Teaching early grade reading ii. Oral reading assessment techniques? iii. Early warning system (EWS)? iv. What else? 3. Please tell us if you have ever administered an early grade reading assessment (EGRA) to students (during or after the coaching training). If yes, please give us details—when? How many? How long on average does an EGRA last? 4. Could you briefly describe how the EGR training addresses the needs of the teachers/schools? 5. Before you received the training/TOT, how familiar were you with these topics? 6. Did the EGR TOT increase your knowledge of teaching EGR, and EGR assessment? If yes, in what ways? If no, 93 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION why not? 7. Did the TOT affect ways in which you provide EGR training to teachers? If yes, please give examples of how you are applying the knowledge and skills from the EGR TOT in training teachers? If no, why not. 8. Could you describe how you conduct the roll-out EGR training? Please describe the process. a. How many training sessions have you provided so far? b. How long a typical EGR teacher training last? 9. In your opinion, does EGR training have any effect on teachers’ teaching practices? If yes, how. If no, why not? 10. Tell us about the kind of feedback you receive from teachers/schools regarding a. How teachers apply in classroom what they learned in the ERG training? Examples? b. Changes in how teachers teach reading and assess students? Please give us examples. c. Changes in students reading achievement? Please describe the kind of changes teachers notice in students’ reading achievement. 11. Could you tell us what challenges teachers face applying the knowledge and skills they gained from EGR training in their classrooms? a. Lack of EGR (TLM) material b. Inadequate coaching c. Lack of cooperation from school administration/principal d. Lack of support from DOPS e. Overcrowded school/classroom f. What else? 12. In your opinion, did EGR training equally address the needs of a. Female and male students? If yes, how do you ensure that teachers equally address boys’ and girls’ reading shortcomings? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how do you ensure that teachers equally address the reading shortcomings of Lebanese and non-Lebanese children? If no, explain why not? 13. What challenges do you face in providing EGR training to teachers? 14. In the future, how do you anticipate using the knowledge and skills that you gained as a result of the EGR TOT? Please describe. a. Do you need any kind of support to continue using the knowledge and skills you gain in your work? If yes, please tell us why kind of support and who you expect to provide you that support—QITABI? CERD? b. What suggestions do you have for improving the training activity in the future? 15. Is there anything you would like to tell us about QITABI that we haven’t asked? Focus group discussions with CERD Trainers (ICT TOT) 1. Please tell us about your work. a. How long have you been working as a CERD trainer? Which region/cazas do you focus on? b. What does you work entail? c. How long have you been providing ICT training? 2. Please describe the ICT training/TOT you received under QITABI. a. How long did the training last—when did it start and end? Who trained you? b. Please describe what topics the training focused on: 3. Before you received the ICT training/TOT, how familiar were you with these topics? 4. Did the ICT training/TOT increase your knowledge of use of ICT in teaching? If yes, in what ways? If no, why 94 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION not? 5. Did the TOT affect ways in which you provide ICT training to teachers? If yes, please give examples of how you are applying or plan to apply the knowledge and skills from the ICT TOT in training teachers? If no, why not. 6. Could you describe how you conduct/plan to conduct the roll-out ICT training? Please describe the process. a. How many training sessions have you provided so far/expect to provide? b. How long a does typical ICT training last? 7. In your opinion, does the ICT training have/will have any effect on teachers’ teaching practices? If yes, how. If no, why not? 8. Tell us about the kind of feedback you receive from teachers/schools regarding (please skip this question, if roll-out hasn’t yet started) a. How teachers apply in classroom what they learned in the ICT training? Examples? b. Changes in how teachers teach reading and assess students? Please give us examples. c. Changes in students’ learning achievement? Please describe the kind of changes teachers notice in students’ reading achievement. 9. In your opinion, did ICT training equally address the needs of a. Female and male students? If yes, how do you ensure that teachers equally address boys’ and girls’ ICT needs/shortcomings? If no, explain why not? b. Lebanese and non-Lebanese children? If yes, how do you ensure that teachers equally address the ICT needs/shortcomings of Lebanese and non-Lebanese children? If no, explain why not? 10. What challenges do you face in providing the ICT training to teachers? 11. In the future, how do you anticipate using the knowledge and skills that you gained as a result of the ICT TOT? Please describe. a. Do you need any kind of support to continue using the knowledge and skills you gain in your work? If yes, please tell us what kind of support and who you expect to provide you that support—QITABI? CERD? b. What suggestions do you have for improving the training activity in the future? 12. Is there anything you would like to tell us about QITABI that we haven’t asked? 95 | FINAL EVALUATION REPORT USAID/LEBANON QITABI MIDTERM PERFORMANCE EVALUATION ANNEX 4: QITABI INTENDED RESULTS/SCOPE OF WORK Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA 22201 Suite 1000 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socialimpact.com