October 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Jehanzaib Khan, Jade Lamb, Rana Taher, Tamara Nasreddine, and Hania Chahal of Social Impact. DEVELOPING REHABILITATION ASSISTANCE TO SCHOOLS AND TEACHERS IMPROVEMENT (D-RASATI 2) FINAL PERFORMANCE EVALUATION REPORT DEVELOPING REHABILITATION ASSISTANCE TO SCHOOLS AND TEACHERS IMPROVEMENT (D-RASATI 2) FINAL PERFORMANCE EVALUATION Cover Photo: Public School Teachers during English Language Teaching Methodology (Photo courtesy of D-RASATI 2). DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available in printed or online versions. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract/Order No. AID-268-C-15-00001 ACRONYMS ADS Automated Directives System ALLC American Lebanese Language Center AMIDEAST America-Mideast Educational and Training Services, Inc. AUB American University of Beirut CAE Cambridge English: Advanced CERD Center for Educational Research and Development COP Chief of Party COR Contract Officer Representative DEC Development Experience Clearinghouse DO Development Objective DOT Digital Opportunity Trust D-RASATI 2 Developing Rehabilitation Assistance to Schools and Teachers Improvement 2 DRASATI I Developing Rehabilitation Assistance to Schools and Teachers Improvement 1 ECA Extracurricular Activities ESDP Education Sector Development Plan ELTM English Language Teaching Methodology ESP Effective School Profile EQUI Evaluation Quality Use and Impact ET Evaluation Team FCE Cambridge English: First FGD Focus Group Discussion GOL Government of Lebanon ICT Information Communication Technology IPs Implementing Partners IR Intermediate Result LDP Leadership Development Program LOE Level of Effort LU Lebanese University KII Key Informant Interview M&E Monitoring & Evaluation MEHE Ministry of Education and Higher Education PCELT Professional Certificate in English Language Teaching PSS Psychosocial Support RNA Rapid Needs Assessment SIP School Improvement Program SOW Statement of Work TOT Training of Trainers USAID United States Agency for International Development WL World Learning i TABLE OF CONTENTS EXECUTIVE SUMMARY.................................................................................................. 1 1. PROJECT BACKGROUND....................................................................................... 8 1.1. PROJECT DISCRIPTION..................................................................................................... 8 1.2. DEVELOPMENT HYPOTHESIS ......................................................................................... 10 2. EVALUATION PURPOSE AND QUESTIONS .................................................... 11 3. METHODS AND LIMITATIONS ........................................................................... 12 3.1. DESK REVIEW.................................................................................................................... 12 3.2. QUALITATIVE DATA COLLECTION.............................................................................. 12 3.3. LIMITATIONS .................................................................................................................... 14 4. FINDINGS AND CONCLUSIONS......................................................................... 16 4.1. FINDINGS .......................................................................................................................... 16 4.1.1. RELEVANCE.................................................................................................. 16 4.1.2. EFFEECTIVENESS ........................................................................................ 19 4.1.3. SUSTAINABILITY......................................................................................... 28 5. RECOMMENDATIONS AND LESSONS LEARNED.............................................. 32 ANNEX I: INCEPTION REPORT ................................................................................. 34 ANNEX 2: DOCUMENTS REVIEWED ........................................................................ 63 ANNEX 3: DATA COLLECTION INSTRUMENTS.................................................... 64 ANNEX 4: DRASATI II INTENDED RESULTS/SCOPE OF WORK......................... 94 LIST OF FIGURES Figure 1: DRASATI 2 Activity Locations...............................................................................iii LIST OF TABLES Table 1: FGD Sample............................................................................................................... 13 Table 2: English Language Training Beneficiaries ............................................................. 21 Table 3:Psychosocial Support Training Beneficiaries....................................................... 24 Table 4: Leadership Development & School Improvement Program Beneficiaries . 26 ii Figure 1: D-RASATI 2 Activity Locations iii EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This final performance evaluation assesses the relevance of the Developing Rehabilitation Assistance to Schools and Teachers Improvement (D-RASATI 2) activities to the needs of the public education sector, the effectiveness in achieving its planned results, and the sustainability of project activities. It provides lessons and recommendations that will assist USAID/Lebanon to make decisions about the type of approach the Mission should adopt in future assistance to the Ministry of Education and Higher Education (MEHE). It will also impact the type and scope of possible future education projects. Secondary users include the D-RASATI 2 implementing partners, MEHE, and the Center for Educational Research and Development (CERD). The evaluation team addressed the following three questions: 1. How relevant is the D-RASATI 2 project to the needs of the public education sector in Lebanon? 2. How and to what extent has the project achieved planned results? 3. What is the likelihood that the results D-RASATI 2 has achieved are sustainable beyond the life of the project? PROJECT BACKGROUND The public education system in Lebanon faces multiple challenges to provide quality education, including a shortage of Information Communication Technology (ICT) in schools, lack of qualified English language teachers, and inadequate school facilities. The low quality of public education has increased the existing disparity between the rich and poor in accessing high quality education. In addition, public school graduates encounter fewer opportunities in the workforce relative to those graduating from private schools. Access to quality educational institutions at all levels has become restricted to those who can afford the high cost of private education. The recent Syrian refugee influx in the country has put further strain on the public school system and aggravated the provision of quality education. USAID/Lebanon designed the $24.5 million D-RASATI 2 project to address the underlying reasons for the public school system’s inability to provide high quality education. World Learning (WL) won the award in September 2013, and the original schedule ran from October 2013 to September 2015. Through subsequent contract modification, however, USAID extended the project’s life until December 31, 2016, increased the budget to $29 million and expanded the scope of work to include additional activities to target schools most affected by the emerging Syrian refugee crisis. D-RASATI 2 is a continuation of the DRASATI I project, which was designed as a 5-year, $75 million project to run from 2010-2015. USAID cancelled DRASATI I in 2013 and released a 1 solicitation for D-RASATI 2 that would complete DRASATI I’s remaining activities in the remaining timeframe with the remaining budget. D-RASATI 2’s overarching goal is to support MEHE to improve the performance of the Lebanese public school system by achieving the following objectives: 1. Improve the learning environment in public schools through the procurement of Information Communication Technology (ICT) equipment and ICT training. 2. Increase learning opportunities through English teacher training and extra‐curricular activities. 3. Increase stakeholder engagement in the public school through leadership development with parents and community involvement; and the development and implementation of School Improvement Plans for a pool of schools with up to $4,000 - grants per school. 4. Increase student access to higher quality instruction and learning environments through managing new pressures on classrooms in host schools. 5. Improve social cohesion among Lebanese and Syrian students and reduce tensions at the school level. To meet these objectives, D-RASATI 2 consists of three components: • Component 1: Provision of ICT equipment and computer labs to public schools • Component 2: Provision of in-service public school teacher training • Component 3: School leadership development with parent and community involvement EVALUATION DESIGN, METHODS AND LIMITATIONS This evaluation mainly relies on qualitative data from key informant interviews (KIIs) and focus group discussions (FGDs). Over a three-week period the evaluation team conducted 40 KIIs and 45 FGDs with project implementing partners, USAID, MEHE, CERD, teachers, and principals. During the analysis, the team systematically analyzed the qualitative data through summarizing FGD and KII notes by evaluation question, coding key themes, organizing these themes in a detailed analysis matrix, and developing findings based on these themes. The evaluation assessed the achievement of the project objectives by measuring the required outcomes. For example, improving the learning environment and increasing stakeholder engagement were measured by assessing the project’s ability to achieve particular outcomes (e.g., ICT procurement and installment, capacity building training, school improvement planning), as listed in Annex 6. Following the analysis, the evaluation team prepared a detailed outline of key findings, conclusions for each study question, and overall recommendations. The team triangulated data where possible from interviews and focus groups with quantitative data obtained from project reports and monitoring data. The public schools were on their summer break during the data collection for this evaluation. Consequently, the evaluation team was unable to conduct site visits or collect student level 2 data. Therefore, the evaluation relies on beneficiaries’ self-reports of the effects of project services and equipment on teacher classroom practices and student learning outcomes. In addition, the evaluation cannot confirm teachers’ claims that student learning actually improved as a result of the D-RASATI 2 project without student level data. The evaluation considers gender dimensions to the extent possible given data limitations. KEY FINDINGS Following are the key findings for each evaluation question by project component. Question 1: How relevant is the D-RASATI 2 project to the needs of the public education sector in Lebanon? Component 1: Provision of ICT equipment and computer labs to public schools Under Component I, D-RASATI 2’s activities focused on improving the learning environment in public schools by developing an ICT Action Plan, developing training manuals for the integration of ICT in teaching, and procuring and installing ICT equipment in 126 public schools. MEHE showed significant appreciation of the ICT Action Plan, ICT training modules, and noted that the plan and the modules responded to MEHE’s ICT most pressing needs as laid out in the National Educational Technology Strategic Plan (NETSP), developed under DRASATI I. In addition, MEHE recognized the importance and relevance of the equipment as a building block for the ICT Action Plan. Component 2: Provision of in-service public school teacher training D-RASATI 2 designed Component 2 interventions to increase learning opportunities, increase student access to higher quality instruction and improve social cohesion in public schools. These interventions focused on building CERD and teachers’ capacity through ICT coaching, developing manuals, and providing capacity building trainings in English language, Extracurricular Activities (ECA), and Psychosocial Support (PSS). MEHE generally considered the capacity building support under Component 2 to have addressed its priority needs except for the Extra Curricular Activities (ECA) and Psychosocial Support (PSS) trainings. While MEHE/CERD saw ECA as important, they did not prioritize it. As for the PSS training, it was not listed in the project Master Plan, and it was added to the project’s mandate through a modification without gaining buy-in from MEHE. MEHE was initially reluctant to consider PSS as part of D-RASATI 2. Component 3: School leadership development and parents and community involvement To increase stakeholder engagement in public schools, the project provided leadership development training to in-service public school principals, school improvement trainings to school representatives, and in-kind grants to schools for implementing school improvement plans (SIP). Since 2005, MEHE has faced challenges in securing funding to provide the Leadership Development Program (LDP) training to approximately 500 in-service principals. MEHE considers D-RASATI 2’s LDP support to have addressed its long-standing need. While SIP is not explicitly highlighted in MEHE’s Education Sector Development Plan, it considers it an extension of the LDP, which it is. Question 2: How and to what extent has the project achieved its planned results? 3 Component 1: Provision of ICT equipment and computer labs to public schools The project overachieved its ICT procurement target, adding an additional 24 schools to the initial target of 126 schools to receive ICT equipment. However, the procurement was significantly delayed due to challenges in getting approvals from MEHE for school selection, from USAID for procurement, and in delivery of the equipment. As a result, the time available for coaching teachers on the ICT equipment after delivery was insufficient to allow teachers to learn to use and integrate it into lessons before the coaching service provider’s contract ended. Despite the delay and short project life (initially two years), stakeholders across the board offered high appreciation for D-RASATI 2 leadership for effectively managing project activities and resolving issues. The project’s design did not include a pilot of the ICT equipment before full procurement. Problems were discovered after the equipment was distributed. In addition, although the selection of tablets was based on MEHE’s ICT strategy, the project did not pilot the tablet model of the ICT integration in schools to ensure that tablets were the most suitable type of ICT for classrooms. Generally, teachers are using the equipment and math, science, and geography teachers found the equipment to be particularly helpful in visualizing complex concepts and improving student learning. However, a number of teachers report that tablets are not effective for writing assignments, limiting their utility for certain subjects. Component 2: Provision of in-service public school teacher training The project exceeded its targets for English training and received positive feedback from trainees on improvements in language skills. The Professional Certificate in English Language Teaching (PCELT) was considered a gold standard of English trainings and was generally appreciated across the board, including for its student-centered learning approaches. The project succeeded in building CERD capacity to provide English language teaching methods training to public school teachers. Teachers’ ability to create more student-centered lesson plans and structured lessons reportedly allowed students to learn better. D-RASATI 2 did not use the same training of trainers (TOT) approach for ECA as for the other training tasks of the project. MEHE and CERD were resistant to ECA approaches that did not align with their understanding. In addition, there was a difference of opinion between D￾RASATI 2, and MEHE and CERD on how it should be implemented, so initial drafts of ECA strategy and modules went through a lengthy review and revision process, and resulted in almost a year-long delay. MEHE approved the strategy and modules in August 2016, allowing insufficient time to institutionalize ECA before the project closes. Similarly, PSS was significantly delayed and MEHE was slow to agree to work on it with D￾RASATI 2. However, MEHE and CERD appreciated the eventual implementation of PSS as it expanded their knowledge of PSS topics. After revising the PSS material to account for the Lebanese local conditions, the project trained CERD trainers who successfully rolled out the training to teachers from approximately 191 host schools. CERD has adopted the PSS training and included it in its current training plan. The PSS training material was the only material that included separate consideration of boys’ and girls’ needs. Overall, teachers considered male and female students’ needs to have been equally well addressed. 4 Component 3: School leadership development and parents and community involvement The LDP improved principals’ skills in management, delegation, teamwork, and ICT use. The training also improved their relationships with teachers. Similarly, SIP built on the LDP skills, acquired under the 2005 LDP training, through helping principals and schools to conduct needs assessments, work as teams, and implement small improvement projects in their schools. Question 3: What is the likelihood that the results D-RASATI 2 has achieved are sustainable beyond the life of the project? Component 1: Provision of ICT equipment and computer labs to public schools The ICT Action Plan for integrating ICT in schools, CERD’s improved ICT skills, and the modules were mainstreamed within the system for continued benefits to public education. As for the ICT equipment, MEHE does not have the funding to replace or expand equipment on a large scale once the current equipment becomes obsolete. Component 2: Provision of in-service public school teacher training CERD plans and has the capacity, built through TOTs, to take D-RASATI 2 capacity development materials particularly ICT, English language teaching methodology, and PSS forward as part of its official teacher trainings. At the time of evaluation, CERD was not yet ready to offer ECA training to teachers. Component 3: School leadership development and parents and community involvement The LDP is part of the MEHE system and Lebanese University has the capacity to implement the training. Providing regular LDP training to principals, however, requires funding which MEHE does not have and for which it depends on donor assistance. For SIP, school representatives plan to use their newly acquired skills for future projects, but identified funding as the biggest challenge CONCLUSIONS MEHE and CERD had buy-in for D-RASATI 2’s plan and generally found it relevant to their priorities. This was especially true for the ICT Action Plan, English, LDP, and SIP. They also accepted the relevance of ECA and PSS, but were less fully invested at the start. Overall, D-RASATI 2 was able to achieve—and some cases overachieved—its intended results during a short project timeframe to meet its objectives. The project’s strong leadership and management, adherence to the Master Plan, and approach to working with MEHE and CERD resulted in successfully implemented activities. Given delays inherent in close collaboration with the MEHE, the short project life—initially two years—was insufficient for a project of this scope and scale. The project over-delivered English language training and ICT procurement. ICT coaching prior to the equipment arrival was appreciated by teachers but significant delays in procurement seriously undermined effectiveness, as coaches were unable to spend sufficient time with teachers specifically to help use the newly arrived equipment. To compensate for lost time, 5 DRASATI II added additional training for CERD, DOPS and champion teachers in the extension period. Despite obstacles, PSS was eventually effectively and successfully implemented as a TOT for CERD and MEHE, which CERD effectively rolled out. Teachers who participated in the trainings consequently saw preliminary improvements in their classroom environments. Most D-RASATI 2 activities that involved TOT are likely to be sustained through institutionalization and adoption by MEHE and CERD. CERD has the capacity to continue conducting ICT, ELTM, and PSS trainings without outside support. MEHE and CERD remain underfunded and rely on donors to replicate and create new activities (e.g. provision and maintenance of ICT equipment). Engaging with CERD was essential to this sustainability. RECOMMENDATIONS • USAID should continue designing future education projects in consultation with MEHE and CERD, as was done generally in D-RASATI 2, so that the design of the projects meets their priorities and garners support from the beginning. All project elements, including modifications, should support MEHE’s vision to ensure ownership. • While capacity building activities are inherently more sustainable than equipment provision, equipment procurement can still serve an important function in future projects, allowing beneficiaries to develop their ICT skills and ensuring MEHE buy-in. Equipment procurement should not be excluded from future projects if it serves a clear function, but if included, should come with a plan for post-project maintenance. • USAID should include findings from the ongoing gender assessment of the education sector in the design of future education projects and require implementing partners to address gender considerations more explicitly. • Activities like PSS that involve alternate teaching methods and behavior changes to promote enabling environments are relevant for all ages and nationalities, and could be expanded to non-primary and non-host schools if the material is adapted. • The TOT model proved effective and successful in many project activities and USAID should consistently apply it in future education projects to promote CERD and MEHE capacity development and ownership. • Future programs should ensure that post-project coaching is included at the outset and that these coaches, whether teachers or Departement d’Orientation Pedagogique Scolaire (DOPS) counselors, have sufficient training and support during the project period. • English language training remains necessary for teachers; if MEHE considers this a continuing priority, USAID should consider supporting continuation of this project element. • Future projects that include development of strategy and modules should ensure that there is sufficient time for training and support for institutionalization during the project period. 6 LESSONS LEARNED • USAID did not consult MEHE when adding PSS activities; this contributed to MEHE’s reluctance to accept PSS as part of D-RASATI 2 and was one of the factors that delayed implementation.. For future changes, USAID should take MEHE onboard before making modifications to ensure MEHE’s cooperation and ownership. • Future project designs that involve large procurement such ICT equipment for education should include a pilot phase where both the model and the equipment are tested before full procurement and delivery. 7 1. PROJECT BACKGROUND 1.1. PROJECT DISCRIPTION The public education system in Lebanon is under strain and lacks resources to provide quality mass education. School challenges include inadequate infrastructure, shortage of basic school facilities (e.g., electricity), limited or non-existent educational and technological equipment, and a shortage of qualified English language and Information Communication Technology (ICT) teachers. Students who do graduate from public schools have fewer opportunities to enter universities or obtain productive jobs than their private school counterparts. The low quality of public education has resulted in a polarized education market where divisions between the rich and poor are stark. Access to innovative educational institutions that provide Lebanon’s youth with the 21st century skills needed to succeed in today’s social and economic environment has become mainly restricted to those who can afford the high tuition fees of private schools. The Syrian refugee influx in the last few years has overwhelmed the public school system and further aggravated the provision of quality education. Of the 500,000 school-age refugee children in the country, approximately 158,000 are enrolled in public schools, constituting over 16 percent of the total enrollment in public schools in Lebanon1 . The United States Agency for International Development (USAID) designed the Developing Rehabilitation Assistance to Schools and Teachers Improvement (DRASATI) project to address the underlying reasons for the public school system’s inability to provide high quality education on par with the private school system. D-RASATI 2 is a continuation of the DRASATI I project, which was originally designed as a five-year (2010-2015), $75 million project. When DRASATI I was cancelled in 2013, USAID released a solicitation for D-RASATI 2 to complete DRASATI I’s remaining activities in the remaining timeframe with the remaining budget. The Ministry of Education and Higher Education (MEHE) and USAID both consider D-RASATI 2 as a continuation of DRASATI I’s work and not a separate project, although this evaluation will only consider the accomplishments of D-RASATI 2. The guiding agreement between MEHE and USAID for DRASATI, the Master Plan, was originally developed during DRASATI I. USAID awarded D-RASATI 2 to World Learning (WL) in October 2013. The $24.5 million contract was originally slated to run from October 2013 to September 2015, though modifications in 2014 and 2016 extended the project’s life until December 31, 2016 increasing the budget to $29 million and expanding the scope of work to include additional activities— mainly trainings in English language instruction methodology and in Psychosocial Support (PSS)—to target schools most affected by the Syrian refugee crisis. D-RASATI 2’s overarching goal is to support MEHE to improve the performance of the Lebanese public school system by enhancing the learning environments in public schools and providing dynamic learning opportunities to school administrators and teachers, while increasing parent and community involvement. It aims to address the underlying reasons 1 Hunan Rights Watch (2016). Growing up Without an Education: Barriers to Education for Syrian Refugee Children in Lebanon. Retrieved from Human Rights Watch website: https://www.hrw.org/sites/default/files/report_pdf/lebanon0716web_1.pdf 8 hindering the delivery of high-quality public education and to contribute to the implementation of MEHE’s Education Sector Development Plan (ESDP). D-RASATI 2 has the following three primary objectives, with specific tasks enumerated below: 1. Improve the learning environment in public schools through the procurement of Information Communication Technology (ICT) equipment and ICT training. 2. Increase learning opportunities through English teacher training and extracurricular activities. 3. Increase stakeholder engagement in the public school through leadership development with parents and community involvement; and the development and implementation of School Improvement Plans for a pool of schools with up to $4,000 USD grants per school. The following two additional project objectives were added in September 2014 (project modification No 2). D-RASATI 2 expanded existing activities, particularly English language, and added PSS trainings for host schools to achieve these objectives: 4. Increase student access to higher quality instruction and learning environments by providing teacher trainings on tropics and methods for managing new pressures on classrooms in host schools. 5. Improve social cohesion among Lebanese and Syrian students and reduce tensions at the school level. The project implemented the following three components/activities to achieve its objectives: Component 1: Provision of ICT equipment and computer labs to public schools Task 1: Develop an ICT Action Plan based on MEHE’s National Educational Technology Strategic Plan (NETSP) produced under DRASATI 1. Task 2: Procure ICT equipment for public schools. Component 2: Provision of In-Service Public School Teacher Training Task 3: Train public school teachers in the English language. Task 4: Train public school teachers, a portion of whom come from schools located in communities most affected by the Syrian crisis, in the use and maintenance of ICT equipment. Task 5: Expand extracurricular activities (ECA) at intermediate and secondary schools, and provide Psychosocial Support (PSS) training to host schools overwhelmed by non-Lebanese students. Component 3: - School Leadership Development with Parents and Community Involvement Task 6a: Implement the school leadership development program (LDP) with parents and community involvement activity. Task 6b: School improvement program (SIP) and community engagement. 9 For more detail, please see Annex 6 for list of objectives, outcomes, and proposed indicators. WL is the lead implementing agency and overall task manager for D-RASATI 2. In addition to coordination and collaboration with D-RASATI 2 implementing partners, it manages coordination with and among MEHE, Center for Educational Research and Development (CERD), Lebanese University (LU), and key public education sector stakeholders. WL also leads the ICT procurement, equipping schools with ICT, facilitating certification of teachers in Professional Certificate in English Language Teachers (PCELT), providing English Language Teaching Methodology (ELTM) trainings and the PSS training program, and providing stipends to schools. In addition, WL works with the following partners and service providers: • AMIDEAST: Leads ECA, LDP and SIP • American University of Beirut (AUB): Leads the development of the ICT Action Plan and ICT Modules and provision of ICT training to CERD. • American Lebanese Language Center (ALLC): Provides English language training • Digital Opportunity Trust (DOT) Provides in-school ICT coaching to teachers. • REED International: Security provider 1.2. DEVELOPMENT HYPOTHESIS No specific development hypothesis was established for D-RASATI 2. Nonetheless, the M&E plan states that the project was designed based on the theory that operational and technical expertise acquired by government officials and school principals and teachers through the project will help them make lasting change to their practice. It also notes that better ICT resources, English language proficiency of teachers, school leadership, community engagement, and extra-curricular activities will enhance learning environments and instruction in Lebanon’s public schools. These interventions combined will likely lead toward better learning outcomes for students. This is assumed to be the development hypothesis for the project. The key underlying assumption for D-RASATI 2 implementation was that strong MEHE support would be crucial to the successful and timely implementation of the project activities and achievement of targets. 10 2. EVALUATION PURPOSE AND QUESTIONS In the final performance evaluation, the evaluation team focused on assessing the relevance of the project activities to the needs of MEHE, the project’s effectiveness in achieving its planned results, and the sustainability of project activities. The report produced by the evaluation team includes recommendations and lessons learned that will assist USAID/Lebanon in making decisions related to (a) the type of approach the Mission should adopt in any future assistance to the public education sector, and (b) the type and scope of possible basic education projects in the future. The evaluation team also anticipates that USAID/Lebanon will use the results from the evaluation during its annual Portfolio Review. Secondary users of the report include the D￾RASATI 2 implementing partners, MEHE and CERD. D-RASATI 2 implementing partners may be able to use findings to adapt future work, and MEHE and CERD may be able to use findings in the management of teacher trainings and coaching following the project. 2.1. EVALUATION QUESTIONS 1. Relevance How relevant is the D-RASATI 2 project to the needs of the public education sector in Lebanon? Explanation: This question addresses the relevance of the project to the MEHE’s long and short term needs as highlighted by its Education Sector Development Plan. 2. Effectiveness How and to what extent has the project achieved planned results? Explanation: The question addresses the effectiveness of D-RASATI 2 overall and its various components specifically. The answer to the question should explore what has worked and how, what has not worked as anticipated and why, and highlight any unintended outcomes. The question should include D-RASATI 2’s effectiveness in improving the learning environment in the public school, increasing learning opportunities through teacher training and extra-curricular activities, and increasing stakeholder engagement in the public school improvement. We will take gender needs in to account when examining results. Answers should also contribute to practical recommendations for making future programs more effective. 3. Sustainability What is the likelihood that the results D-RASATI 2 has achieved are sustainable beyond the life of the project? Explanation: This question should generate conclusions about which D-RASATI 2 results are likely to be sustainable beyond USAID support, with focus on MEHE and CERD’s role in sustainability. It should also explore the factors that contribute to sustainability or unsustainability. It should generate recommendations for any additional actions D-RASATI 2 or USAID can take that would enhance prospects for sustainability in the future, e.g., follow-on activities to replicate or scale up the project elements. 11 3. METHODS AND LIMITATIONS The evaluation team used a primarily qualitative approach to answer the evaluation questions, conducting interviews and focus group discussions with D-RASATI 2 project staff, USAID/Lebanon staff, MEHE and CERD officials, and direct project beneficiaries. No primary quantitative data was collected, though the evaluation occasionally used project monitoring data for triangulation where relevant to findings. Data collection began on August 29 and finished on September 23, 2016. The evaluation employed the following methods: 3.1. DESK REVIEW The evaluation team conducted a systematic review of relevant internal and external documents, including the original contract agreement, contract modifications, initial assessments, annual plans, progress reports (annual, quarterly, periodic assessments), and all other project documents relevant to answering the evaluation questions. In addition, the evaluation team reviewed relevant publicly available MEHE documents that provided insights into MEHE’s strategy and education sector priorities. 3.2. QUALITATIVE DATA COLLECTION Key Informant Interviews (KII): The evaluation team conducted 40 KIIs with USAID key staff members, World Learning, AMIDEST, AUB, ALLC, LU, DOT, MEHE and CERD. The KIIs focused on project activities and their relevance to MEHE needs, project achievements, factors that contributed towards achievement of the project results, sustainability of the project results, and implementation challenges. Key informants were individuals purposively selected because of their intimate knowledge of and perspective on different project components. Key informants consisted of a diverse group of stakeholders including decision makers within partner organizations, MEHE and CERD, activity managers, points of contact, and monitoring and evaluation (M & E) officials. The interview guides and the list of interviewees can be found in Annexes 4 and 5 respectively. Focus Group Discussions (FGDs): The evaluation team facilitated a total of 45 FGDs. Through SI’s data collection partner, Information International, the evaluation included 39 FGDs with project beneficiaries. Thirty-one FGDs included school level beneficiaries—teachers and principals. The FGD questions for school level beneficiaries focused on efficacy of trainings, other project activities, and changes teachers and principals observed in their practices, classrooms and schools following participation in D-RASATI 2 activities. FGD sampling was conducted purposively to ensure that the different regions (governorates and cazas) were proportionately represented and to cluster FGDs together geographically for ease of access for participants. The evaluation team also took into account representation of both primary and secondary schools and the inclusion of both men and women in the FGDs where possible. Each FGD focused on a specific project task, with 3-5 FGDs per task. The number of FGDs per task was determined based on how wide the coverage of the task was and the relative sensitivity or importance of the findings to the evaluation purpose. 12 The evaluation included four FGDs each with CERD and Departement d’Orientation Pedagogique Scolaire (DOPS) level beneficiaries who participated in D-RASATI 2 trainings of trainers (TOTs). These FGD questions focused on the relevance and effectiveness of the training, the effectiveness and sustainability of the rollout trainings CERD conducted, and the effectiveness and sustainability of DOPS’s school coaching. Each FGD was conducted with people from the same center, again to facilitate attendance. The CERD and DOPS centers were chosen to represent different regions in Lebanon. The number of CERD or DOPS staff per center that had received training was small enough that all training participants were invited to the FGD. Members of the evaluation team also conducted six focus groups with sets of project trainers in order to triangulate their feedback with those of trainees and get a more complete picture of the effectiveness and challenges of trainings. Where the population of trainers was relatively small (fewer than ten), all were invited. Where it was larger, respondents were selected based on their interest and availability, with the assistance of the implementing partners. A two-member team—a moderator and a note-taker—was responsible for each FGD. While the moderator’s job was to make sure all questions included in the FGD guides were adequately addressed, the note-taker’s assigned role was to take detailed notes for each question. The FGD sessions were also audio recorded. Table 1 below illustrates the number of FGDs, schools and respondents for each stakeholder group. Table 1: FGD Sample Activities Beneficiaries # FGDs Component I: Provision of equipment and computer labs to public schools Task1 Develop an ICT action plan based on MEHE's National Education Technology Strategic Plan This was assessed through KIIs only Task2 Procure ICT equipment for public schools Principals 5 Component 2: Provision of In-service Public School Teacher Training Task 3 Train public school teachers in the English language Teachers 5 Task 4 Train public school teachers in the use and maintenance of ICT equipment Teachers 5 Task 5 Expand extra-curricular activities to intermediate and secondary schools Teachers 3 Sub Task 5 Provide support to targeted Host Schools to alleviate pressure on public schools hosting high numbers of Syrian refugee students Teachers 5 Component 3: School Leadership Development with Parents and Community Involvement Task 6a Implement the school leadership development with parents and community involvement activity Principals 3 Task 6b School improvement program and community engagement School representatives 5 13 Total of FGDs with school beneficiaries 31 Government Trainees CERD trainers (ICT, English, PSS, ECA) 4 DOPS (ICT coaching and PSS) 4 Project Trainers ALLC English Language Trainers 1 DOT Coaches 1 AMIDEAST ECA Trainers 1 AMIDEAST LDP Trainers 1 AMIDEAST SIP Trainers 1 World Learning PSS Trainers 1 FGD Grand Total 45 Host and non-host schools: A host school is defined as a school where more than 20% of the population is non-Lebanese, this did not end up being a relevant category for stratification because the majority of the primary schools are currently hosting non-Lebanese students, while almost no secondary schools are. Consequently, purposive selection of both primary and secondary schools ensured that there is representation of both host and non-host schools. The list of targeted schools is found in Annex 2. Data Analysis: Once the evaluation team completed data collection and received all the focus group notes, it reconvened in the Social Impact Beirut Office for data analysis and development of findings, conclusions, and recommendations. The study team used a structured and systematic approach to analyzing the qualitative data and, where possible, triangulated multiple methods and sources to ensure the reliability and validity of results. The study team analyzed the qualitative data as follows: • Summarized notes from individual interviews and FGDs; • Coded KII and FGD data according to themes relevant to the evaluation questions; • Prepared an evaluation data analysis matrix identifying the themes that emerge in the KIIs and FGDs to facilitate systematic and rigorous data analysis aimed at identifying key study findings as they related to the evaluation questions; and • Prepared a detailed outline summarizing key findings based on all the data analysis, conclusions for each study question and overall recommendations. 3.3. LIMITATIONS D-RASATI 2 closes in December 2016. The evaluation fieldwork was consequently scheduled for September 2016 so that D-RASATI 2 partners and USAID will have sufficient time to learn from the evaluation’s findings before the project closes. However, the timing of evaluation had implications for data collection. The public schools were not in session during the data collection period due to summer break, meaning that the evaluation team could not conduct site visits or collect student level data. This also means that the evaluation is highly reliant on teachers’ perceptions and self-reports of the effect of services and equipment the project provided on teacher practices in classroom and student learning outcomes. It is possible that 14 teachers may be overestimating effects on student learning, either because they are saying what they think the evaluation team wants to hear (social desirability bias) or because their view of the effects of their changed teaching methods on student outcomes is more optimistic than that of the students. Some respondents, though, were reluctant to make claims about changes in outcomes given how little time had elapsed since their training, reducing this possibility somewhat. Although the evaluation team considers teachers’ perceptions of improved learning outcomes to be an important data source, since student level data was not collected for this evaluation, the evaluation cannot confirm teachers’ claims that student learning or social cohesion actually improved as a result of D-RASATI 2 project. The short timeline between the end of trainings and this evaluation is another limitation on the evaluation team’s ability to draw conclusions regarding changes in students’ learning outcomes, as for several activities the full effects of changed teaching methodologies or additional resources would not have had time to be implemented in the classroom and be borne out in student outcomes. Teachers and principals were in schools during the data collection period, although the Eid holiday in mid-September and the busy registration period both made it difficult to mobilize teachers. The evaluation team compensated by scheduling all focus groups for teachers and principals during a three-day period (September 15, 16 and 19) right after Eid and before the beginning of school. In addition, the evaluation team invited approximately double the number of focus group participants as would usually be appropriate. Most focus groups consequently did have the expected or higher level of participation (average 7.5/FGD), with just a few with low attendance. However, this high response rate still leaves room for the possibility of selection bias. Those who made the time to come may have been those most enthusiastic about or critical of D-RASATI 2. In addition, because the trainers who participated were chosen by the implementing partners, they may have been those especially enthusiastic about the project. The evaluation team was unable to visit sites to observe project activities because the schools were not in session, and most D-RASATI 2 training activities have already ended. The evaluation discusses gender and social dimensions to the extent possible given sample and data limitations. While the report includes gender disaggregated data by project achievements when possible, it does not disaggregate findings by gender as FGDs do not produce such data. Therefore, discussion of gender and social dimensions is qualitative. The short, three-year life of the project and delays to some activities resulted in a brief amount of time between the completion of some trainings and the evaluation. This means that some activities which might have a stronger impact as teachers and principals apply them over the course of the next academic year have not had the opportunity to be truly tested, so additional challenges or successes may arise that the evaluation was unable to capture. 15 4. FINDINGS AND CONCLUSIONS 4.1. FINDINGS This section reports findings from three distinct sources: (I) key informant interviews conducted with representatives of D-RASATI 2 implementing partners, and MEHE and CERD officials, (2) focus groups discussions with primary beneficiaries and implementing partners’ trainers, and (3) project documents. Counts from the key informant interviews and focus group discussions are cited next to their corresponding findings below, and include the number of KIIs and/or FGDs out of the total number in the following manner: (KII 12/13; FGD 4/5). 4.1.1. RELEVANCE Question 1: How relevant is the D-RASATI 2 project to the needs of the public education sector in Lebanon? Component 1 - Provision of ICT equipment and computer labs to public schools Under Component 1, D-RASATI 2 developed an ICT Action Plan and ICT teacher training modules, built CERD trainers’ capacity, and procured and equipped 126 schools and six CERD training centers with ICT. The project was in the process of equipping 24 additional schools to this original target at the time of this evaluation. The results generated by these activities were intended to achieve the first objective of the project—improve the learning environment in the public schools through the procurement of ICT equipment2 . Action Plan and ICT Training Modules AUB led the process to develop the ICT Action Plan, working closely with MEHE in years one and two of the project. This Action Plan operationalizes the National Educational Technology Strategic Plan (NETSP) developed with MEHE under DRASATI I, laying out concrete steps that MEHE will take to achieve the NETSP. D-RASATI submitted the ICT Action Plan to MEHE in March 2015 and MEHE endorsed it in December 2015. Evidence from document review and KIIs indicates that MEHE accepted and had high regard for the ICT Action Plan, that the plan corresponds well to the NETSP, and that MEHE sees the ICT Action Plan as its official roadmap for integrating ICT in schools (KII 7/14). During year one, AUB also developed 15 ICT training modules for CERD. The modules focused on integrating ICT in the teaching of English, math and sciences for secondary schools. Through a Training of Trainers (TOT) approach, AUB trained CERD trainers. After the modules were translated into French in year two, CERD trainers began to roll out the ICT training 2 DRASATI II Contract Modification # 2. September 2014 “ICT training was based on teacher’s needs and the curriculum, so it was easy to integrate.” (Government stakeholder KII) 16 to public school teachers. During the roll out, AUB provided coaching to CERD trainers including on the use of Moodle, an open-source online learning platform, which was intended to help CERD integrate ICT in teaching. According to the KIIs, the ICT training modules match the official curriculum well and provide teachers with specific lessons that they can immediately adopt in their classrooms. These modules therefore match MEHE’s needs (KII 14/14). They are also a part of the ICT Action Plan, contributing its Component 2: Technology in Curriculum and Instruction. Procurement of ICT Equipment D-RASATI 2 provided ICT equipment to 126 public secondary schools and six CERD training centers. World Learning led the equipment procurement and distribution process in June 2014 and completed in March/April 2015. During the extension period (year three), D-RASATI 2 had some project savings that it chose to use, in part because of high MEHE demand for this particular intervention, to target 24 additional schools to receive ICT equipment. Stakeholders interviewed for this evaluation considered provision of equipment to be important to MEHE’s goals; while the ICT Action Plan calls for the installation of ICT equipment in schools as a prerequisite for using ICT to enhance learning, MEHE does not have funding to do this without donor support (KII 14/14). Component 2 - Provision of In-Service Public-School Teacher Training D-RASATI 2 activities under Component 2 included building CERD capacity and provision of English language training, ICT coaching, ECA, and ECA and PSS training to public school teachers. Results from Component 2 interventions were intended to achieve project objectives two, four, and five. English Language Training WL and ALLC both implemented parts of the English language training. ALLC delivered English language training to over 2500 public school teachers, and WL provided training to CERD trainers leading toward a Professional Certificate in English Language Teaching (PCELT). WL also offered a TOT to the same trainers in English Language Teaching Methodology (ELTM) and facilitated certification of teachers in Cambridge English: First (FCE) and Cambridge English: Advanced (CAE). Even though English language training is not specifically named in the MEHE Education Sector Development Plan (ESDP), teachers’ professional development is a component of the ESDP. English, math, and science teachers are supposed to teach in English. As stakeholders noted and language placement testing by the project demonstrated, the English language level of many teachers is below the level it should be for them to effectively teach in English. Therefore, improving English language is a key area for teachers’ professional development. In addition, English language training, like ICT training and ICT equipment provision to public schools, is included in the DRASATI Master Plan which was developed in cooperation with and endorsed by MEHE (KII 7/14) and provides agreed-upon guidelines between MEHE and USAID for the project implementation activities. ICT Coaching 17 DOT provided ICT coaching to teachers in the 126 schools that were selected to receive ICT equipment under Component 1. ICT facilitators were assigned to schools to coach teachers in ICT use and maintenance. This coaching is a part of the ICT Action Plan Component 4, Technology and Professional Development. Also according to the Action Plan, DOPS is the responsible agency for providing on-going coaching to teachers and eventually received ICT TOT during the no-cost extension period (KII 5/8). Extracurricular Activities (ECA) Training D-RASATI 2 partner AMIDEAST led the ECA task, which focused on developing an ECA strategy and training modules for MEHE and building schools’ capacity to implement ECAs. ECA is listed in the Master Plan and included as one of MEHE’s strategic priorities in ESDP. Evidence from stakeholder interviews indicates that while MEHE/CERD officials saw ECA as important, they did not necessarily prioritize it (KII 5/10) for D-RASATI II. In addition, the ESDP lists ECA as a strategic priority, MEHE and CERD did not have a pre-developed definition or vision for ECA. Psychosocial Support (PSS) Training World Learning led the development of a comprehensive PSS training program, which included building CERD trainers’ capacity through TOTs and supporting CERD in rolling out the training to public schools located in communities most affected by the Syrian refugee crisis. WL worked to provide these services holistically: e.g., coaching teachers during rollout and identifying lessons learned as the work unfolded. PSS was not part of D-RASATI 2’s original scope and is not listed in the Master Plan. According to KIIs, USAID identified PSS as an emerging need for assistance to host communities in the wake of the Syrian refugee influx, which overwhelmed public schools. USAID perceived PSS as addressing MEHE’s emerging needs and included it in D-RASATI 2’s scope through a project modification in September 2014 (KII 4/4). Since MEHE was not consulted on the modification to add PSS to D-RASATI 2’s scope, it was reluctant to consider PSS as part of the project’s scope initially (KII 6/6). Component 3 - School Leadership Development with Parents and Community Involvement Component 3 activities included providing leadership development trainings to in-service public school principals, school improvement trainings to school representatives, and in-kind grants to schools for implementing school improvement plans. The results from these activities intended to contribute toward achieving the third objective of the project Leadership Development Program (LDP) AMIDEAST worked closely with Lebanese University (LU) to provide required training to school principals during years one and two. The leadership development training mechanism was already in place prior to the D-RASATI 2 implementation, although MEHE has faced challenges funding the implementation of the training since 2005. As civil servants, principals must complete the LDP as an official requirement, which comes with a commensurate pay raise. Project and official MEHE documents and stakeholder interviews show that the LDP is a permanent part of MEHE’s professional development requirements for principals. MEHE consider D-RASATI 2’s LDP support to have addressed its long standing need to fund this part of principals’ professional development (KII 6/6). 18 School Improvement Program (SIP) The SIP task focused on training school representatives—principals, administrators and teachers—in school improvement planning and implementation activities. The objective was to provide opportunities to school stakeholders to learn skills required to develop and implement school improvement planning and participate in school improvement activities. While SIP is not explicitly highlighted in the ESDP, MEHE considers it an extension of the LDP, which it is. The Master Plan lists it as a key mechanism for engaging school stakeholders in school improvement activities. During DRASATI I, SIP activities were aligned with MEHE’s Effective School Profile and with school and principals’ needs in coordination with MEHE. D-RASATI 2 implemented these activities with minor changes (KII 2/2). Overall Finding Evidence from KIIs (12/13) and existing MEHE and D-RASATI 2 and II documents indicate that the project was designed and adapted in cooperation with MEHE to address MEHE’s education sector priorities. Question 1 (Relevance): Conclusions Component 1 • There was a clear link between the NETSP, the ICT Action Plan, the Master Plan, and D￾RASATI 2’s activities. MEHE was very clear on the contribution of Component 1 to its strategic goals, and consequently, this component was highly relevant to MEHE’s needs. Component 2 • The ICT coaching was relevant not only as a specific part of the ICT Action Plan, but also to support D-RASATI 2’s Component 1, which ensured that equipment and modules were used effectively in schools. Similarly, English language training was designed to promote teachers’ professional development in a way that improves the quality of teaching in public schools, supporting MEHE’s priorities. • MEHE and CERD saw the relevance of ECA to the needs of public school in Lebanon, but MEHE and CERD did not consider it a top priority. Similarly, there was a disconnect between USAID and MEHE on the relevance of PSS to D-RASATI 2’s mandate. Component 3. • D-RASATI 2’s support for the LDP activity was fully relevant to MEHE’s needs and structures. SIP extends the LDP skills and promotes the same leadership development relevant to MEHE’s ESDP. Overall MEHE and CERD had buy-in for D-RASATI 2’s plan and generally found it relevant to its strategic education priorities. This was especially true for the ICT Action Plan, English, LDP, and SIP. MEHE and CERD also saw the relevance of ECA and PSS activities but were less fully invested at the beginning. 4.1.2. EFFEECTIVENESS Question 2: How and to what extent has the project achieved planned results? Component 1 - Provision of ICT equipment and computer labs to public schools 19 Action Plan and ICT Training Modules There was widespread appreciation for the Action Plan as well as the training on the AUB modules linking ICT to the curriculum. The 15 training modules AUB developed matched the public school curriculum. The project adapted when necessary, including translating modules into French to meet needs expressed by CERD (KII 12/12). D-RASATI 2 successfully trained 30 female and 22 male CERD trainers through ICT TOTs using the 15 modules. The modules primarily focused on integrating ICT use in teaching English, math, physics, biology, and chemistry. CERD began rolling out the ICT training to public school teachers and received positive feedback according to CERD staff (KII 7/12). The CERD trainers also received the Moodle that AUB developed as an online training resource for CERD training centers. The Moodle was handed over to MEHE and CERD in January 2016, but they doubt that they have the internal capacity to use it for online training and teacher communication, as they were not sufficiently trained to use the Moodle effectively. CERD also believes the Moodle needs to be adapted to best serve as an effective online training platform for its trainers. CERD centers are currently using the Moodle for internal communication (KII 4/7; 1 /4 FGD). Procurement of ICT Equipment D-RASATI 2 is on track to overachieve the ICT procurement target, adding an additional 24 schools to the initial target of 126 schools to receive ICT equipment. Though the project conducted a readiness assessment of schools prior to finalizing the target list, the list ultimately had a number of challenges including: (a) principals who retired after the readiness assessment and were replaced by principals with less enthusiasm for integrating ICT into the school; (b) edits to the list to ensure that the targeted schools were representative of all major demographics in Lebanon; and (c) the time constraints of the assessment—i.e., the project conducted the assessment in a single day using over 200 volunteers to compensate for the lost time and therefore had to collect a limited set of data (KII 3/7; 10/14 FGD). The school readiness assessment included an assessment of cultural readiness in schools, access to the internet and electricity, space for ICT equipment, and presence of an IT person to troubleshoot technical issues. Of those schools that were finally on the target list of 126, some principals were not supportive of ICT, but accepted the equipment for their schools anyway. Similarly, not all teachers within those schools showed interest in trying ICT in classrooms (3/5 FGD). The procurement of ICT equipment was significantly delayed due to challenges with approval from MEHE on the final 126 schools, the contractor delivery, and the final procurement approval from USAID. These delays added up to almost a year of lost potential coaching on the ICT equipment. Equipment was initially due to be delivered at the beginning of the 2014-2015 school year, but was actually delivered in April of 2015, shortly before final exams began. Evidence from the focus groups and stakeholder interviews (KII 4/7; FGDs 4/5) suggest the time available for coaching on the ICT equipment after delivery was too short to allow teachers to properly learn to use it and integrate it into lessons before exams and the end of the DOT contract. Despite the delay and short project life (initially two years), stakeholders across the “ICT equipment is effective but we wish to have continuous follow up, support and maintenance on ICT equipment” (School Principals FGD) 20 board offered high appreciation for D-RASATI 2 leadership for effectively managing project activities and resolving issues (KII 7/7). The project design did not include a pilot of the ICT equipment before full procurement. Problems with equipment such as access points and fragility of the tablets were discovered after the full complement of equipment was procured and distributed (KII 4/6; FGD 4/7). In addition, although the selection of tablets was based on MEHE’s ICT Strategy, D-RASATI 2 did not pilot the tablet model of ICT integration in schools to ensure that tablets were the most suitable type of ICT for classrooms (FGD 3/7). Teachers reported that tablets are not effective for assignments that involve writing and are most suitable for multiple choice questions, which is not how official exams are conducted (FGD 2/5). The project used a tiered approach for providing ICT equipment as recommended by the NETSP. The equipment provided included tablets, laptops for teachers, LCD projectors and mobile carts for devices. Tier Four included only four schools and received more equipment— e.g. a tablet for every student—than other tiers followed by Tier Three that consisted of 10 schools. Tier Two included 112 schools that received fewer tablets (30) per school than Tiers Three and Four school. The limited number of tablets in Tier Two schools made it difficult for teachers to share them among all classes in the school (FGD 6/10). Regarding the use of ICT equipment, data from the FGDs indicate that generally teachers are using it in classrooms and they have high appreciation for LCD screens that they can use to project pre-prepared lessons (FGD 9/10). Math, science and geography teachers found the equipment to be particularly useful in visualizing complex concepts (FGD 2/10). Teachers also report that ICT use in classroom helped improve student learning (FGD 8/10). However, some schools reported that they were not using equipment. Individual reasons varied but included lack of electricity or internet and challenges to use and integrate ICT into lessons because of the time required for preparing ICT-based lessons due to the routine workload (FGD 4/10). Component 1I - Provision of In-Service Public-School Teacher Training English Language Training Over the course of the project, ALLC provided English language training to 2,509 public schools teachers—73% female—slightly overachieving the project’s original target of 2,500 teachers. Of the total teachers receiving English language training, approximately 39% came from schools in host communities. During years one and two, WL provided training and certification support to 33 CERD and MEHE trainers in PCELT and trained them in ELTM through TOTs. Table 2 provides details of this task by beneficiary type and gender. Table 2: English Language Training Beneficiaries Training/Certification Beneficiary Type Female Male Total English Language Training Teachers 1837 672 2509 PCELT CERD Trainers 30 03 33 ELTM TOT CERD Trainers 30 03 33 ELTM Training Teachers 175 08 183 FCE Teachers 180 40 220 21 CAE Teachers 37 03 40 D-RASATI 2 used BULATS, a standardized, internationally validated English test to determine teachers’ English proficiency level. ALLC also used a customized first-day assessment to ensure teachers were placed in the right level. While CERD is skeptical that the BULATS test was the correct assessment tool for the education sector because the test is often used to assess English in business settings (KII 3/3), ALLC selected the test because it is standardized and internationally validated. ALLC also consulted the Cambridge English Language Assessment on the selection of BULATS test to ensure it was selecting the right test. Few trainees complained about their placement and ALLC considered the BULATS to have placed trainees well (KII 3/4). The project exceeded its targets for English training and received positive feedback from trainees, including on improvements in language skills and application of methods in their own classrooms (KII 5/5; FGD 3/5). In addition, teachers report that the training increased their ability to explain subjects to students and communicate with them effectively (2/5 FGD). Trainees said that they appreciated trainers’ innovative teaching methodologies and tried to adopt them to the extent possible in their own classrooms. However, not all activities could be transferred to classes because teachers lacked certain resources, including ICT equipment for showing videos and space for active lessons that required moving around (3/6 FGD). Some math and science teachers found the English training to be of limited use because the technical nature of their subjects meant that specific terminologies were not covered in the training. The learning activities used in the trainings were not specifically relevant to the lessons they taught, so some teachers did not think that they could benefit by adopting them to their classes (3/3 FGD). There were exceptions, however; some teachers reported that they are able to better explain subject concepts now that they have improved language skills (FGD 1/3). Teachers who received 150 hours of English language training were offered the FCE certification test. D-RASATI 2 certified 220 teachers in FCE—93% of the target. Forty trainees took the CAE test (a level above FCE), but results were pending at the time of this writing. While D-RASATI 2 overachieved its target, there were a number of challenges related to mobilizing teachers and preparing suitable training facilities. Initially teachers complained about the timing of the training. Some training venues were also problematic for teachers due to proximity and mobility issues. The project responded quickly in coordination with MEHE by addressing issues like teacher availability and changing training times (KII 6/6; FGD 1/1). PCELT was considered to be a gold standard of English Training and was generally appreciated across the board, including the student-centered learning approaches. Data from KIIs and FGDs indicate that the ability to create better lesson plans through developing learning objectives supported by multiple activities and for teachers to be able to conduct self- “We got outstanding teachers with certificates from Cambridge. DRASATI II built CERD and DOPS capacity to provide excellent English training. ” “The English language training built teachers’ confidence” (Government stakeholder KII) “The benefit of the training was developing the ability to write smart and measurable objectives and apply them at the classroom level. ” (ELTM Teachers FGD) 22 assessments were considered highly valuable. Teachers report that the better structured lessons allowed students to learn better (KII 4/5; FGD 5/5). D-RASATI 2 was able to further build the capacity of the PCELT certified CERD trainers through ELTM TOTs. CERD began to successfully roll out ELTM to 183 English teachers (KII 4/5). ICT Coaching D-RASATI 2 piloted ICT coaching in 26 schools in year one. Before the roll out of the in-school coaching, DOT and WL trained DOT coaches/teacher facilitators who were later deployed in 126 schools to provide coaching in ICT use and troubleshooting. Coaching ended in May 2015, approximately one month after DRASATI delivered the ICT equipment to schools. In the interim, DOT generally coached schools using ICT equipment that already existed in schools. Some schools had little or no existing ICT equipment which made coaching and ICT use in classroom for teachers more challenging (FGD 3/5; KII 3/6). Due to limited ICT in some schools, the coaches used their personal laptops to train teachers. Apart from these challenges, teachers generally appreciated the ICT coaching for improving their knowledge and skill of ICT use in the classroom (FGD 4/6). Stakeholder interviews and FGD data suggest the schools were not able to benefit sufficiently from ICT coaching due to the delayed provision of the ICT equipment. The coaching period lasted for about a month after the equipment arrived in March 2015 and installation of equipment and initial troubleshooting completed in April 2015 (final exams began in May), which was too short to ensure effective teacher orientation with and use of the equipment. DOT’s contract ended after the schools closed in June. Because the approval of DOT’s contract extension was significantly delayed, they were not able to continue coaching during the following academic year. Teachers requested more training to make up for this gap (FGD 3/5; KII 3/5). To address this gap, D-RASATI 2 conducted a refresher workshop in September 2016 for champion teachers—i.e., teachers with better ICT knowledge and skills relative to their other colleagues in their schools who would act as advocates and models of ICT use within their schools (3/5 KIIs). Though there was no TOT planned for CERD and DOPS on ICT coaching in the original plan, this was added during the extension period. The project trained 36 CERD trainers and 119 DOPS teacher mentors. DOPS participants considered the training to be too short and basic (2/4 FGD). The lack of coordination between AUB and DOT was another shortcoming. Although they both supported ICT use in schools and coordinated schedules prior to the beginning of activities, there was no coordination for content or sharing of lessons learned once activities began (4/6 KII + 1/1 FGD). Stakeholder interviews indicate that DOT’s experience, particularly during the pilot phase of coaching with schools could have provided valuable insights to AUB when it was developing the ICT training modules for CERD. Extracurricular Activities (ECA) Training D-RASATI 2 did not take the same TOT approach for ECA as for the rest of the training tasks of the project, even though the DRASATI Master Plan recommends the TOT approach for all training activities. D-RASATI 2, CERD, and MEHE have varying understanding of what ECA is 23 and its purpose. MEHE and CERD were resistant to approaches that did not align with their understanding of ECA. In addition, there was a difference of opinion between D-RASATI 2, and MEHE and CERD on how it should be implemented, so initial drafts of ECA strategy and modules went through a lengthy review and revision process (KII 4/5; FGD 7/7). These coincided with turnover within CERD, further lengthening the review process resulting in almost a yearlong delay. MEHE finally approved the modules in August 2016. D-RASATI 2 trained 143 teachers—83% female—in implementing ECA activities in schools. Data from the focus groups indicates that ECA training increased teachers’ knowledge of ECA and that they learned ECA skills and techniques, including planning activities and teamwork (FDG 4/4). They were able to do this through small one-off projects at their schools, which they reported that student enjoyed (FDG 2/3). Teachers also reported lack of resources— space in schools, time from their routine workload—as challenges to implementing ECA in their schools (FDG 3/4). Since the ECA task did not include sensitizing parents about ECA, teachers reported parents were not always supportive of their children’s involvement (FGD 2/3). In addition, support from the school principals and approvals from MEHE for after school extracurricular activities could help more regular implementation in schools (FGD 3/4). Psychosocial Support (PSS) Training PSS was significantly delayed, initially for 13 months, for several reasons. MEHE initially believed it would be receiving support for PSS trainings from another donor. CERD had turnover at the director level during that time which further delayed the process of taking CERD onboard. There was some skepticism from MEHE on the validity of the negative results—e.g., tension between Lebanese and non-Lebanese students, bullying—shown by the rapid needs assessment D-RASATI 2 had conducted in year two. PSS was not in the DRASATI Master Plan, so MEHE saw it as outside the project’s scope and was slow to agree to work with D-RASATI 2 on PSS. They continued to be sensitive about the way PSS was initially added to D￾RASATI 2’s scope (4/5 KII). Both MEHE and CERD appreciated the eventual implementation of PSS, and expanded their knowledge of PSS topics. WL developed the PSS training material, “Essentials of PSS for Teachers,” and adapted it to the Lebanese context including translating it into Arabic. D￾RASATI 2 trained CERD trainers and DOPS counselors through TOTs. CERD successfully rolled out the training to teachers from approximately 191 host schools. CERD adopted the PSS training and included it in its training plan for the current academic year (KII 4/5). Table 3 below contains PSS beneficiary details by type and gender. Table 3:Psychosocial Support Training Beneficiaries Training Beneficiary Type Female Male Total PSS Trainings Teachers 495 28 523 PSS TOT CERD Trainers 36 05 41 DOPS Counselors 46 09 55 MEHE Officials 02 00 2 “PSS activities helped build a friendly relationship between students and teachers, which in turn improved students’ behavior in class” (Teachers FGD) 24 Evidence from focus groups indicates that at the school level, teachers have increased their knowledge of PSS and introduced experiential learning techniques into their work, improving their classroom management and student learning outcomes. Teachers reported that they were better able to deal with students with problems (FGD 9/10). Teachers generally found the practical exercises and PSS activity applications more effective than the theoretical parts of the training (FGD 5/7). The PSS training was the only material that included separate consideration of boys’ and girls’ needs. Overall teachers considered students’ needs equally addressed (FGD 4/5). Some of the challenges teachers faced when implementing PSS activities included principals’ limited understanding of PSS. The PSS process included minimal sensitization with principals on the topics that would be covered and the learning approach that would be taken. Consequently, some principals objected to the noise made when teachers implemented the activities they learned from the PSS training in their classes (FGD 4/8). In addition, PSS did not include parent awareness raising, which meant that parents sometimes did not understand the importance of PSS (FGD 4/5). Implementation within schools was limited to a few classes as only a small number of teachers (three to four) per school were trained and they were the only ones in the school implementing PSS-related activities (FGD 4/7). Teachers received PSS training toward the last quarter of the academic year (March 2016), resulting in teachers having limited time to implement PSS activities in addition to the material they were required to cover as part of the school curriculum before the final exams (FGD 5/6). The project developed the PSS material as a single training unit, but PSS is not a one-size-fits-all subject. In reality, teachers had different levels of previous exposure to PSS content. The classroom contexts (including student age) varied widely and so did the need for adapting PSS accordingly (FGD 4/7). Component 3 - School Leadership Development with Parents and Community Involvement Leadership Development Program (LDP) The project provided LDP training to 587 school principals — 46% female—exceeding its original target of 485. Part of the training required principals to plan and implement a school improvement project in their schools. Approximately 488 LDP schools designed and implemented their projects in line with MEHE standards. Stakeholder interviews and FGD data suggest that the LDP improved principals’ skills in a number of areas, including management, delegation, teamwork and ICT knowledge and use. The training also improved their relationships with teachers. (FGD 4/4; KII 3/4). Principals felt that the skills they gained through the LDP should be something that all principals receive at the beginning of their assignment (FGD 2/4). School Improvement Program (SIP) The SIP objective was to provide opportunities to school stakeholders to learn skills required to develop and implement school improvement planning and engage in school improvement “We learned how to plan, supervise, manage, delegate and work as a team. We are able to lead and help teachers to improve their skills.” (LDP Principals FGD) 25 activities. Between December 2013 and July 2015, D-RASATI 2 trained 711 representatives from 191 schools. In addition, the project provided $ 1,100 in stipends to 65 host schools. The stipends to selected host schools were disbursed with the intent to give small, targeted support in teaching and learning material (KII 3/5). Table 4 shows LDP and SIP details by beneficiary type and gender. Table 4: Leadership Development & School Improvement Program Beneficiaries Beneficiary Type Female Male Total LDP In-service Principals 269 318 587 SIP School Representatives 408 303 711 The evidence from KII and FGD show that SIP built on the LDP skills through helping principals and schools to be able to conduct needs assessments, work as teams, and implement small projects in their schools. Approximately 90% of schools receiving $ 4,200 worth of in-kind grants successfully completed their planned projects (FGD 6/6; KII 2/2). While schools appreciated any extra funding available, they considered a $4,200 grant to be quite limited for addressing their schools’ needs (FGD 4/6). Students received some benefits from the projects that included improved access to technology, library and lab equipment (3/6). Overall Findings For the most part, D-RASATI 2 took a TOT approach to institutionalizing trainings developed with DOPS and CERD. While most project delays were due to the long period of approvals required from both MEHE and USAID, the Master Plan helped the project avoid deviation in planning activities and from the agreed-upon guidelines between USAID and MEHE. MEHE, CERD, and D-RASATI 2 partners report that WL’s leadership and management played a strong role in the successful implementation of project activities. The vast majority of interviewees, across all components, did not see gender as something the project needed to take into consideration. The project document speaks of gender only in terms of disaggregation of data and participation numbers. Question Two (Effectiveness): Conclusions Component 1 • Because the ICT modules matched the school curriculum well, CERD and teachers were able to integrate the new material into the standard trainings and lessons effectively. • The successful procurement and installment of the ICT equipment contributed to improving the learning environment in public schools (objective one). The project ultimately over￾delivered on the ICT, but the significant delay in procurement seriously undermined the coaching’s effectiveness as coaches were not able to spend sufficient time with teachers specifically helping them to use the newly arrived equipment. In addition, not all schools had the necessary human resources and infrastructure to effectively use the equipment. These are all threats to the ultimate use of the equipment for enhancing learning in classrooms. 26 • The lack of piloting of equipment procured meant that technical issues the equipment faced were unanticipated. Solutions were deployed on a large scale rather than through a pilot and then scaled up. In addition, the lack of pilot of the D-RASATI 2 model meant that the project lost an opportunity to test the overall choice of tablets and tier design as the most suitable for the majority of classes. This is a particular weakness in light of some feedback from teachers that tablets were not a good fit for their classes and from key informants that other ICT solutions may have been less challenging (such as laptops or interactive whiteboards). • While some schools are unable to use the ICT equipment, other schools are using it effectively. The LCDs in particular were easy for teachers to integrate with their lessons. When teachers were able to use the equipment effectively, the equipment made teaching and learning easier. • There was a missed opportunity for sharing lessons between partners working on the same component—e.g. ICT. Though this did not threaten the effectiveness of the project, further collaboration could have enhanced it. Component 2 • The English language training was effective at mobilizing teachers to attend the training, improving their language skills, and improving their teaching methodologies, especially for English teachers. The certification was particularly valuable in giving the training prestige and credibility. The ELTM and PCELT were both very effective trainings for CERD (and those DOPS attendees) and its rollout was valuable to improving teacher skills. The results from the English language training activities contributed to achieving project objectives two and four. • Overall teachers ended up in the appropriate English class level, but teachers of specialized subjects could have benefitted from English classes that more specifically targeted their subject matter (particularly math and sciences). • Efficacy of coaching was mixed due to delays in ICT procurement and service providers’ contract extension. Despite these delays, the ICT topics covered during the initial coaching time were somewhat effective and contributed to the eventual use of the equipment by a subset of teachers. To compensate for lost time, D-RASATI 2 added additional training for CERD, DOPS and champion teachers in the extension period. Because this training was delivered recently, this evaluation cannot judge their effectiveness. • D-RASATI 2’s achievements toward increasing learning opportunities in public schools (objective two) included designing the ECA strategy, conducting the training with the 123 schools targeted, and developing the thematic modules. The project was unable to conduct training for the thematic modules due to delays. Without this training, it is unclear how effective the modules would be. Although the project was largely able to transfer ECA skills, obstacles on the ground limited the ultimate efficacy of ECA implementation. Because ECA is not fully a part of MEHE’s structure, teachers face challenges further expanding on initial successes. Additionally, without a TOT approach, D-RASATI 2 missed opportunities to build CERD’s capacity in ECA. 27 • Despite obstacles, PSS was eventually effectively and successfully implemented as a TOT for CERD and DOPS, which CERD then effectively rolled out. When implementing the practical exercises, teachers who participated in the trainings consequently saw preliminary improvements in their classroom environments. The material was gender sensitive and considered the differing PSS needs of boys and girls. PSS activities could have bolstered effective implementation through sensitization of principals and parents. The results from the PSS intervention contributed toward project objective five. Component 3 • D-RASATI 2’s support for LDP was effective in implementation and in improving principals’ leadership skills. Under SIP activities, school representatives effectively gained the intended skills. Overall, projects were effective in making small improvements in their schools, but the size of the grants and stipends was too small to make large differences. Instead, the benefit to schools came mainly from the experience of conducting needs assessments and working together as a team, which could be transferrable to other school initiatives. • School representatives effectively gained the intended skills from the SIP activity. Overall, projects were effective in making small improvements in their schools, but the size of the grants and stipends was too small to make large differences. The benefit to schools instead primarily came from the experience of conducting needs assessments and working together as a team, which could be transferrable to other school initiatives. Cross-cutting • The project overlooked including gender considerations into many activities. There was no assessment of differing needs of boy and girl students for activities such as ICT, ECA or SIP, where worldwide gender differences often exist. Because the evaluation team could not speak to students, we were unable to assess whether this oversight negatively affected outcomes for students. PSS was an exception. Overall, the project’s strong leadership and management, adherence to the Master Plan, and approach to working with MEHE and CERD resulted in successfully implemented activities during a short project timeframe and a strong relationship with stakeholders. The short project life—initially two years—was inconsistent with how closely the project was required to work with MEHE on a project of this scope and scale. 4.1.3. SUSTAINABILITY Question 3: What is the likelihood that the results D-RASATI 2 has achieved are sustainable beyond the life of the project? Component 1 - Provision of ICT equipment and computer labs to public schools Action Plan, ICT Training Modules and ICT Equipment Document review and KII data indicate that the ICT Action Plan as a roadmap for integrating ICT in schools, improved ICT skills for CERD, and the 15 modules supporting the curriculum are mainstreamed within the system for continued benefits to public education (KII 8/9). As for the ICT equipment, MEHE lacks funding to replace or expand equipment at a large scale once the equipment becomes obsolete or breaks (5/8 KII). 28 Component 2- Provision of In-Service Public-School Teacher Training English Language Training For teachers with intermediate and elementary level English, it will be difficult to continue practicing and improving on their own. Teachers whose language skills reached a more advanced level now have the ability to continue speaking English in their classes. (KII 3/3, FGD1/1). As a result of D-RASATI 2 capacity development support, CERD has qualified ELTM and PCELT trainers who are able to continue providing training to public schools (KII 2/3, FGD 3/5). However, CERD considers the ELTM to be an exceptionally long training (two weeks) and difficult to implement in its entirety given the usual five-day length of their trainings (2/3 KII). ICT Coaching The majority of teachers coached still have a limited ICT capacity, especially in integrating ICT tools into the classroom, which is time-consuming to do for the first time. The majority of the focus group participants reported that they need more training to be able to use ICT effectively in their work (FGD 9/10). While DOPS received training to help them coach teachers following the end of the project, the training was limited in scope and length (FGD 4/5). For continued ICT support available within schools, D-RASATI 2 has identified champion teachers in all 126 schools with relatively high ICT skills or interest in ICT to provide motivation and mentoring to other teachers in their schools. These teacher champions are receiving 5 days additional training in September 2016 (FGD 3/5). Extracurricular Activities (ECA) Training MEHE, CERD and teachers have received the ECA strategy and modules to continue implementation. However, CERD is not ready to implement these as part of their teacher training for several reasons. First, they want to pilot the ECA thematic modules to test their suitability for schools, which has not yet been done. Second, they plan curriculum revisions and want the ECA to match the revised curriculum (KII 1/1). Since D-RASATI 2 did not take a TOT approach toward ECA, CERD and DOPS have not been trained in the ECA material at the time of the evaluation (FGD 3/3). Psychosocial Support (PSS) Training CERD has already begun to take steps to adapt the PSS materials and expand them, and included them in the teacher training plan for this year. They and DOPS have been trained in PSS (4/5 KII). However, DOPS coaches are assigned based on subject matter and cross-cutting issues like PSS do not easily fit into their coaching model, so no unit at DOPS is assigned to PSS specifically (KII 1/1). Principals have the power to stop activities such as PSS from proceeding if they are not interested in or opposed to them, or to devote time and resources to activities that they are interested in (KII 3/5). Currently, MEHE has some funding from other donors (UNICEF) to continue the PSS work (KII 1/1). Component 3- School Leadership Development with Parents and Community Involvement Leadership Development Program (LDP) The LDP is part of the MEHE system and the capacity to implement these trainings and activities does exist - i.e., through LU (KII 4/4). However providing LDP trainings to school 29 principals on a regular basis will require stable funding. LU and MEHE do not have dedicated funds to continue LDP without donor assistance (2/2 KII). School Improvement Program (SIP) SIP helped develop school representatives’ skills such as needs assessments, project planning and project management. Data from the focus groups indicate that most of the school representatives plan to use these skills for future projects, but identified funding as the biggest challenge (4/5). Overall Finding The evidence collected shows that CERD plans and has the capacity, built through TOTs, to take D-RASATI 2 capacity development material forward as part of its official teacher trainings and likewise for MEHE. However, KIIs with MEHE and CERD indicated that they are strongly dependent on outside funding in order to continue activities. KII data also indicated that DOPS and CERD operate as separate bodies through different channels of reporting, and that D￾RASATI 2 was exceptional at targeting both bodies. Question 3 (Sustainability): Conclusions Component I • The ICT action plan, capacity built, and modules are likely to be sustainable and MEHE/CERD are likely to use them for their immediate needs and to base future ICT work on these initial inputs. The equipment will not be sustainable without further donor support. The continued use of the ICT equipment by teachers will be will be highly sensitive to the efficacy of champion teachers. DOPS’s ICT capacity is not yet developed enough to fully support teachers. Component 2 • The improvements in English skills are only sustainable without further training for the teachers who completed a relatively advanced level. CERD has the expertise to continue conducting ELTM trainings without outside support. However, they will need to adapt the materials to fit their usual training structure, which has not yet been done. • ECA is not yet institutionalized. CERD expects to have to test and modify the ECA materials substantially in order for these materials to meet their vision of ECA that explicitly supports the public school curriculum. They will also need to ensure that their trainers and DOPS are familiar with ECA once the new materials are developed. It is not clear that they have the resources to do all of these activities. • CERD has taken ownership over the PSS trainings and committed to continuing to offer these trainings to schools. It is not clear that DOPS has an effective plan to continue supporting PSS, but MEHE’s continued funding from other sources to support this work seems promising for the sustainability of the PSS work, at least for the short term. Component 3 • LDP is completely institutionalized. Its continuance depends on MEHE’s ability to secure funds for the training, either through its own budget allocation or through external funding. The skills that school representatives gained through SIP are likely to persist because they 30 are general management skills. In order to continue using them to improve their schools, they will likely need further support. Broadly, most D-RASATI 2 activities that involved TOT are likely to be sustained through institutionalization and adoption by MEHE and CERD. They are still underfunded and rely on donors to replicate and create new activities (provision and maintenance of ICT equipment). Engaging with CERD was essential to this sustainability. At the school level, principals are a major factor for sustainability. Activities appeared more likely to be sustainable where principals were committed to them. 31 5. RECOMMENDATIONS AND LESSONS LEARNED • USAID/Lebanon should continue, as was done for the most part in D-RASATI 2, designing any future education projects in close consultation with MEHE and CERD. This will allow USAID to design projects that meet MEHE and CERD’s expressed priorities and needs, has their support from the beginning, and is coordinated with other donor efforts. All project elements should support MEHE and CERD’s vision in order to ensure their ownership. This should include modifications to the project in response to changing country context. • While the capacity building activities are inherently more sustainable than equipment provision, equipment procurement can still serve an important function in future projects, allowing teachers and CERD/MEHE to develop their ICT skills using that equipment and helping ensure MEHE buy-in to the project. Equipment procurement should not be excluded from future projects if it serves a clear function, but should come with a plan for post-project maintenance if it is included. • Future projects can more effectively leverage their consortia by ensuring that time for learning and sharing between partners in order to enhance implementation and technical synergies is built into project design. • USAID should design project time periods that are consistent with the expected activities and results. For a project of D-RASATI 2’s size and scope, four to five years would have been a more appropriate length. • USAID should include findings from the ongoing gender assessment of the education sector in the design of the next education project as planned and require implementing partners to more explicitly address gender considerations. • Activities like PSS that involve alternate teaching methods and behavior changes to promote enabling environments should ensure that parents (as well as principals and the school at large) are sensitized to and supportive of the activities. PSS is a topic that is relevant for all ages and nationalities, and could be expanded to non-primary and non-host schools if the material is appropriately adapted. • The TOT model for both CERD and DOPS proved effective in many of D-RASATI 2’s other components, and should be consistently applied in future education projects to promote CERD and MEHE capacity development and ownership. • Engaging principals should be the first step to introducing and carrying out any activity in a school. • Future programs should ensure that post-project coaching is included as part of the plan from the beginning and that these coaches, whether teacher champions or DOPS, have sufficient training and follow up during the project period to fill this role. • There is still a need for English training for teachers in Lebanon, so if MEHE considers this a continuing priority, USAID should consider supporting continuation of this effective project element. 32 • Future projects should ensure that TOT material for CERD fits within CERD’s training structure. Future projects that include development of strategy and modules should ensure that there is sufficient time for training and support for institutionalization during the project period as well. LESSONS LEARNED • USAID did not consult MEHE when modifying the project scope of work by adding PSS activities; this contributed to MEHE’s reluctance to accept PSS as part of D-RASATI 2 and was one of the factors that delayed implementation. For future project modifications, USAID should take MEHE onboard before making modifications to ensure MEHE’s cooperation and ownership. • Future project designs that involve large procurement such as ICT equipment for education should include a pilot phase where both the model and the equipment are tested before full procurement and delivery. This may include offering more customized combinations of options depending on teacher and school readiness, skill, and class subject matter. 33 ANNEX I: INCEPTION REPORT DEVELOPING REHABILITATION ASSISTANCE TO SCHOOLS AND TEACHERS IMPROVEMENT (DRASATI II) Final Performance Evaluation Inception Report August 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Social Impact, Inc. 34 DEVELOPING REHABILITATION ASSISTANCE TO SCHOOLS AND TEACHER IMPROVEMENT (DRASATI II) Final Performance Evaluation Inception Report This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from: DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract/Order No. AID-268-C-15-00001 ACRONYMS ADS Automated Directives System ALLC American Lebanese Language Center AUB American University in Beirut CDCS Country Development Cooperation Strategy CERD Center for Education and Research COP Chief of Party COR Contract Officer Representative DEC Development Experience Clearinghouse DO Development Objective DOT Digital Opportunity Trust DRASATI II Developing Rehabilitation Assistance to Schools and Teachers Improvement II DRASATI I Developing Rehabilitation Assistance to Schools and Teachers Improvement I ECA Extracurricular Activities ESP Effective School Profile EQUI Evaluation Quality Use and Impact ET Evaluation Team FGD Focus Group Discussion GOL Government of Lebanon ICT Information Communication Technology IPs Implementing Partners IR Intermediate Result LDP Leadership Development Program LOE Level of Effort LU Lebanese University KII Key Informant Interview M&E Monitoring & Evaluation MEHE Ministry of Education and Higher Education PCELT Professional Certificate in English Language Teaching PIRS Performance Indicator Reference Sheet PMP Performance Management Plan RNA Rapid Needs Assessment RF Results Framework SIP School Improvement Program SOW Statement of Work TOT Training of Trainers USAID United States Agency for International Development TABLE OF CONTENTS EXECUTIVE SUMMARY................................................................................................ 44 INTRODUCTION........................................................................................................... 44 PROJECT BACKGROUND............................................................................................ 45 DEVELOPMENT HYPOTHESIS ................................................................................... 10 PURPOSE AND INTENDED USE ................................................................................ 47 INTENDED AUDIENCE ................................................................................................ 48 EVALUATION DESIGN AND METHODS.................................................................. 49 EVALUATION MANAGEMENT ................................................................................... 58 ANNEX 1: LIST OF TARGETED KEY INFORMANTS ....... ERROR! BOOKMARK NOT DEFINED. ANNEX 2: LIST OF TARGETED FGD PARTICIPANTS.... ERROR! BOOKMARK NOT DEFINED. ANNEX 3: DATA COLLECTION INSTRUMENTS:................................................... 63 EXECUTIVE SUMMARY The Lebanon Developing Rehabilitation Assistance to Schools and Teacher Improvement (DRASATI) II is a contract awarded by USAID to World Learning in 2013 and ending in December 2016. DRASATI II, implemented by World Learning and its partners, aims to support the Ministry of Education and Higher Education (MEHE) to improve the performance of the Lebanese public school system by enhancing the learning environments in public schools and increasing parent and community involvement. As DRASATI II reaches the end of its term in December 2016, USAID requested a final evaluation to review the performance of the project. USAID will be the primary users of the evaluation, using the findings to help develop its future education sector work in Lebanon. Social Impact, as the prime implementer of the Performance Monitoring Support Project for USAID Lebanon, is conducting the final performance evaluation. The evaluation team has worked with USAID to agree on the following evaluation questions: 1. Relevance: How relevant is the DRASATI II project to the needs of the public education sector in Lebanon? 2. Effectiveness: How and to what extent has the project achieved planned results? 3. Sustainability: What is the likelihood that the results DRASATI II has achieved are sustainable beyond the life of the project? The main evaluation data sources will be project and secondary documents identified during desk review, key informant interviews and focus group discussions. Most of the data will be qualitative, though where possible, the evaluation team will also draw on the project’s quantitative monitoring and evaluation data. Key informants will primarily be project, USAID and the Ministry of Education and Higher Education (MEHE) staff, while focus groups will consist mainly of direct beneficiaries—principals and teachers. Following fieldwork, the evaluation team will systematically assess, code and analyze the data to answer the evaluation questions, prepare a presentation for USAID sharing preliminary findings, and produce a draft and final report with findings and recommendations. SI will use its Evaluation Quality and Use and Impact (EQUI®) management approach to ensuring that key stakeholders are engaged throughout the evaluation process and that all evaluation deliverables adhere to best practices. INTRODUCTION The Lebanon Developing Rehabilitation Assistance to Schools and Teacher Improvement (DRASATI) II is a contract awarded by USAID to World Learning for $24.5 million. DRASATI II aims to support the Ministry of Education and Higher Education (MEHE) to improve the performance of the Lebanese public school system by enhancing the learning environments in public schools and increasing parent and community involvement. The project was originally slated to run from October 2013 to September 2015, though contract modifications in 2014 and 2016 extended the project’s life until December 31, 2016 increasing the budget to $29 44 million and expanding the scope of work to include additional activities to target schools most affected by the Syrian refugee crisis. As DRASATI II reaches its end of term in December 2016, USAID requested a final evaluation to review the performance of DRASATI II project. Social Impact, as the prime implementer of the Performance Monitoring Support Project for USAID Lebanon, is conducting the final performance evaluation. This document comprises the inception report for the evaluation, laying out the plan that the team will follow to answer USAID’s evaluation questions. This report will first describe the project’s background, then the evaluation’s purpose and intended use and users, the evaluation questions, and the methodology. PROJECT BACKGROUND The DRASATI program is designed to address the underlying reasons for the public school system’s inability to provide high-quality education on par with the private school system. Those students who do graduate from public schools have fewer opportunities to enter universities or obtain productive jobs. The low quality of public education has resulted in a polarized education market where divisions between the rich and poor are stark. Access to innovative educational institutions that provide Lebanon’s youth with the 21st century skills needed to succeed in today’s social and economic environment has become mainly restricted to those who can afford them, i.e. afford the high tuition fees of the private schools. DRASATI II’s overarching goal is to support the Ministry of Education and Higher Education (MEHE) to improve the performance of the Lebanese public school system by enhancing the learning environments in public schools and providing dynamic learning opportunities to school administrators and teachers, while increasing parent and community involvement. It addresses the underlying reasons hindering the delivery of high-quality public education. It contributes to the implementation of MEHE’s Education Sector Development Plan and complements the achievements of DRASATI I. DRASATI II was designed and awarded starting in FY2013 prior to the approval of the USAID/Lebanon Country Development Cooperation Strategy (CDCS) in 2015. D-RASATI II is designed to support Development Objective I (DO I), IR 1.1 and sub-IR 1.1.1 and 1.1.2. Specifically, DRASATI II has the following three objectives, with specific tasks enumerated below: 1) Improve the learning environment in public schools through the procurement of Information Communication Technology (ICT) equipment and ICT training Task 1) Develop an ICT action plan based on MEHE’s National ICT in Learning Strategy produced under DRASATI 1. Task 2) Procure ICT Equipment for public schools. 45 2) Increase learning opportunities through English teacher training and extra‐curricular activities Task 3) Train public school teachers in the English language Task 4) Train public school teachers, a portion of whom come from schools located in communities most affected by the Syrian crisis, in the use and maintenance of ICT equipment Task 5) Expand extra‐curricular activities at intermediate and secondary schools 3) Increase stakeholder engagement in the public school through leadership development with parents and community involvement; and the development and implementation of School Improvement Plans for a pool of schools with up to $4,000 USD grants per school. Task 6 a) Implement the school leadership development with parents and community involvement activity Task 6 b) School improvement program and community engagement World Learning is the lead implementing agency, and overall Task Order Manager for DRASATI II. World Learning manages the coordination among MEHE, Center for Education and Research (CERD) and Lebanese (LU), key public education sector stakeholders. In addition, it regularly communicates and collaborates with DRASATI II partners and service providers to ensure smooth implementation. World Learning monitors and provides periodic performance evaluation of all activities, and provides regular updates to the technical and coordination committees. World Learning works with two other partners to implement project activities: 1) AMIDEAST: AMIDEAST leads the extra-curricular (ECA), school improvement (SIP), and LDP/community involvement programs. 2) American University of Beirut (AUB): AUB leads the development of an ICT Action Plan and related ICT Training Modules, provides TOT on ICT for CERD trainers, and develops standards that will contribute to MEHE’s ESP. Other DRASATI II Partners • Service Providers. DRASATI II works with two service providers to deliver training to teachers. The American Lebanese Language Center (ALLC-International House of Beirut) is responsible for delivering English language training and learning assessments to teachers of English, Math and Science. Digital Opportunity Trust (DOT) is responsible for delivering intensive in-school coaching to teachers working in schools that receive ICT equipment. 46 • Universities: DRASATI II partnered with AUB to develop an ICT Action Plan and related ICT Training Modules, provide TOT on ICT for CERD trainers, and develop standards that will contribute to MEHE’s ESP. It also collaborates with Lebanese University (LU) for the design of leadership development standards which entails LU providing DRASATI II with Leadership Development Program (LDP) regulations, guidelines, curriculum, and training materials to be updated; the development of a work plan to implement LDP activities; approval of the trainers recruited according to criteria developed by LU; and training them on the content and the learning and teaching strategies required by the program. • Ministries: DRASATI II is the only USAID-funded project which works with a Lebanese Ministry. Although DRASATI II operates under a contract with USAID that clearly defines activity project deliverables and timelines and successful project implementation, the ability of the activity to achieve targets depends on a commitment from the MEHE and its teacher development entity, CERD. DEVELOPMENT HYPOTHESIS There was no specific development hypothesis established for DRASATI II statement of work. Nonetheless, the M&E plan states that the project was designed based on the theory that operational and technical expertise acquired by government officials and school employees through the project will help them make lasting change to their practice. It also notes that use￾focused standards, better ICT resources, English language proficiency of teachers, school leadership, community engagement, and extra-curricular activities will enhance learning environments and instruction in Lebanon’s public schools. The combination of these will ultimately result in better learning outcomes for students and will raise public trust and confidence in Lebanon’s public school system. This is assumed to be the development hypothesis for the project. PURPOSE AND INTENDED USE The evaluation will focus on assessing the relevance of the project activities to the needs of MEHE, and the project’s effectiveness in achieving its planned results along with sustainability of the project activities. The evaluation will generate lessons learned and provide recommendations that will assist the USAID/Lebanon in decision making related to (a) the type of approach the Mission should adopt in any future assistance to the MEHE, and (b) the type and scope of possible basic education projects in the future. The evaluation team also anticipates that USAID/Lebanon will use the results from the evaluation during its annual Portfolio Review. Secondary users include the DRASATI II implementing partners, MEHE and CERD. DRASATI II implementing partners may be able to use findings to adapt future work, and MEHE and CERD may be able to use findings in the management of teacher trainings and coaching following the project. 47 The evaluation will also provide insights into how the project activities complied with USAID’s cross cutting themes including gender equality and where/if applicable, environment protection. This should all be addressed within the context of SI’s Evaluation Quality Use and Impact (EQUI®) approach, processes and protocols. INTENDED AUDIENCE The primary audience for the evaluation includes: (i) the USAID/Lebanon Mission, particularly the Education Office Team, (ii) MEHE and CERD, and (iii) DRASATI II implementing partners. The final evaluation report will become publicly available on the Development Experience Clearinghouse (DEC). EVALUATION QUESTIONS 1. Relevance How relevant is the DRASATI II project to the needs of the public education sector in Lebanon? Explanation: This question addresses the relevance of the project to the MEHE’s long and short term needs as highlighted by its Education Sector Development Plan. 2. Effectiveness How and to what extent has the project achieved planned results? Explanation: The question addresses the effectiveness of DRASATI II, overall and specifically its various components. The answer to the question should explore what has worked and how, what has not worked as anticipated and why, and highlight any unintended outcomes. The question should include DRASATI II’s effectiveness in improving the learning environment in the public school, increasing learning opportunities through teacher training and extra-curricular activities, and increasing stakeholder engagement in the public school improvement. The question will take gender needs in to account when examining results. The answer to the question should also contribute to practical recommendations and lessons learned for making future programs more effective. 3. Sustainability What is the likelihood that the results DRASATI II has achieved are sustainable beyond the life of the project? Explanation: This question should generate conclusions about which DRASATI II results are likely to be sustainable beyond USAID support, with focus on MEHE and CERD’s role in sustainability. It should also explore the factors that contribute to sustainability or unsustainability. It should generate recommendations for any additional actions DRASATI II or 48 USAID can take that would enhance prospects for sustainability in the future, e.g., follow-on activities to replicate or scale up the project elements. EVALUATION DESIGN AND METHODS The evaluation team will use a primarily qualitative approach to answer the evaluation questions, conducting interviews with key stakeholders and focus group discussions with DRASATI II project staff, USAID/Lebanon staff, MEHE, CERD and direct project beneficiaries. This will be supplemented with quantitative data available from project monitoring data, but no primary quantitative data will be collected. The evaluation questions are mapped against data sources, collection methods and analysis methods are mapped at the end of this section in Table 1. Data Collection Data Collection will begin on August 29 and finish by September 19. The evaluation will employ the following data collection methods: 1. Desk Review: The evaluation team will do a systematic literature review of relevant internal and external documents, including the CDCS, RFA, contract agreement, contract modifications, initial assessments, reports (annual, quarterly, periodic assessments) and all other project documents relevant to answering the evaluation questions. In addition, the ET will review relevant external studies on Lebanon’s education sector that may lend additional context to the project’s relevance, effectiveness and sustainability. 2. Key Informant Interviews (KII): The qualitative interview protocols prepared by the evaluation team will particularly focus on exploring the relevance of the DRASATII to MEHE’s ESP, how effective the project has been and how sustainable the results it achieved will be. The team will conduct KII with USAID key staff members, World Learning, AMIDEST, AUB, ALLC, LU and DOT. In addition, the evaluation team will also interview key officials from MEHE and CERD. The KIIs will provide in-depth understanding about project achievements, factors that contributed towards achievement of the project results, and implementation challenges. The evaluation team will also conduct several exploratory interviews with key officials from UNICEF, UNDP, and the World Bank. These interviews are expected to provide information about current education projects being implemented/supported by these organizations and their future plans for supporting the education sector in Lebanon. This information will provide valuable insights into formulating recommendations for future USAID programing. The interview guides can be found in Annex 3. A preliminary list of key informants is found in Annex 1. This list includes the main points of contact and decision-makers from each organization for the project as identified by USAID and World Learning, supplemented with stakeholders identified during desk review. The evaluation team (ET) will update this list during the course of initial interviews with any 49 additional key informants interviewed and a full list of key informants consulted will be included as an annex in the final evaluation report. The ET anticipates conducting approximately 30 KIIs. 3. Focus Group Discussions: The ET, leveraging SI’s data collection resource partner Information International, will conduct focus group discussions (FGD) with selected project beneficiaries, primarily teachers and principals. The sampling for the FGDs, described in the following section, was determined based on a purposive sample of the different schools, principals and teachers involved in the project across Lebanon so that both high- and low-uptake participants will be included in the sample. The FGD questions will focus on what changes teachers and principals have observed in their practices, classrooms and schools following participation in DRASATI II activities. This information from FGDs will help the evaluation to answer the effectiveness and, to some extent, sustainability questions. The ET will also conduct FGDs with CERD and MEHE’s Guidance and Counseling Department (DOPS) staff who participated in DRASATI II trainings of trainers. These FGD questions will focus on the relevance and effectiveness of the training, the effectiveness and sustainability of the rollout trainings CERD conducts, and the effectiveness and sustainability of DOPS’s school coaching based on DRASATI’s TOTs. In addition, the ET expects to conduct FGDs with a sample of trainers from each of the main trainings in order to triangulate their feedback with those of trainees and get a more complete picture of the effectiveness and challenges of trainings. For the FGD sample, see Table 1 in the following section. 4. Direct observation of the ICT refresher course: DOT will be conducting the final refresher course for the ICT teacher training in early September at seven regional sites. ET members will observe part of the training and conduct the Task 4 FGDs with teachers who attend these trainings at the end of the training days. FGD Sampling by Component Task and Beneficiary Type To identify a representative set of direct beneficiaries for interviews that can give diverse points of view and ultimately allow the ET to document best practices and lessons learned, the ET has determined a purposive sample of DRASATI II’s direct beneficiaries. The final sample size balances time available against interviewing a broad set of participants in each component and sub-task. To maximize the number of people interviewed, these beneficiaries will mainly be interviewed through focus group discussions. The ET plans to conduct 39 FGDs with direct beneficiaries, 31 with teachers and principals and 8 with CERD and DOPS staff who participated in DRASATI trainings. The total number of FGDs will be 47 when including the FGDs with trainers. The ET estimates that this will cover approximately 231 schools and 366 individuals, although these numbers may increase or decrease depending on the ability of the invited participants to attend and MEHE’s approval of the schedule (discussed further in the limitations section). Table 1: FGD Sample 50 Stakeholder title # FGD s # Schools represente d (est.) # FGD Respondent s (est.) Component I: Provision of equipment and computer labs to public schools Task 1 Develop an ICT action plan based on MEHE's National Education Technology Strategic Plan This will be assessed through KIIs only Task 2 Procure ICT equipment for public schools Principals 5 28 28 Component 2: Provision of In-service Public School Teacher Training Task 3 Train public school teachers in the English language Teachers 5 50 50 Task 4 Train public school teachers in the use and maintenance of ICT equipment Teachers 5 35 35 Task 5 Expand extra-curricular activities to intermediate and secondary schools Teachers 3 29 29 Sub Task 5c Provide support to targeted Host Schools to alleviate pressure on public schools hosting high numbers of Syrian refugee students Teachers 5 30 40 Component 3: School Leadership Development with Parents and Community Involvement Task 6a Implement the school leadership development with parents and community involvement activity Principals 3 25 25 Task 6b School improvement program and community engagement School representatives 5 39 47 Total of FGDs with school beneficiaries 31 196 Government Trainees CERD trainers (ICT, Eng Lang+ELTM, PSS, ECA) 4 28 DOPS (ICT coaching and PSS) 4 28 Project Trainers ALLC English Language Trainers 1 8 AUB ICT TOT Trainers 1 8 DOT Coaches 1 8 AMIDEAST ECA Trainers (We’re still waiting for AMIDEAST’s 1 8 51 trainer list) AMIDEAST LDP Trainers 1 8 AMIDEAST SIP Trainers 1 8 World Learning PSS Trainers 1 9 World Learning ELTM Trainers 1 8 FGD Grand Total 47 231 366 The evaluation team categorized FGD participants according to component task. Furthermore, the categorization process took into account (where possible) participant mobility, geographic spread of schools, FGD venue proximity to FGD participants, and total number of DRARASTI II beneficiaries in each category. As a result five focus groups will be conducted each with principals who received ICT equipment, EL trainees, ICT trainees, SIP school representatives and host school teachers. It should be noted that the host school intervention is of special interest to USAID and DRASATI II for its focus on alleviating pressure on public school hosting high numbers of Syrian refugee students. For ECA activities and LDP training tasks, the evaluation will conduct three FGDs each. Table1 shows the breakdown of the FGDs by task and beneficiary type. In addition, the evaluation team considered the following factors when selecting the school FGD participants: • Regions: In order to ensure that different geographic areas in Lebanon are represented and facilitate teachers’ and principals’ ease of attending FGDs, the ET has stratified FGD selection by governorate, and within governorate, clustered FGDs in Cazas with high participation (or, for smaller Cazas where travel is easier, grouped 2-3 together). • Degree of uptake of the project: Based on feedback from project implementers, the evaluation team will identify a sample of schools where the project was relatively successful and a sample where the project struggled. This will allow the team to assess variation in factors that might affect the project results and therefore provide valuable information for identifying lessons learned. • School level: The project worked with both secondary and primary schools. Task 5c targeted only primary schools and Tasks 2 and 4 targeted only secondary schools, Tasks 3, 5 as a whole and 6 targeted both. The ET will ensure that both primary and secondary schools have been adequately represented in FGDs related to these tasks. The evaluation team will select the CERD and DOPS staff who participated in the training activities based on their availability For the relatively smaller trainer populations—e.g., trainings with only 8-9 trainers—the ET will include all trainers available in the FGD. For trainer populations that exceed the reasonable 52 size for a focus group, the ET will select 8-10 trainers based on availability to participate in FGDs. Because the ET expects that the primary value of these FGDs will be to triangulate responses with those of the direct beneficiaries to verify their feedback, the ET will conduct only one FGD with each trainer category. Host and non host-schools, a host school being defined as a school where more than 20% of the population is non-Lebanese, did not end up being a relevant category for stratification because the majority of the primary schools are currently hosting non-Lebanese students, while almost no secondary schools are. Consequently, school level (primary and secondary) stratification will ensure that there is representation of both host and non-host schools, though there will also be important differences between these schools unrelated to the non-Lebanese population. The list of targeted schools is found in Annex 2. Data Analysis Methods Once the ET completes data collection and receives all the focus group notes, it will reassemble in Social Impact Beirut Office for data analysis and development of findings, conclusions and recommendations. The study team will use a structured and systematic approach to analyzing the qualitative data and will triangulate multiple methods and sources to ensure the reliability and validity of results. The study team will analyze the qualitative data as follows: • Summarize notes from individual interviews and FGDs; • Code KII and FGD data according to themes relevant to the evaluation questions; • Prepare an evaluation data analysis matrix identifying the themes that emerge in the KIIs and FGDs to facilitate systematic and rigorous data analysis aimed at identifying key study findings as they relate to the evaluation questions; and • Prepare a detailed outline summarizing key findings based on all the data analysis, conclusions for each study question and overall recommendations. Social and Gender Analysis The evaluation team recognizes that gender and other social factors play a role in how direct and indirect beneficiaries are affected by and participate in project activities. The team anticipates that beneficiary gender and the representation of women in the overall DRASATI II direct beneficiaries may have implications for project relevance, effectiveness and sustainability. The majority of the public school teachers in Lebanon are female. DRASATI’s M&E data show that there is a slightly different participation rate between men and women in the activities—in general, women tend to attend trainings at lower rates, and the project hypothesized that this because of their household duties or are less able to travel. However, those who attend seem to more engage in trainings relative to men. 53 The ET will also explore how the trainings may have affected male and female students’ learning environments and classroom performance differently. However, because school is not yet in session, speaking with students directly is prohibitively difficult to organize given the evaluation timeline (see limitations below), so this will be based on the assessment of teachers and principals only. Lebanese public schools are under immense pressure as a result of the increasing number of non-Lebanese primary school students. These refugee children have special educational and psychosocial support needs. The project directly addresses this social factor through Psychosocial Support (PSS) interventions in a select number of schools. In addition to the 65 schools hosting refugee students, the evaluation team will explore how refugee children may have benefitted from other project interventions, whether these benefits were differently apportioned, how targeting of host schools directly or indirectly influenced project effectiveness and relevance. Limitations DRASATI II closes in December 2016 and the evaluation consequently is scheduled to be completed by mid October so that DRASRATI II partners and USAID will have sufficient time to benefit from the evaluation’s finding before the project closes. However, the timing of the data collection required by this schedule has implications for data collection. The public schools are currently not in session due to summer break. Teachers are expected to start attending in the first week of September for one week before the schools close again for the Eid holidays in the second week of September. Students will likely start attending by the end of September. Since September is the beginning of the academic year, school principals will be busy with new student registration. The strong likelihood of the limited availability of the direct beneficiaries (teachers and principals) might affect attendance in FGDs. Since the schools are not in session, including school children FGDs in the evaluation data collection is not possible. In addition to schools not being in session, most DRASATI II training activities have already ended. Thus the evaluation team is unable to visit sites and observe project activities except for the final DOT refresher course with ICT target schools. Conducting the FGDs with teachers, principals and CERD representatives and KII with MEHE and CERD officials are subject to permission from MEHE. The ET has submitted a request to MEHE, via USAID, to conduct the FGDs and KIIs. The ET anticipates that MEHE’s permission will come in time, but significant delay in approval or an unexpected denial of permission could result in excluding FGDs from the evaluation. In case of MEHE denial, the evaluation team will make modifications to the evaluation methodology and primarily rely on KIIs and FGDs with trainers. In order to ensure maximum participation of the stakeholder in KIIs, the evaluation team will conduct these interviews in a phased approach, interviewing first MEHE and implementing partner stakeholders whose schedules are less influenced by the beginning of the school year; then principals, who typically begin work prior to teachers; and finally teachers. 54 55 Table 1: Evaluation Design Matrix Evaluation Criteria and Question Sub-topic Data Sources Data Collection Methods Data Analysis Plan Relevance: How relevant is the DRASATI II project to the needs of the public education sector in Lebanon? MEHE’s long- and short-term needs as highlighted in the ESDP -RACE and other official MEHE documents Desk review Map DRASATI work to RACE plan Interview coding of context from MEHE officials and background documents -Ministry officials KIIs -Lebanon education sector background documents -Other education sector project donors and implementers, including UNICEF, World Bank , DFID, GIZ, Qitabi Effectiveness: How and to what extent has the project achieved planned results? Improving the learning environment in public schools through: Component 1: Provision of Equipment and Computer Labs to Public Schools -Teachers participating in ICT coaching/training KIIs Interview coding -Principals of schools with ICT procurement FGDs Comparison of plans to implementation -MEHE Desk Review -CERD and DOPS -DOT staff -AUB staff -World Learning staff -Project documents on procurement and distribution, training, use of equipment -ICT Action Plan Component 2: English teacher training and extracurricular activities -Teachers targeted for English language training (participants and non-participants) or ECA FGDs Interview coding Comparison of pre/post scores 56 -AMIDEAST staff and trainers Desk Review -ALLC staff and trainers -MEHE and CERD staff, including PCELT trainees -World Learning staff -BULATS and FCE exam scores Increasing stakeholder engagement in public school improvement through: -School representatives participating in SIP Component 3: School Leadership Development with Parents and Community Involvement -Principals of schools targeted for LDP -CERD and DOPS FGDs KIIs Interview coding -AMIDEAST staff Desk Review -World Learning staff -Project M&E data, reports -SIP grant applications and associated documents -LU staff Sustainability: What is the likelihood that the results DRASATI II has achieved are sustainable beyond the life of the project? Ministry uptake of activities during and after the project -Ministry officials FGDs KIIs Interview coding -CERD and DOPS Desk Review Project reports -DRASATI staff School (principal and teacher) continuation of activities or retention of training -Principals KIIs Interview coding -Teachers FGDs -Project data Desk Review 57 EVALUATION MANAGEMENT Deliverables, Timeline and Dissemination Plan As set out in the initial SOW, the ET will deliver the following to USAID: Deliverable Expected Delivery Date Scope of Work: PMSPL II team will prepare the SOW including methodologies, tools, and evaluation work plan and time schedule. The SOW will be submitted to the COR at USAID/Lebanon for approval. Completed and approved July 2016 Inception Report: PMSPL II team will conduct a systematic literature search, assembling key documents, and meeting with possible sub-contractors. PMSPL II will also prepare a draft methodology plan including, sampling methodology, beneficiary matrix, primary research questions, interview protocols, focus group questions and a list of proposed individuals to be interviewed. The methodology plan, interview schedules, and interview protocol will be presented to USAID staff prior the start of the evaluation. Submitted to USAID August 25, 2016 Briefings: The evaluation team will provide a briefing to USAID prior to the start of the evaluation and at the end of the evaluation. The initial briefing will focus on presenting the evaluation team plan for conducting the evaluation. Discussions with the Mission on the proposed methodologies for the evaluation will result in an Inception Report (a deliverable) that states the agreed methodologies. Toward the end of the field work, prior to the development of the Draft Report, the team will carry out a preliminary findings presentation at USAID. In addition to this, a final briefing meeting will be done after the final report is written. Initial in-briefing with USAID conducted August 19 Weekly email updates from the ET to the DRASTI II and PMPSL CORs, with meeting as necessary Preliminary findings presentation o/a September 28, subject to USAID agreement Final presentation by PMPSL in October or November, date TBD Draft report and outline of the final report: A draft report of the findings and recommendations will be submitted to USAID COR clearly describing findings, conclusions, and recommendations, within two weeks of completing the data collection. USAID will provide comments on the draft report within one week of submission. The evaluation team will consider USAID Due October 6 58 comments and revise the draft report accordingly and as appropriate for an independent evaluation. Where differences of opinion exist these will be presented in a Statement of Differences Annex. Final Report: The final evaluation report will be provided to USAID in MS Word and Adobe PDF within 15 calendar days following receipt of final comments from USAID. The report will include all of the components outlined in the “Structure of the Evaluation Report”, but not necessarily in the order specified below. The report will not exceed 25 pages (excluding appendices). Appendices should at a minimum include the scope of work for the evaluation; a list of individuals interviewed; a complete description of the methodology used for the evaluation; and any survey or questionnaires used. The report will be submitted in English, electronically in MS Word format and compliant with USAID Graphic Standards. Expected delivery October 26, subject to receipt of comments from USAID by October 12 All deadlines assume timely approval from MEHE for fieldwork and USAID’s ability to give comments on the inception report and draft evaluation report on the timelines discussed during the initial kickoff meeting. These deadlines and other evaluation activities are illustrated in the work plan found on the following two pages. PMPSL will upload the final approved evaluation report to the DEC for public access and distributed to WL, DRASATI and other stakeholders at the discretion of USAID/Lebanon. During the kickoff meeting, the ET discussed with USAID other stakeholders who might be interested in the evaluation results such as agencies implementing education-related work including UNICEF, DFID, World Bank and GIZ; USAID will share the report with these stakeholders at its discretion. The report will follow USAID branding requirements. Comments received after the budgeted end date for this evaluation may be addressed in a Statement of Differences Annex at the discretion of USAID/Lebanon. 59 Workplan 8/16 8/17 8/18 8/19 8/20 8/21 8/22 8/23 8/24 8/25 8/26 8/27 8/28 8/29 8/30 8/31 9/1 9/2 9/3 9/4 9/5 9/6 9/7 9/8 9/9 9/10 9/11 9/12 9/13 9/14 9/15 9/16 9/17 9/18 9/19 9/20 Document Review + TPM Interviews for Local Edu Expert USAID Kickoff Meeting X Inception Report Draft S X World Learning Meetings USAID Comments on inception report Check in with COR re: inception report Inception report finalized X KIIs and FGDs Interim Finding Analysis Work in Lebanon USAID involvement required International Travel Saturdays Sundays Eid Holiday Sep 10-12 Deliverables X Internal draft submission to SI HQ S 60 9/21 9/22 9/23 9/24 9/25 9/26 9/27 9/28 9/29 9/30 10/1 10/2 10/3 10/4 10/5 10/6 10/7 10/8 10/9 10/10 10/11 10/12 10/13 10/14 10/15 10/16 10/17 10/18 10/19 10/20 10/21 10/22 10/23 10/24 10/25 10/26 Presentation Prep S X Presentation to USAID X Depart Beirut Draft Report S X USAID comments Final Report S X Work in Lebanon USAID involvement required International Travel Remote Work Saturdays Sunday Eid Holiday Deliverables X Internal draft submission to SI HQS 61 Team Structure The evaluation team will consist of a Team Leader, Local Education expert, Performance Evaluation Specialist, technical support from PMSPL II evaluation team, and data collection from Information International, PMPSL’s data collection sub-contractor. SI’s head office will support evaluation management and quality. 1) Evaluation Team Leader, Dr. Jehanzaib Khan: Dr. Khan is responsible for coordinating and directing the overall evaluation, including preparation and submission of the draft and final evaluation reports. (45 days LOE) 2) Lebanon Education Expert, Hania Chahal: Ms. Chahal will contribute expert technical inputs into designing the evaluation methodology, creating FGD and KII instruments, conducting FGDs and KIIs, participating in data analysis, and contributing to report writing. (30 days LOE) 3) Performance Evaluation Specialist, Jade Lamb: Ms. Lamb contributes quality assurance to ensure that the evaluation follows SI’s EQUI® process. She additionally serves as a team member to contribute to evaluation methodology and instrument design, facilitate FGDs and KIIs, and contribute to data analysis and reporting. (30 days LOE) 4) Information International: Given the limited time available for fieldwork and complexity of the DRASATI project, SI will retain the PMPSL data collection partner, Information International, to conduct the bulk of the FGDs. A team of approximately 10 Information International facilitators will conduct the FGDs with direct beneficiaries over a period of 1-2 weeks. They will create detailed summary sheets for each focus group daily and share these with the ET as they are created. PMSPL II staff work alongside the other team members as part of the evaluation team, providing logistical and technical support. PMSPL II resident staff and HO staff participates in drafting, reviewing and editing the draft final report before circulating to the stakeholders for comment and review. Preparation of the evaluation report is the responsibility of the team leader/technical expert. Final editorial responsibility and quality control for the report is with Social Impact. Comments received are to be address by the technical expert supported by the PMSPL II staff. The draft evaluation report is to be reviewed by USAID/Lebanon and, at the request of USAID/Lebanon, a broad range of stakeholders. PMSPL II will certify that there is no conflict of interest or potential conflict of interest with respect to the performance of this evaluation on the part of the contractor and the contractor’s team members. Social Impact will guarantee that substitutions will not be made for individuals proposed as team members without the approval of USAID. 62 ANNEX 2: DOCUMENTS REVIEWED List of Key Documents Reviewed 1 Original DRASRAT II Contract Agreement 2 Project Modifications (Four) 3 Quarterly Report (11) 4 Work Plans (Three) 5 M&E Plans 6 School Selection Criteria 7 ICT Action Plan 8 ECA Strategy 9 Training Modules 10 DRASATI Master Plan 11 Contractor Performance Assessment Report CPAR) 12 MEHE’s Education Sector Development Plan 13 MEHE’s National Educational Technology Strategic Plan 14 Country Development Cooperation Strategy (CDCS) 2014-2018 63 ANNEX 3: DATA COLLECTION INSTRUMENTS It should be noted that these protocols are general guides with questions to ask across respondents/groups from each category of stakeholders, such that certain results can be compared across these groups. Depending on the stakeholder, questions may be omitted if they are not relevant. Ultimately, protocols were specifically tailored for each respondent and group as more relevant details became available. The protocols listed here are the ones that were finalized during the early phase of data collection. KII Guides Developing Rehabilitation Assistance to Schools and Teacher Improvement-II (DRASATI II) Final Evaluation Qualitative Interview Protocol Introduction: We work for Social Impact, a USAID contractor that provides performance monitoring and evaluation support for various development projects in Lebanon. The purpose of this interview is to gain further knowledge and insight into the work of the USAID-sponsored Developing Rehabilitation Assistance to Schools and Teacher Improvement (DRASATI II). As a knowledgeable person within your organization we value your input to our evaluation. We anticipate that your responses will help us gain further knowledge of DRASATI II’s contributions to basic education sector in Lebanon, understand challenges faced during implementation, and develop recommendations for future programming. Your answers will be used for data analysis purposes only and will be kept strictly confidential. No Interview respondents will be identified by name in the report. Before the interview begins, I would like to ask your permission to take notes to capture your responses and to tape record the interview so that we can further enhance our notes for data analysis. Organization: Date: Respondent’s Name: Venue: Official Title: Interviewee: Note-taker: USAID # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? - Activities/responsibilities - DR2 design and oversight - Role in DR2 modifications 64 b. How long have you been involved with DR2 and in what ways? RELEVANCE: 2. What is your understanding of the objectives of DR2 and their relevance to MEHE’s needs? a. Please describe the relevance of each component: i. C1: Provision of ICT equipment ii. C2: Provision of in-service teacher training iii. LDP with parent and community involvement - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 3 Could you explain when and why the Assessment component was cancelled? a. What factors led to this decision? b. Did the cancellation of the assessment component affect/delay project activities? If yes, how? - The World Bank’s similar assessment activity commissioned by MEHE 4 How were the standards developed for ECA, LDP, SIP and parents/community involvement? a. Who developed these standards? b. How were these standards used by DRASATI II? Explain? - The World Bank; MEHE? - Through a collaborative process? 5 Please share your understanding of the factors that led to the contract modifications a. What changes were included in the DR2 scope of work, and why? b. How relevant are those changes to the MEHE’s ESP, USAID’s Education Strategy? - Purpose of Mod 1 and Mod 2 - Schools overwhelmed by Syrian refugee students - Provide support to MEHE EFFECTIVENESS: 6 In what ways has the project made progress toward the desired outcomes? a. Component 1 i. Develop an ICT action plan ii. Procure ICT equipment b. Component 2 iii. Train teachers in the English language iv. Train teachers in the ICT use and maintenance v. Expand ECA to intermediate and secondary schools c. Component 3 vi. Implement the school leadership development SIP and community engagement Examples of achievements/progress for each task 7 Please describe how the PSS and school stipend activities contributed to the project objectives? a. What kinds of challenges do the prevalence of refugee children in primary schools pose for the effectiveness of - Access to quality instruction and learning environment Better cooperation and relationship (social cohesion) among Lebanese and Syrian students 65 project activities? b. Please describe ways in which host schools overwhelmed by non-Lebanese students benefited from the project? c. In your understanding how PSS schools in general and the targeted 65 host schools in particular benefited from PSS intervention? Please tell us how the targeted 65 host schools benefited from school stipend intervention? 8 To what extent were the planned interventions sufficient for achieving desired results? a. How did these results contribute toward achieving project objectives? Examples? - Improvement in school learning environment - Increased learning opportunities through ELT and ACA - Increased stakeholder engagement through LDP and SIP 9 Please tell us about some of the key DR2 outcomes/products that benefited MEHE What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details. Examples: - Development of ICT Action Plan - Development and institutionalization of ICT strategy 10 In your opinion, what factors played key role in achieving planned results? a. What do you think is the project’s biggest success? 11 Did DR2 encounter any particular challenges implementing the project? Explain? a. Partnership? b. Management? c. Other? - Managing grants, selecting appropriate partners - Working with MEHE - Working with other implementing partners/service providers 12 Did the project design/implementation include a specific strategy for including women? a. What steps were taken to ensure participation of women? Explain? b. What steps were taken to ensure male and female students equally benefited from the interventions? - ICT and English language trainings - LDP, ECA trainings? 13 Do you have knowledge of other education programs implemented since 2013 that supported public education in Lebanon? a. Please tell us if they might have contributed directly or indirectly toward DRARASTI II achievements? Please explain. - The World Bank - UNICEF - UNDP - GIZ - DFID SUSTAINABILITY 14 To what extent did the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Why? b. What factors contributed to sustainability of DR2 results c. Please describe factors that contributed to unsustainability of DR2 results d. Is there anything else the project or future projects should do to ensure sustainability - ICT and other action plans developed - Post DR2 utility of various standards developed by DR2 - MEHE’s role in sustainability of DR results - Support to schools overwhelmed by Syrian refugee students 15 If you were to redo/redesign the project, what improvements would you introduce? Suggestions/recommendations for improving future programming 66 DRASATI II COP and DCOP # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities - DR2 design and oversight - Role in DR2 modifications RELEVANCE: 2. What is your understanding of the objectives of DR2 and their relevance to MEHE’s needs? a. Please describe the relevance of each component: i. C1: Provision of ICT equipment ii. C2: Provision of in-service teacher training iii. LDP with parent and community involvement - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 3 Could you explain when and why the Assessment component was cancelled? a. What factors led to this decision? b. Did the cancellation of the assessment component affect/delay project activities? If yes, how? - The World Bank’s similar assessment activity commissioned by MEHE 4 How were the standards developed for ECA, LDP, SIP and parents/community involvement? a. Who developed these standards? b. How were these standards used by DRASATI II? Explain? - The World Bank; MEHE? - Through a collaborative process? 5 What was the purpose of the contract modifications (1 and 2)? Explain? a. What changes were included in the DR2 scope of work, and why? b. How relevant are those changes to the MEHE’s ESP, USAID’s Education Strategy? - Support schools overwhelmed by Syrian refugee students - Provide support to MEHE EFFECTIVENESS: 6 In what ways has the project made progress toward the desired outcomes? d. Component 1 i. Develop an ICT action plan ii. Procure ICT equipment e. Component 2 i. Train teachers in the English language ii. Train teachers in the ICT use and maintenance iii. Expand ECA to intermediate and secondary schools f. Component 3 i. Implement the school leadership development ii. SIP and community engagement Examples of achievements/progress for each task 67 7 Please describe how the PSS and school stipend activities contributed to the project objectives? a. What kinds of challenges do the prevalence of refugee children in primary schools pose for the effectiveness of project activities? b. Please describe ways in which host schools overwhelmed by non-Lebanese students benefited from the project? c. In your understanding how PSS schools in general and the targeted 65 host schools in particular benefited from PSS intervention? d. Please tell us how the targeted 65 host schools benefited from school stipend intervention? - Access to quality instruction and learning environment - Better cooperation and relationship (social cohesion) among Lebanese and Syrian students 8 To what extent were the planned interventions sufficient for achieving desired results? a. How did these results contribute toward achieving project objectives? - Improvement in school learning environment - Increased learning opportunities through ELT and ACA - Increased stakeholder engagement through DLP and SIP - Increased access to quality instruction & learning environment in host schools - Improved social cohesion among Lebanese and Syrian students 9 Please tell us about some of the key DR2 outcomes/products that benefited MEHE a. What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? - Development of ICT action plan, training manuals, - Institutionalization of ECA strategy - LDP training 10 In your opinion, what factors played key role in achieving planned results? a. What do you think is the project’s biggest success? - Examples of key factors 11 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. 12 Did DR2 encounter any particular challenges implementing the project? Explain? a. Partnership? b. Management? c. Other? - Managing grants, selecting appropriate partners - Working with MEHE - Working with other implementing partners/service providers - USAID 13 Did the project design/implementation include a specific strategy for including women? a. What steps were taken to ensure participation of women? Explain? b. What steps were taken to ensure male and female students equally benefited from the interventions? - ICT and English language trainings - LDP, ECA trainings? - PSS - ICT equipment use - Participation in ECA 14 Do you have knowledge of other education programs implemented since 2013 that supported public education in Lebanon? a. Please tell us if they might have contributed directly or indirectly toward DRARASTI II achievements? Please explain. - The World Bank - UNICEF - UNDP - GIZ - DFID 68 SUSTAINABILITY 15 To what extent did the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Why? b. What factors contributed to sustainability of DR2 results c. Please describe factors that contributed to unsustainability of DR2 results d. Is there anything else the project or future projects should do to ensure sustainability - ICT and other action plans developed - Post DR2 utility of various standards developed by DR2 - MEHE’s role in sustainability of DR results - Support to schools overwhelmed by Syrian refugee students 16 If you were to redo/redesign the project, what improvements would you introduce? - Suggestions/recommendations for improving future programming 17 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. DRASATI II M&E Manager # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities RELEVANCE: 2. What is your understanding of the objectives of DR2 and their relevance to MEHE’s needs? - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 3 Were the ECA, LDP and SIP standards (developed by the World Bank for MEHE) used/referenced in carrying out and monitoring DRASATI II performance? a. If yes, explain how? If no, explain why not? EFFECTIVENESS: 69 4 In what ways has the project made progress toward the desired outcomes? g. Component 1 i. Develop an ICT action plan ii. Procure ICT equipment h. Component 2 i. Train teachers in the English language ii. Train teachers in the ICT use and maintenance iii. Expand ECA to intermediate and secondary schools i. Component 3 i. Implement the school leadership development ii. SIP and community engagement Examples of achievements/progress for each task 5 What M&E mechanisms were developed to monitor project activities/progress? a. How did DR2 track (1) project activities targeting host schools and (b) benefits these schools received as a result of DR2? (e.g., PSS, Stipend, SIP) 6 Please describe how the PSS and school stipend activities contributed to the project objectives? e. Please describe ways in which host schools overwhelmed by non-Lebanese students benefited from the project? f. In your understanding how PSS schools in general and the targeted 65 host schools in particular benefited from PSS intervention? g. Please tell us how the targeted 65 host schools benefited from school stipend intervention? 7 To what extent were the planned interventions sufficient for achieving desired results? a. How did these results contribute toward achieving project objectives? - Improvement in school learning environment - Increased learning opportunities through ELT and ACA - Increased stakeholder engagement through DLP and SIP 8 Please tell us about some of the key DR2 outcomes/products that benefited MEHE a. What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? - Development of ICT action plan, training manuals, - Institutionalization of ECA strategy - LDP training 9 In your opinion, what factors played key role in achieving planned results? a. What do you think is the project’s biggest success? - Examples of key factors 10 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. - Role of partners/service provides 11 What support did you receive from the implementing partners and service providers in monitoring project performance? a. What M&E support did you provide the implementing partners/service providers 70 12 Please describe how the PSS and school stipend activities contributed to the project objectives? a. In your understanding how PSS schools in general and the targeted 65 host schools in particular benefited from PSS intervention? b. In your understanding how the targeted 65 host schools in benefited from school stipend intervention? - 13 Did DR2 encounter any particular challenges in monitoring project performance? Explain? a. Delay in provision of monitoring data by the implementing partners (Partnership)? b. Other? - Difficulties in having access to MEHE data - Quality and relevance of data 14 What steps did DR2 take to ensure participation of women? Explain? a. How were women participation monitored? Please explain? b. How were the women participation data used for project implementation? (e.g., project target, resource allocation etc.) c. In what ways can the DR2 monitoring data—particularly on women participation-be useful to USAID and MEHE? - Project activities including, ICT and English language trainings - LDP, ECA trainings? - PSS 15 Do you have knowledge of other education programs implemented since 2013 that supported public education in Lebanon? Please tell us if they might have contributed directly or indirectly toward DRARASTI II achievements? Please explain. - The World Bank - UNICEF - UNDP - GIZ - DFID SUSTAINABILITY 16 To what extent did the project implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Why? b. In your opinion, can the DR2 monitoring data be used (USAID, MEHE) toward the sustainability of the project results? Explain? c. What other factors contributed to sustainability of DR2 results d. Please describe factors that contributed to unsustainability of DR2 results e. Is there anything else the project or future projects should do to ensure sustainability - ICT and other action plans developed - Post DR2 utility of various standards developed by DR2 - MEHE’s role in sustainability of DR results - Support to schools overwhelmed by Syrian refugee students 17 If you were to redo/redesign the project, what improvements would you introduce? - Suggestions/recommendations for improving future programming 18 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. 71 DRASATI II Tech-ED Manager # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities RELEVANCE: 2. What is your understanding of the ICT related objective (#1) of DR2 and it’s relevance to MEHE’s needs? - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan - Directly or indirectly supported USAID’s Education Policy 3 Please describe how was the ICT action plan developed? Describe the process? a. What contribution did DR2 partners make toward the development of the ICT action plan? b. What role, if any, MEHE play in the development of the ICT action plan? - Collaborative process - EFFECTIVENESS: 4 In what ways has the provision of ICT equipment and ICT training contributed toward the project results a. Component 1 i. Develop an ICT action plan ii. Procure ICT equipment b. Component 2 iii. Train teachers in the ICT use and maintenance iv. Improved school capacity in ICT maintenance Examples of achievements/progress: 5 In your opinion, what benefits the various tiers of schools received from the ICT equipment and ICT coaching - 6 To what extent were the planned ICT related interventions sufficient for achieving desired results? a. How did these results contribute toward achieving project objective #1? - Improvement in school learning environment through ICT activities 7 Please tell us about some of the key DR2 outcomes/products (related to ICT equipment and ICT training) that benefited MEHE What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? 72 9 In your opinion, what factors played key role in achieving planned results? a. What do you think is the project’s biggest success? - Examples of key factors 10 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. - Role of AUB 11 Did DR2 encounter any particular challenges in procurement and provision of ICT equipment to public schools? Explain? a. Please describe your experience working with AUB? - Difficulties in having feedback/approval from MEHE on ICT action plan - Working with AUB—good/poor relationship 12 What steps did DR2 take to ensure participation of women? Explain? a. What role if any AUB play in ensuring women representation in the ICT action plan, ICT coaching? b. What steps were taken to ensure male and female students equally benefited from the interventions? - ICT Action plan, ICT coaching 13 Do you have knowledge of other similar education programs implemented since 2013 that supported public education in Lebanon? a. Please tell us if they might have contributed directly or indirectly toward DRARASTI II achievements? Please explain. - SUSTAINABILITY 14 4To what extent did the project implementation specifically address sustainability and how? a. In your opinion, are the project results related to ICT sustainable? Why? b. What other factors contributed to sustainability of DR2 results c. Please describe factors that contributed to unsustainability of DR2 results d. Is there anything else the project or future projects should do to ensure sustainability - ICT and other action plans developed - Post DR2 utility of various standards developed by DR2 - MEHE’s role in sustainability of DR results - Support to schools overwhelmed by Syrian refugee students 15 If you were to redo/redesign the project, what improvements would you introduce? Suggestions/recommendations for improving future programming 16 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. MEHE # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? - Activities/responsibilities - DR2 design and oversight - Role in DR2 modifications 73 b. How long have you been involved with DR2 and in what ways? RELEVANCE: 2. What is your understanding of the objectives of DR2 and their relevance to MEHE’s needs? a. Please describe the relevance of each component: i. C1: Provision of ICT equipment ii. C2: Provision of in-service teacher training iii. LDP with parent and community involvement - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 3 How were the standards developed for ECA, LDP, SIP and parents/community involvement for MEHE’ Effective School Profile? c. Who developed these standards? d. How were these standards used by DRASATI II? Explain? - The World Bank; MEHE? - Through a collaborative process? 4 In your understanding, what was the purpose of the contract modifications (1 and 2)? Explain? a. Do you know the changes that were included in the DR2 scope of work, and why? b. How relevant are those changes to the MEHE’s ESP? - Support schools overwhelmed by Syrian refugee students - Provide support to MEHE EFFECTIVENESS: 5 Please describe ways in which DRASARTI II activities supported public schools in Lebanon? a. Component 1 i. Develop an ICT action plan ii. Procure ICT equipment b. Component 2 iii. Train teachers in the English language iv. Train teachers in the ICT use and maintenance v. Expand ECA to intermediate and secondary schools c. Component 3 vi. Implement the school leadership development vii. SIP and community engagement Examples of support/progress for each task 6 What kind of support did MEHE provide to DRASRATI II to achieve its planned results? 7 Please describe DRASATI II’s support to host schools overwhelmed by non-Lebanese students h. What kinds of challenges do the prevalence of refugee children in primary schools pose for the MEHE? - Access to quality instruction and learning environment - Better cooperation and relationship (social cohesion) among Lebanese and Syrian students 74 8 How did the PSS and school stipend activities contribute to providing access to quality instruction & learning environment? a. How these did activities contribute to improving cooperation & relationship among Lebanese and Syrian Students? b. In your understanding how PSS schools in general and the targeted 65 host schools in particular benefited from PSS intervention? 9 To what extent were the planned DRASRATI II interventions sufficient for achieving desired results? - Improvement in school learning environment - Increased learning opportunities through ELT and ACA - Increased stakeholder engagement through DLP and SIP - Increased access to quality instruction & learning environment in host schools - Improved social cohesion among Lebanese and Syrian students 11 Please tell us about the DR2 outcomes/products that benefited MEHE b. What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? c. Future use by MEHE of strategies, manuals, curricula developed by DR2 - Development of ICT action plan, training manuals, - Institutionalization of ECA strategy - LDP training 12 In your opinion, what factors played key role in helping DRASRATI achieve planned results? b. What do you think is the project’s biggest success? - Examples of key factors 13 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. 14 Tell us about your experience of working DRSARATI II partners. What kind of challenges did MEHE face in supporting DRASATI II implementation? Explain? d. Partnership? e. Management? f. Other? - Working with WL - Working with other implementing partners/service providers - Working with USAID 15 In your knowledge, did the project design/implementation include a specific strategy for including women? c. What steps did MEHE take to ensure participation of women? Explain? d. What steps did MEHE take to ensure male and female students equally benefited from the DR2 interventions? - Action plans - ICT and English language trainings - ECA curricula and trainings? - PSS and school stipends - LDP and SIP - ICT equipment use - Participation in ECA 16 Please tell about other similar programs implemented since 2013 to support public education in Lebanon? Please tell us if they might have contributed directly or indirectly toward DRARASTI II achievements? Please explain. - The World Bank - UNICEF - UNDP - GIZ - DFID SUSTAINABILITY 75 17 In your knowledge, to what extent did the project design/implementation specifically address sustainability and how? e. In your opinion, are the project results sustainable? Why? f. What factors contributed to sustainability of DR2 results g. Please describe factors that contributed to unsustainability of DR2 results h. How do you see MEHE’s role in the sustainability of DR2 results? i. Is there anything else the project or future projects should do to ensure sustainability - ICT and other action plans developed - Post DR2 utility of various standards developed by DR2 - Support to schools overwhelmed by Syrian refugee students 18 If you were to redo/redesign the project, what improvements would you introduce? - Suggestions/recommendations for improving future programming 19 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. CERD # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities - DR2 design and oversight - Role in DR2 modifications RELEVANCE: 2. What is your understanding of the objectives of DR2 and their relevance to MEHE’s needs? a. Please describe the relevance of each component: iv. C1: Provision of ICT equipment v. C2: Provision of in-service teacher training vi. LDP with parent and community involvement - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan - Support schools overwhelmed by Syrian refugee students 3 How were the standards developed for ECA, LDP, SIP and parents/community involvement for MEHE’ Effective School Profile? a. Who developed these standards? b. Were these standards used by DRASATI II? Explain? - The World Bank; MEHE? - Through a collaborative process? EFFECTIVENESS: 76 4 Please describe ways in which DRASARTI II activities supported public schools in Lebanon? a. Component 1 i. Develop an ICT action plan ii. Procure ICT equipment b. Component 2 i. Train teachers in the English language iii. Train teachers in the ICT use and maintenance iv. Expand ECA to intermediate and secondary schools c. Component 3 i. Implement the school leadership development ii. SIP and community engagement Examples of support/progress for each task 5 What kind of support did CERD provide to DRASRATI II to achieve its planned results? 6 Please describe DRASATI II’s support to host schools overwhelmed by non-Lebanese students d. What kinds of challenges do the prevalence of refugee children in primary schools pose for the CERD? - Access to quality instruction and learning environment - Better cooperation and relationship (social cohesion) among Lebanese and Syrian students 7 How did the PSS and school stipend activities contribute to providing access to quality instruction & learning environment? a. How these did activities contribute to improving cooperation & relationship among Lebanese and Syrian Students? b. In your understanding how PSS schools in general and the targeted 65 host schools in particular benefited from PSS intervention? 8 To what extent were the planned DRASRATI II interventions sufficient for achieving desired results? - Improvement in school learning environment - Increased learning opportunities through ELT and ECA - Increased stakeholder engagement through DLP and SIP - Increased access to quality instruction & learning environment in host schools - Improved social cohesion among Lebanese and Syrian students 9 Please tell us about the DR2 outcomes/products that benefited CERD a. What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by CERD. Please give details? b. Future use by CERD of strategies, manuals, curricula developed by DR2 - Development of ICT action plan, training manuals, - Institutionalization of ECA strategy - LDP training 10 In your opinion, what factors played key role in helping DRASRATI II achieve planned results in relation trainings? c. What do you think is the project’s biggest success? - Examples of key factors 11 In your opinion, what factors supported timely project - Examples of factors in relation to timely achievement of project results, 77 performance? Examples? - Factors that helped in overachievement, underachievement. 12 Tell us about your experience of working DRSARATI II partners. What kind of challenges did CERD face in supporting DRASATI II implementation? Explain? a. Partnership? b. Management? c. Other? - Working with WL - Working with other implementing partners/service providers - Working with USAID 13 In your knowledge, did the project design/implementation include a specific strategy for including women? a. What steps did CERD take to ensure participation of women? Explain? b. What steps did CERD take to ensure male and female students equally benefited from the DR2 interventions? - Action plans - ICT and English language trainings - ECA curricula and trainings? - PSS - LDP and SIP - ICT equipment use - Participation in ECA 14 Please tell about other similar programs implemented since 2013 to support public education in Lebanon? Please tell us if they might have contributed directly or indirectly toward DRARASTI II achievements? Please explain. - The World Bank - UNICEF - UNDP - GIZ - DFID SUSTAINABILITY 15 In your knowledge, to what extent did the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Why? b. What factors contributed to sustainability of DR2 results c. Please describe factors that contributed to unsustainability of DR2 results d. How do you see CERD’s role in the sustainability of DR2 results? e. Is there anything else the project or future projects should do to ensure sustainability - ICT and other action plans developed - Post DR2 utility of various standards developed by DR2 - Support to schools overwhelmed by Syrian refugee students 16 If you were to redo/redesign the project, what improvements would you introduce? - Suggestions/recommendations for improving future programming 17 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. AUB # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities 78 RELEVANCE: 2 What is your understanding of the ICT related objective (#1) of DR2 and it’s relevance to MEHE’s needs? - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 3 Please describe how was the ICT action plan developed? Describe the process? a. What contributions did AUB make toward the development of the ICT action plan? b. What role, if any, did MEHE play in the development of the ICT action plan? - Collaborative process - AUB’s role EFFECTIVENESS: In what ways has the development of ICT TOT modules contributed toward the project results a. What steps did AUB take to ensure MEHE’s support and ownership of these modules? Examples of achievements in relation to: 4 In what ways has the provision of ICT equipment and ICT training contributed toward the project results = a. Component 1 i. Develop an ICT action plan ii. Procure ICT equipment b. Component 2 iii. Train teachers in the ICT use and maintenance iv. Improved school capacity in ICT maintenance In your opinion, how did the various tiers of schools receiving ICT equipment and ICT coaching influence results? a. Was the targeting of tiers appropriate? - 5 To what extent were the planned ICT related interventions sufficient for achieving desired results? a. How did these results contribute toward achieving project objective #1? - Improvement in school learning environment 6 In your opinion, what factors played key role in achieving planned results? a. What do you think is the AUB’s biggest success? - Examples of key factors - MEHE’s willingness to provide ICT based education to the public schools 7 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. - Role of AUB 8 Please describe ways in which CERD trainings (TOT) contributed toward improving ICT use and maintenance in the public schools? - 79 9 Please tell us about some of the key DR2 outcomes/products that benefited MEHE What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? - 10 Did AUB encounter any particular challenges in the development of the manuals, and procurement and provision of ICT equipment to public schools? Explain? a. Please describe you experience working with CERD trainers? - Difficulties in having feedback/approval from MEHE on ICT action plan - Working with CERD—good/poor relationship 11 What steps did AUB take to ensure participation of women? Explain? a. How did the ICT Action Plan ensure male and female students have equal opportunity to ICT based learning? b. What role if any AUB play in ensuring women representation in the ICT action plan, and CERD TOT? - ICT Action plan, - Working with CERD to ensure women participation SUSTAINABILITY 12 To what extent did the project implementation specifically address sustainability and how? a. In your opinion, are the project results related to ICT sustainable? Why? b. What other factors contributed to sustainability of DR2 results c. Please describe factors that contributed to unsustainability of DR2 results d. Is there anything else the project or future projects should do to ensure sustainability - ICT and other action plans developed - Post DR2 utility of the ICT TOT modules developed by AUB - MEHE’s role in sustainability of these results 13 If you were to redo/redesign the project, what improvements would you introduce? Suggestions/recommendations for improving future programming 14 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. AMIDEAST # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities - DR2 design and oversight - Role in DR2 modifications RELEVANCE: 2. What is your understanding of the objectives of DR2 and their relevance to MEHE’s needs? a. Please describe the relevance of component: i. C2: TASK 5Provision ECA ii. LDP with parent and community involvement - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 80 3 Please describe how was the ECA action plan/strategy developed? Describe the process? a. What contributions did AMIDEAST make toward the development of the ECA action plan? b. What role, if any, MEHE play in the development of the ECA action plan? - Collaborative process - AMIDEAST’s role 5 Please describe how was the LDP and SIP action plans/strategies developed? Describe the process? a. What contributions did AMIDEAST and LU make toward the development of the LDP action plan/Strategy? b. What contributions did AMIDEAST make toward the development of the SIP action plan/Strategy c. What role, if any, MEHE played in the development of the LDP and SIP action plans/strategies? - AMIDEAST’s specific role in LDP relative to LU EFFECTIVENESS: ECA 6 In what ways has the development of ECA strategy and training modules contributed toward the project results a. What steps did AMIDEAST take to ensure MEHA’s support and ownership of the strategy and modules? 7 In what ways has the implementation of ECA interventions contributed toward the project results? Examples of achievements in relation to: Component 2 - Expand ECA to intermediate and secondary schools - ECA in host schools 8 Please describe how the host schools overwhelmed by non-Lebanese students benefited from the ECA intervention? a. What challenges did the large number of non-Lebanese students in primary schools pose for implementation and how were they addressed? 9 Please describe ways in which ECA trainings (MEHE staff) contributed toward increasing learning opportunities in the public schools? a. Please describe way in which ECA training provided to the public school staff increase learning opportunities in the public schools EFFECTIVENESS: LDP and SIP 10 In what ways has the provision of LDP training contributed toward the project results? a. In what ways has the development of SIP strategy/plan contributed toward the project results? Examples of achievements in relation to: Component 3 - Implement the school leadership development - SIP development and implementations Achievements: 81 11 Please describe ways in which the implementation of SIP trainings contributed toward the project results? a. What steps did AMIDEAST take to ensure MEHE’s support and ownership of the strategy and training? Increased stakeholder engagement improved schools (environment) - Support from LU 12 Please describe how the host schools, overwhelmed by non-Lebanese students, benefited from the SIP intervention? a. Please provide specific examples b. What challenges did the number of non-Lebanese students in primary schools pose to results and implementation? 13 Please describe ways in which SIP trainings provided to the public schools contributed toward increase stakeholder engagement in school development? a. Please describe way in which SIP implementation provided increased stakeholder participation in school development 14 To what extent were the planned interventions sufficient for achieving desired results? a. How did these results contribute toward achieving project objectives? - Improvement in school learning environment - Increased learning opportunities through ELT and ECA - Increased stakeholder engagement through LDP and SIP 15 Please tell us about some of the key ECA, LDP and SIP outcomes/products that benefited MEHE a. What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? - Development of ECA action - Institutionalization of ECA strategy - LDP training - Training manuals/material 16 In your opinion, what factors played key role in achieving planned results? b. What do you think is the project’s biggest success? - Examples of key factors 17 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. 18 Did DR2 encounter any particular challenges implementing the ECA, LDP and SIP activities? Explain? a. Please describe your experience of working with LU b. Other? - Working with MEHE - Working with LU - Working other DR2 partners/service providers - USAID 19 Did the project design/implementation include a specific strategy for including women? a. What steps were taken to ensure participation of women? Explain? b. What steps were taken to ensure female students’ and teachers’ participation in ECA and SIP? - ECA strategy and ECA trainings - LDP, SIP trainings? - SIP implementation 20 Do you have knowledge of other similar education programs implemented since 2013 that supported public education in Lebanon? a. Please tell us if they might have contributed directly or indirectly toward DRARASTI II achievements? Please explain. - The World Bank - UNICEF - UNDP - GIZ - DFID SUSTAINABILITY 82 21 To what extent did the project design/implementation specifically address sustainability and how? a. In your opinion, are the project results sustainable? Why? b. What factors contributed to sustainability of DR2 results c. Please describe factors that contributed to unsustainability of DR2 results d. Is there anything else the project or future projects should do to ensure sustainability - ECA strategy developed and institutionalized - Post DR2 utility of ECA strategy and trainings material developed by DR2 - MEHE’s role in sustainability of DR results - Support to schools overwhelmed by Syrian refugee students 22 If you were to redo/redesign the project, what improvements would you introduce? Suggestions/recommendations for improving future programming 23 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. LU # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities RELEVANCE: 2 What is your understanding of the objectives of DR2 and their relevance to MEHE’s needs? a. Please describe the relevance specifically of (component 3, task 6a) LDP with parent and community involvement - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 3 Please describe how was the LDP action plan/strategy developed? Describe the process? a. What contributions did LU make toward the development of the LDP plan? b. What role, if any, MEHE play in the development of the LDP plan? - Collaborative process - AMIDEASTs role EFFECTIVENESS: 4 In what ways has the implementation of LDP activities contributed toward school leadership development? Assisted/built leadership skills in school principals to manager schools with limited resources and increased student enrollment 5 In your opinion how can/are schools benefiting from these LDP trainings? 83 6 Please describe how the host schools overwhelmed by non-Lebanese students benefited from the LDP intervention? a. Did the number of non-Lebanese students in primary schools pose any challenges to or opportunities for the LDP work? b. How were those challenges addressed in the LDP training? 7 Please describe way in which LDP training provided to the public school principal increased stakeholders (school staff, community, parents, MEHA) engagement in school development? 8 Please tell us about some of the key LDP outcomes/products that benefited MEHE a. What strategic plans, policies, manuals, curricula developed by LU for DR2 are adopted and used by MEHE. Please give details? b. Please tell us about some of the key LDP outcomes/products that benefited DR2 - LDP training 9 What do you think is the project’s (LDP’s) biggest success? - Examples of key factors 10 In your opinion, what factors supported timely project performance? Examples? 11 Did LU encounter any particular challenges implementing the LDP activities? Explain? a. Please describe your experience of working with AMIDEAST b. Other? - Working with MEHE - Working other DR2 partners/service providers - USAID 12 Did the LDP design/implementation include a specific strategy for including women? c. What steps were taken to ensure participation of women? Explain? - LDP strategy and LDP trainings SUSTAINABILITY 13 To what extent did the LDP plan/implementation specifically address sustainability and how? a. In your opinion, are the LDP results sustainable? Why? b. What factors contributed to sustainability of these results c. Please describe factors that contributed to unsustainability of the results d. Is there anything else the project or future projects should do to ensure sustainability - Post DR2 utility of LDP trainings - MEHE’s role in sustainability of LDP results - Support to schools overwhelmed by Syrian refugee students 14 If you were to redo/redesign the project/LDP, what improvements would you introduce? Suggestions/recommendations for improving future programming 15 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. ALLC # Questions Prompts 84 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities RELEVANCE: 2. What is your understanding of the English language training related objective (#2) of DR2 and it’s relevance to MEHE’s needs? - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan 3 Please describe how the English language-training (ELP) plan was developed? Describe the process? a. What was ALLC’s role? b. What contributions did WL make toward the development of the ELP? c. What role, if any, MEHE play in the development of the ELP plan? - Collaborative process - DR2 partners’ role EFFECTIVENESS: 4 In your opinion, the administration of BULATS was sufficient in identifying English language learning needs of the public school teachers? Explain how? a. How did teachers generally responded to BULATS 5 In what ways has the development of EL training contributed toward increasing learning opportunities in the public school? a. What steps did AUB take to ensure MEHE’s support and ownership of the modules developed by ALLC? Examples of achievements in relation to: Component 2 - Train teachers in EL 6 Please tell us about some of the key DR2 outcomes/products that benefited MEHE What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? 7 To what extent were the planned EL training related interventions sufficient for achieving desired results? a. How did these results contribute toward achieving project objective #2? Increasing learning opportunities at the public schools 8 In your opinion, what factors played key role in achieving planned results? a. What do you think is the ALLC’s biggest success? - Examples of key factors - MEHE’s willingness to support EL use in the public schools 9 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. - Role of ALLC and WL 85 10 Please describe ways in which EL trainings contributed toward improving teaching of English, Math and Science subjects the public schools? - 11 Did ALLC encounter any particular challenges in the development of the EL training manuals, providing trainings and facilitating FCE? Explain? a. Please describe you experience working with PCELT trainers? - Difficulties in having feedback/approval from MEHE on EL training plan - Working with CERD—good/poor relationship 12 What steps did ALLC take to ensure participation of women? Explain? b. What role if any did ALLC play in ensuring women representation in the EL plan, training and FCE? c. What steps were taken to ensure male and female students equally benefited from the interventions? SUSTAINABILITY 13 To what extent did the project implementation specifically address sustainability and how? a. In your opinion, are the project results related to EL training sustainable? Why? b. What other factors contributed to sustainability of these results c. Please describe factors that contributed to unsustainability of the results d. Is there anything else the project or future projects should do to ensure sustainability - EL plan and trainings developed - Post DR2 utility of the PCELT trainers and EL training - MEHE’s role in sustainability of these results 14 If you were to redo/redesign the project, what improvements would you introduce? - Suggestions/recommendations for improving future programming 15 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. DOT # Questions Prompts 1. Could you describe your role within your organization and your involvement in DRASRATI II? a. How long have you served at this position? b. How long have you been involved with DR2 and in what ways? - Activities/responsibilities RELEVANCE: 2. What is your understanding of the ICT related objective (#1) of DR2 and it’s relevance to MEHE’s needs? - MEHE’s short and long term needs - MEHE’s National Education Technology Strategic Plan - MEHE’s Education Sector Plan - USAID’s CDCS & policy 86 3 Please describe how was the ICT action plan developed? Describe the process? a. What contributions did TOT make toward the development of the ICT action plan? - Collaborative process - TOT specific role, if any. - MEHE’s role EFFECTIVENESS: 4 Please describe DOT’s contribution toward developing the ICT coaching strategy/plan? 5 In what ways has the provision of ICT coaching contributed toward the project results a. What steps did TOT take to ensure MEHE’s support/ownership? Examples of achievements in relation to: Component 1 - Develop an ICT action plan Component 2 - Train teachers in the ICT use and maintenance 6 Please tell us about some of the key DR2 outcomes/products that benefited MEHE a. What strategic plans, policies, manuals, curricula developed by DR2 are adopted and used by MEHE. Please give details? 6 In your opinion, how did the different tiers of ICT equipment support influence results? Were the tiers targeted appropriately? Explain? Improved school capacity in ICT use and maintenance 7 To what extent did the planned ICT coaching intervention contributed in achieving project objective # 1? - Improvement in school learning environment 8 In your opinion, what factors played key role in achieving planned results? d. What do you think is the DOT’s biggest success? - Examples of key factors - Support to MEHE’s in providing ICT based education to the public schools 9 In your opinion, what factors supported timely project performance? Examples? - Examples of factors in relation to timely achievement of project results, - Factors that helped in overachievement, underachievement. - Role of DOTs and AUB 10 Did DOT encounter any particular challenges in the development of the ICT coaching material, and in provision of ICT coaching to public schools? Explain? a. Please describe your experience working with public school ICT teachers b. Please describe your experience working with AUB ? - Difficulties in having feedback/approval from MEHE on ICT action plan - Working with CERD—good/poor relationship - Type of support received from AUB 11 What steps did DOT take to ensure participation of women? Explain? a. What role if any DOT play in ensuring women representation in the ICT action plan, and ICT coaching? - ICT Action plan, - Working with AUB to ensure women participation - Working with public schools to ensure women participation 87 b. What about the participation of girl students in classroom ICT use? SUSTAINABILITY 12 To what extent did the project implementation specifically address sustainability and how? a. In your opinion, are the project results related to ICT coaching sustainable? Why? b. What other factors contributed to sustainability of these results? c. Please describe factors that contributed to unsustainability of DR2 results d. Is there anything else the project or future projects should do to ensure sustainability - ICT action plans developed - Post DR2 utility of the ICT coaching - MEHE’s role in sustainability of these results 13 If you were to redo/redesign the project, what improvements would you introduce? Suggestions/recommendations for improving future programming 14 Please tell us about any additional project achievements, or challenges that you may not have mentioned in response to the earlier questions. 88 Focus Group Discussion Protocols Introduction: Please go around and briefly introduce yourselves, name of your school/organization, your role within your school/organization and the kind of training/support (name and duration) you received. FGD Group: _______________________________ Total Participants: Male ____Female ____ Location (facility & city/caza): __________________________ Date _______________ Moderator: _______________________ Note-taker: _______________________ Focus Groups with public school teachers who received trainings [ICT, English Language Program (ELP), English Language Teaching Methodology (ELTM), Extra Curricular Activities (ECA), and Psychosocial Support (PSS)] 1. How long have you been teaching in public schools? Which subjects? What levels? 2. How many training have you attended? Which subjects? How long? When? 3. Please briefly describe how relevant the DR2 training was to your work? 4. Please briefly describe how familiar you were with training topic (ELP, ELTM, ICT, ECA, PSS) before you took this training? 5. Please describe how the training increased your familiarity with/knowledge of (the training topic, e.g., ECA or ICT) following this training? 6. What did you gain as a result of your participation in this program/activity, and how it improved your work/teaching? 7. Can you give examples how you are applying the training in the classroom and how your students are benefiting from the improvement in your work/teaching? 8. Did the training address equally the needs of female and male students? How? 9. What are the key challenges in applying what you learned in the classroom? 10. How do you anticipate using the knowledge/skills that you gained as a result of your participation in the training in the future? Please describe. 11. What aspect of this training did you find to be most valuable? Least valuable? 12. What suggestions do you have for improving this program/activity in the future? 13. In your own words, please describe ways in which the DRASART II training intervention has affected your school/organization. Focus Groups with public school principals who received LDP training, and ICT Equipment and school representatives who participated in SIP training. LDP - Principals 1. How long you have been a principle? Which Schools? Which Areas? 2. Tell us about the LDP training you attended and what it entailed? How long? When? 3. Please briefly describe how relevant the training was to your work? 4. Please briefly describe how familiar you were with topics covered in the training before you took this training? 5. Please describe how the training increased your familiarity with/knowledge of those topics following this training? 6. What did you gain as a result of your participation in LDP and how it improved your work/school management? 7. Can you give examples how you are applying the training in the school and how your teacher and students are benefiting from the improvement in your role as principals? 8. Did the training address the needs of female and male teachers? How? 9. What are the key challenges in applying what you learned in your school? 10. How do you anticipate using the knowledge/skills that you gained as a result of your participation in the training in the future? Please describe. 11. What aspect of this training did you find to be most valuable? Least valuable? 89 12. What suggestions do you have for improving this program/activity in the future? 13. In your own words, please describe ways in which the DRASART II training intervention has affected your organization. ICT Principals 1. How long you have been a principle? Which Schools? Which Areas? 2. Tell us about the ICT equipment you received from DR2 project—what equipment did you receive? When? 3. Please briefly describe how relevant the ICT equipment was to the needs of your school? 4. Please briefly describe how ICT was used in your school before you received the equipment and how the ICT use changed as a result of having the ICT equipment? 5. What did your gain overall as a result of your receiving the ICT equipment and how it improved you school? 6. Can you give examples how the ICT equipment is used in your school? How your teachers and students are benefiting from the equipment? 7. Do male and female children have equal access to the equipment? How? 8. What are the key challenges in using the ICT equipment in your school? (Maintenance?) 9. How do you anticipate using the equipment in the future in your school? Please describe. 10. What aspects of this training did you find to be most valuable? Least valuable? 11. What suggestions do you have for improving this program/activity in the future? 12. In your own words, please describe ways in which the DRASART II training intervention has affected your school/organization. SIP Trainees/Participants. 1. How long you have been a principle/teacher/administrator? Which Schools? Which Areas? 2. Tell us about the SIP training you attended and what it entailed? How long? When? 3. Please briefly describe how familiar you were with topics covered in the training before you took this training? 4. Please describe how the training increased your familiarity with/knowledge of those topics following this training? 5. What did you gain as a result of your participation in SIP training? 6. Please tell us how you developed and implemented an improvement plan for your school? 7. Please describe way in which the training was instrumental in developing and implementing SIP in your school? 8. Did you include parents and community members in the SIP development and implementation? Why not? 9. Can you give examples areas of school improvement your SIP focused on? 10. Please give examples of improvements in your school as result of the implementation of the SIP. 11. Please describe ways in which your teachers and students are benefiting from the SIP implementation? 12. When developing and implementing SIP, how you ensured the needs of your male and female teachers and male and female students? Examples? 13. What are the key challenges in developing and implementing SIP? 14. How do you anticipate using the knowledge and experience that you gained from participating in SIP activities in the future? Please describe. 15. What aspects of this training did you find to be most valuable? Least valuable? 16. What suggestions do you have for improving this program/activity in the future? 17. In your own words, please describe ways in which the DRASART II training intervention has affected your school/organization. ICT Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. What subject matters did you train teachers /principles on? 3. How many trainings did you conduct? How long? When? Where? 4. Please tell how relevant was the training to the needs of the teachers/principals? 5. Please tell us ways in which the training improved teachers’ and principals’ skill/knowledge of ICT use? 6. Were the teachers and principals able to apply what they learned from the trainings in their work? Please give examples? 7. What challenges did they face using the knowledge and skills they gained from the training in their work? 8. What challenges did you face in delivering the ICT training? 9. How do you anticipate the teachers and principals using the knowledge/skills that they gained from the training in the future? Please describe. 10. In your opinion what aspects of the trainings were most and least valuable to the teachers/principals? 11. What suggestions do you have for improving the program/training in the future? 90 English Language Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. What subject matters you trained teachers on (focus of your training)? 3. How many trainings you conducted? How long? When? Where? 4. Please tell how relevant was the training to the needs of the teachers? 5. Please tell us ways in which the training improved teachers’ English language skills? 6. Were the teachers able to apply what they learned from the trainings in their work/classrooms? Please give examples? 7. What challenges did they face using the knowledge and skills they gained from the training? 8. What challenges did you face in delivering the English training? 9. How do you anticipate the teachers using the knowledge/skills that they gained from the training in the future? Please describe. 10. In your opinion what aspects of the trainings were most and least valuable to the teachers? 11. What suggestions do you have for improving the program/training in the future? ECA Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. What subject matters you trained teachers on (focus of your training)? 3. How many trainings you conducted? How long? When? Where? 4. Please tell how relevant was the ECA training to the needs of the teachers and schools? 5. Please tell us ways in which the training improved how teachers worked at their school? 6. Were the teachers able to apply what they learned from the ECA trainings in their work/classrooms? Please give examples? 7. What challenges did they face using the knowledge and skills they gained from the training? 8. What challenges did you face in delivering the training? 9. How do you anticipate the teachers using the knowledge/skills that they gained from the ECA training in the future? Please describe. 10. In your opinion what aspects of the trainings were most and least valuable to the teachers? 11. What suggestions do you have for improving the program/training in the future? SIP Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. What subject matters did you train the principals and teachers on (focus of your training)? 3. How many trainings you conducted? How long? When? Where? 4. Please tell how relevant was the SIP training to the needs of the teachers, principals and schools? 5. Please tell us ways in which the training helped teachers and principals think of school needs and addressing those needs? 6. Were the SIP training participants able to apply what they learned from the SIP trainings in their schools? Please give examples? 7. What challenges did they face using the knowledge and skills they gained from the training in the schools? 8. What challenges did you face in delivering the training? 9. How do you anticipate the principals and teachers using the knowledge/skills that they gained from the SIP training in the future? Please describe. 10. In your opinion what aspects of the trainings were most and least valuable to the teachers? 11. What suggestions do you have for improving the program/training in the future? LDP Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. What subject matters did you train principals on (focus of your training)? 3. How many trainings you conducted? How long? When? Where? 4. Please tell how relevant was the LDP training to the needs of the principals and schools? 5. Please tell us ways in which the training helped principals think of school needs and addressing those needs? 6. Were the LDP training participants able to apply what they learned from the SIP trainings in their schools? Please give examples? 7. What challenges did they face using the knowledge and skills they gained from the training in the schools? 8. What challenges did you face in delivering the training? 9. How do you anticipate the principals using the knowledge/skills that they gained from the LDP training in the future? Please describe. 91 10. In your opinion what aspects of the trainings were most and least valuable to the teachers? 11. What suggestions do you have for improving the program/training in the future? ELTM Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. What subject matters you trained teachers on (focus of your training)? 3. How many trainings you conducted? How long? When? Where? 4. Please tell how relevant was the training to the needs of the teachers? 5. Please tell us ways in which the training improved teachers’ English language teaching methods? 6. Were the teachers able to apply what they learned from the trainings in their work/classrooms? Please give examples? 7. What challenges did they face using the knowledge and skills they gained from the training? 8. What challenges did you face in delivering the ELTM training? 9. How do you anticipate the teachers using the knowledge/skills that they gained from the training in the future? Please describe. 10. In your opinion what aspects of the trainings were most and least valuable to the teachers? 11. What suggestions do you have for improving the program/training in the future? Coaches 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. What subject matters did you train teachers on? 3. How did you provide coaching support to the teachers? How long? When? Where? 4. Please tell how relevant was the training to the needs of the teachers schools? 5. Please tell us ways in which the training improved teachers’ skill/knowledge of ICT use? 6. Were the teachers able to apply what they learned from the trainings in their work? Please give examples? 7. What challenges did they face using the knowledge and skills they gained from the training in their work? 8. What challenges did you face in delivering the ICT training? 9. How do you anticipate the teachers using the knowledge/skills that they gained from the training in the future? Please describe. 10. In your opinion what aspects of the trainings were most and least valuable to the teachers/principals? 11. What suggestions do you have for improving the program/training in the future? CERD ICT Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. Please describe the ICT training you received under DR2 and what it focused on? 3. Please describe how familiar you were with the ICT topics covered in the training before you took the training? 4. Please describe how the DR2 training increased your knowledge of those topics and improved you work? 5. How many ICT teacher trainings did you conduct? How long? When? Where? 6. Please tell us how relevant was the training to the needs of the teachers/schools? 7. Please tell us ways in which the training improved teachers’ knowledge of ICT use in classroom? 8. What challenges did they face using the knowledge and skills they gained from the training in their work? 9. What challenges did you face in delivering the ICT training? 10. How do you anticipate the teachers using the knowledge/skills that they gained from the training in the future? Please describe. 11. In your opinion what aspects of the trainings were most and least valuable to the teachers? 12. What suggestions do you have for improving the program/training in the future? CERD English Language Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. Please describe the English language trainings you received under DR2 and what it focused on? 3. Please describe how familiar you were with the topics covered in the PCELT and ELTM training before you took the trainings? 4. Please describe how these trainings increased your knowledge of those topics and improved you work? 5. How many English teacher trainings did you conduct? How long? When? Where? 6. Please tell us how relevant was the training to the needs of the teachers/schools? 7. Please tell us ways in which the training improved how English teachers taught English language in classroom? 8. What challenges did they face using the knowledge and skills they gained from the training in their work? 9. What challenges did you face in delivering these training? 92 10. How do you anticipate the teachers using the knowledge/skills that they gained from the training in the future? Please describe. 11. In your opinion what aspects of the trainings were most and least valuable to the teachers? 12. What suggestions do you have for improving the program/training in the future? CERD PSS Trainers 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. Please describe the PSS training you received under DR2 and what it focused on? 3. Please describe how familiar you were with the topics covered in the PSS training before you took the trainings? 4. Please describe how these trainings increased your knowledge of those topics and improved you work? 5. How many PSS trainings did you conduct? How long? When? Where? 6. Please tell us how relevant was the training to the needs of the teachers/schools? 7. Please tell us ways in which the training improved how teachers provided psychosocial support to students? 8. What challenges did they face using the knowledge and skills they gained from the training in their work? 9. What challenges did you face in delivering these trainings? 10. How do you anticipate the teachers using the knowledge/skills that they gained from the training in the future? Please describe. 11. In your opinion what aspects of the trainings were most and least valuable to the teachers? 12. What suggestions do you have for improving the program/training in the future? DOPs ICT Trainers/Coaches 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. Please describe the ICT training you received under DR2 and what it focused on? 3. Please describe how familiar you were with the ICT topics covered in the training before you took the training? 4. Please describe how the DR2 training increased your knowledge of those topics and improved you work? 5. How often do you provide ICT support to teachers/public schools? 6. Could you describe the nature of ICT assistance teachers/schools request from you? Examples? 7. In your opinion how often you have been able to successfully provide ICT assistance to teachers/schools? 8. What challenges did teachers face using the knowledge and skills they gained from ICT training in their work? 9. What challenges did you face in providing the ICT support to schools? 10. How do you anticipate providing ICT support to teachers/schools in the future? Please describe. 11. In your opinion what aspects of the trainings were most and least valuable to you? 12. What suggestions do you have for improving the program/training in the future? DOPS PSS Trainers/Coaches 1. How did you get involved in the DRASATI II project and what was the scope of your participation in the project? 2. Please describe the PSS training you received under DR2 and what it focused on? 3. Please describe how familiar you were with the PSS topics covered in the training before you took the training? 4. Please describe how the DR2 training increased your knowledge of those topics and improved you work? 5. How often do you provide PSS support to teachers/public schools? 6. Could you describe the nature of PSS assistance teachers/schools request from you? Examples? 7. In your opinion how often you have been able to successfully provide PSS assistance to teachers/schools? 8. What challenges did teachers face using the knowledge and skills they gained from PSS training in their work? 9. What challenges did you face in providing the PSS support to schools? 10. How do you anticipate providing PSS support to teachers/schools in the future? Please describe. 11. In your opinion what aspects of the trainings were most and least valuable to you? 12. What suggestions do you have for improving the program/training in the future? 93 ANNEX 4: DRASATI II INTENDED RESULTS/SCOPE OF WORK 94 For more information, please visit http://www.socialimpact.com Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA 22201 Suite 1000 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socialimpact.com