This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) Prepared under Contract No.: GC-10F-0033M/AID-OAA-M-13-00010 - PN 7617.027.01 USAID READING AND ACCESS (R&A) EVALUATIONS EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) (THIS VERSION: DECEMBER 17, 2021) Prepared under Contract No.: GC-10F-0033M/AID-OAA-M-13-00010 - PN 7617.027.01 Submitted to: Benjamin Sylla, USAID COR Submitted by: Alicia Menendez, Alejandro Ome, Cathy Zimmerman, Jessica Wallach, Ninar Taha, Mithila Iyer and Newton Toe Contractor: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Varuni Dayaratna DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | i TABLE OF CONTENTS EXECTUTIVE SUMMARY VI BACKGROUD VI EVALUATION METHODS VI RESULTS VII IMPLICATIONS XI SUMMARY RECOMMENDATIONS XII INTRODUCTION 1 BACKGROUND 1 1.1 THE AQE MODEL 1 1.2 TRANSITION TO THE ONE SCHOOL MODEL 3 RESEARCH OBJECTIVES AND QUESTIONS 5 METHODS 5 1.1 RESEARCH DESIGN 5 1.2 DATA SOURCES AND DATA COLLECTION: 6 QUALITATIVE DATA SOURCES AND COLLECTION 6 QUANTITATIVE DATA SOURCES AND COLLECTION 7 ETHICS AND COVID-19 PRECAUTIONS 8 QUALITATIVE SAMPLE 8 QUANTITATIVE SAMPLE 9 1.3 DATA ANALYSIS: 10 QUALITATIVE DATA ANALYSIS 10 QUANTITATIVE DATA ANALYSIS 10 1.4 LIMITATIONS 11 FINDINGS 11 2.1. WHAT ARE THE AQE ELEMENTS THAT HAVE BEEN TRANSITIONED TO AND SUSTAINED BY THE MOE OF LIBERIA? 11 2.2. WHAT ARE THE AQE ELEMENTS THAT WERE NOT TRANSITIONED? WHAT ARE THE IMPLICATIONS FOR PROGRAM SUCCESS THAT SOME ELEMENTS OF THE AQE WILL NOT BE ADOPTED BY GOL, IN PARTICULAR RELATED TO FACILITATORS’ COMPENSATION AND WORKLOAD? 11 ALP CENTER CLOSURES AND OPERATIONS 12 ONE SCHOOL APPROACH MODEL 14 2.2.B. MINIMUM PACKAGE 17 MONITORING AND SUPERVISION 17 FACILITATOR RECRUITMENT, TRAINING, AND COMPENSATION 19 TEACHER BURNOUT 24 RECORD KEEPING 29 ASSESSMENT OF LEARNER PROGRESS AND LEARNING CERTIFICATIONS 30 TEACHING AND LEARNING MATERIALS 30 2.3. AQE’S STRATEGIES TO ENSURE SUSTAINABILITY AND INSTITUTIONALIZATION INCLUDING CURRICULUM, AQE MODELS, AND TEACHER TRAINING APPROACHES.32 ii | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG SUCCESS FACTORS FOR MINISTRY OF EDUCATION CAPACITY 32 CHALLENGES FOR MINISTRY OF EDUCATION CAPACITY 33 SCALABILITY, REPLICABILITY, AND THE FUTURE OF THE ALP PROGRAM 34 2.4. WHAT IS THE CONTRIBUTION OF AQE TO EQUITABLE TRANSITION TO FORMAL EDUCATION, RETENTION, AND COMPLETION IN LIBERIA? HOW DID THE AGE, GENDER, LOCATION, OF A LEARNER AFFECT THEIR TRAJECTORY? 35 ADMINISTRATIVE DATA – AQE MODEL 35 FOLLOW-UP SURVEY – ONE SCHOOL MODEL 40 MARGINALIZED STUDENTS AND EQUITY CONSIDERATIONS 42 CHANGES IN COSTS OF ENROLLMENT IN ALP 43 2.5. WHAT STRATEGIES FOR AQE DELIVERY COULD BE USED TO SERVE LEARNERS AFFECTED BY AN EMERGENCY LIKE COVID? 44 2.6. WHAT ARE UNIT COSTS FOR AQE PROGRAM PARTICIPANT ACCESS OR COMPLETION? 48 2.7 WHAT WOULD BE THE COST TO THE MOE TO CONTINUE TO SCALE UP THE AQE MODEL? 52 CONCLUSIONS AND RECOMMENDATIONS 54 SUSTAINABILITY 54 EQUITY 56 COSTS 57 RECOMMENDATIONS 58 REFERENCES 62 ANNEXES 63 ANNEX I: EVALUATION DESIGN FRAMEWORK 63 ANNEX II: QUALITATIVE INSTRUMENTS 64 ANNEX III: QUANTITATIVE INSTRUMENTS 85 ANNEX IV: FACE-TO-FACE (F2F) SURVEY COVID – 19 RISK MITIGATION STRATEGY 122 ANNEX V: USING ADMINISTRATIVE DATA TO ANALYZE SURVIVAL IN THE AQE PROGRAM 124 ANNEX VI: SAMPLING WEIGHTS OR THE FOLLOW-UP SURVEY 126 ANNEX VII COST SIMULATION UNDER MINISTRY OF EDUCATION ALP IMPLEMENTATION 128 ANNEX VIII: CONFLICT OF INTEREST (COI) FORMS 131 USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | iii TABLES Table 1: Number of schools surveyed and selected. 3 Table 2: ALP-CS integration models 4 Table 3: Qualitative Data Collected per Respondent Category: 8 Table 4: Quantitative Data Collected per Respondent Category 10 Table 5. Surveyed of schools by whether they offer conventional, ALP or both 12 Table 6: Percentage of schools where ALP classes are pooled with classes from the conventional system 13 Table 7: Mean and median student age by grade/level and system for schools with both systems (n=47) 14 Table 8: Monitoring and supervision in schools with ALP centers (n=73) 17 Table 9: Descriptive statistics by teacher type 20 Table 10. Recruitment process for ALP facilitators (%) 20 Table 11. ALP Facilitator Opinions on ALP Trainings (n=120) 21 Table 12. Teachers who were observed during lessons in this academic year (n=128) 22 Table 13. Percentage of teachers/facilitators that are compensated 23 Table 14. Hours per week worked by type of teacher/facilitator 25 Table 15. Maslach Burnout Inventory (MBI) scores classification 25 Table 16. Attendance and lesson planning 29 Table 17. AQE students, by years of enrollment 36 Table 18. Students’ characteristics 36 Table 19. Completion, passing assessments, and retention for AQE students 37 Table 20. Correlations between students’ characteristics and completion, passing and retention rates 38 Table 21. Surveyed students by level in 2019-2020 40 Table 22. Surveyed students’ descriptive statistics by level 40 Table 23. Enrollment status in 2020-2021 by ALP level in 2019-2020 41 Table 24. Determinants of school enrollment 42 Table 25. Teachers’ strategies to ensure learning continuation 46 Table 26. AQE Full Costs Categories 51 Table 27. AQE Implementation Costs Categories 52 Table 28. ALP center simulation – Number of students and facilitators 52 Table 29. Number of students in completion data not in intake data 124 iv | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG FIGURES Figure 1. Evaluation framework for the Transition Process of the AQE-ALP from EDC to MoE. ...............6 Figure 2. Reasons for shutting down the ALP program (N=16)..........................................................................12 Figure 3: Fraction of teachers that receive ALP training by year.........................................................................21 Figure 4: Frequency of teacher observations (n=115)............................................................................................22 Figure 5: Maslach Scale - Levels of Emotional Exhaustion Amongst Teachers.................................................26 Figure 6: Maslach Scale - Levels of Depersonalization Amongst Teachers........................................................26 Figure 7: Maslach Scale - Levels of Personal Accomplishment Amongst Teachers.........................................27 Figure 8: Frequency of teacher utilization of guides (n=125)................................................................................29 Figure 9: Assessment prevalence according to teachers (%) (N=73) .................................................................30 Figure 10. Percentage of teachers who report that students have their own workbook .............................31 Figure 12. Students who returned to school that reopened after COVID-19.................................................48 Figure 13. Distribution of AQE costs by category (2017-2021) ..........................................................................49 USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | v ACRONYMS AE Accelerated Education ALP Accelerated Learning Program ALP-CSP Alternative Learning Program-Conventional School Pilot AQE Accelerated Quality Education CEO County Education Officer CS Conventional School DEO District Education Officer EDC Education Development Center GoL Government of Liberia MoE Ministry of Education PTA Parent Teacher Association TKG The Khana Group USAID United States Agency for International Development vi | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG EXECTUTIVE SUMMARY BACKGROUD The Accelerated Quality Education for Liberian Children Activity (AQE) was a four-year (March 2017– August 2021) activity established by USAID/Liberia and the Education Development Center (EDC) to increase access to education for out-of-school children and youth in six counties across Liberia - Montserrado, Margibi, Bong, Grand Bassa, Nimba and Lofa. As part of the USAID Reading and Access Evaluation Contract, NORC at the University of Chicago conducted an external evaluation of the transition of the activity from the EDC management to the Ministry of Education (MoE), focusing on the sustainability of the model, equity considerations, and the cost of the original program and the costs to the MoE if it were to be brought to scale. EVALUATION METHODS This evaluation utilized a mixed-methods approach to evaluate the transition from the AQE model to a “One School Model,” managed by the Government of Liberia Ministry of Education which combined both conventional school and ALP during the morning shift. The evaluation focused substantially on the changes in the program design and how this affected the teaching and learning. We also analyzed how the AQE model affected equity in the Liberian alternative education system. We examined the proportion of students transitioning to conventional education, retention, and completion, and explored whether the AQE strategies could be used to support student learning in crises, such as the COVID-19 pandemic. Importantly, we also conducted a cost analysis of AQE and estimated the fiscal cost of scaling ALP-CS. The evaluation questions cover program sustainability, equity, and cost: SUSTAINABILITY 1. What are the AQE elements that have been transitioned to and sustained by the MoE of Liberia? 2. What are the AQE elements that were not transitioned? What are the implications for program success when some elements of the AQE are not adopted by GoL, in particular, related to facilitators’ compensations and current workload? 3. How successful were AQE’s strategies to ensure sustainability and institutionalization related to the curriculum, AQE models, and teacher training approaches? EQUITY 4. What is the contribution of AQE to equitable transition to formal education, retention, and completion in Liberia? How did the age, gender, or location of a learner affect their trajectory? 5. What strategies for AQE delivery could be used to serve learners affected by an emergency like COVID? COST 6. What are unit costs for AQE program participant access or completion? 7. What would be the cost to the MoE to continue to scale up the AQE model? To assess the implications of the transition from EDC to the Ministry of Education and adaptation of ALP to the ‘One School’ model, we triangulated findings from the quantitative and qualitative study components and different stakeholder perceptions. We examined the implementation of the transition USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | vii and current outcomes of the revised model. For the quantitative component we visited 90 schools and for the qualitative component, we conducted interviews and focus group discussions in 12 schools. Interviews with high-level key informants were conducted with various ministry of education officials at the national and county levels, USAID and EDC staff. The mixed methods included the following data collection and analysis components: QUANTITATIVE - Administrative data on school enrollment - School observation surveys with the guide of principals - Teachers and facilitators survey - Follow-up survey of AQE graduates (with parents) QUALITATIVE - Key Informant Interviews (KIIs) - Focus Group Discussions (FGDs) - Semi-Structured Interviews (SSIs) - Document review COST ANALYSIS - Secondary analysis of AQE reports from USAID/Liberia and the EDC costs from March 2017 to March 2021. RESULTS SUSTAINABILITY 1. What are the AQE elements that have been transitioned to and sustained by the MOE of Liberia? 2. What are the AQE elements that were not transitioned? What are the implications for program success that some AQE elements will not be adopted by GOL, in particular related to facilitators’ compensation and workload? Results indicate that out of 90 AQE surveyed schools, 73 (81 percent) have active ALP centers. Even though most schools sustained the core content of the ALP program, there remained concerns for the sustainability of the program long-term, particularly related to potential teacher burnout and retention, availability of teaching and learning resources (e.g., students’ workbooks), infrastructure (e.g., sufficient classroom seating), and continued teacher training. Although 95 percent of principals said they received support from the District Education Officers (DEO) or County Education Officers (CEO) who were tasked with monitoring the ALP center, many of the interviewed officers reported logistical challenges (e.g., transport) that prevented them from fully completing their assignments. Before transitioning to the MOE facilitators received a stipend for teaching the separate, afternoon ALP sessions. Once the AQE stipends ceased, some facilitators who were not in the government payroll viii | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG were unable to continue teaching. Additionally, 37 percent of ALP facilitators reported they were working more than 25 hours per week, most of whom were teachers who taught both conventional classes and ALP. The “One School Approach” was intended to have ALP and conventional students attend the same school shift during the morning, and the separate ALP afternoon sessions were terminated. Reports suggested some benefits to this combined classroom approach. For example, some parents explained that students were more likely to come to school if they left home in the mornings at the same time as their caregivers versus afternoon sessions when no one was home to urge the youth to go to school. However, the combined model also appeared to strain school resources, especially schools with limited teachers, classroom space and seating to run conventional school and ALP simultaneously. We found that, among schools that have both ALP and conventional schooling, 19.1 percent of Level 1, 17.1 percent of Level 2 and 25.8 percent of Level 3 classes share the classroom with conventional students. Teachers working in classrooms that include ALP and conventional students, most frequently used a combination of conventional and ALP materials. While this is a clear divergence from the original model, it was also a testament to the quality of the ALP materials that are now being used for both conventional and ALP teaching, demonstrating how well they are aligned with the conventional curriculum. Teachers like the ALP approach and materials. However, at the same time, teachers reported that it was very challenging to teach the combined class because of the shortage of workbooks and learning materials and the larger number of students in the classroom. Regarding teacher training, we found that only 8 percent of ALP facilitators had not received training. Most of the trained facilitators were trained while EDC was implementing the program and do not need further training. It is too early to assess whether this program component survived the transition. Lastly, while not a consequence of the transition, perhaps the greatest challenge for sustainability of the ALP is the fact that schools are not using the ALP to separate children by age. In effect, in schools that run both the conventional system and the ALP, there are almost as many overage children in the conventional system as they are in ALP. The ALP can only be sustainable if the conventional system serves on-age children so that ALP can focus on overage children. As they are currently functioning, the conventional system and ALP are basically two parallel systems serving the same children, in which both groups may be disadvantaged. 3. How successful were AQE’s strategies to ensure sustainability and institutionalization related to the curriculum, AQE models, and teacher training approaches? Overall, findings indicate that the ALP model has been well-received. In particular, the quality of the curriculum was praised by focus group and interview participants. Most programmatic problems were associated with funding constraints, availability of teaching and learning materials, and merging the classes, not with the quality of the original AQE model itself. Comments from study participants suggest that they are keen to find ways to sustain and scale up the model, especially given the negative impact of the COVID-19 pandemic on student enrollment and potential implications for possible future school closure. In terms of the efficacy of strategies to ensure sustainability and institutionalization, constraints on resources from the MoE severely undermined their success. Shortages of teaching and learning materials is one of the key challenges that schools are facing. When asked about the impact of the transition of USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | ix the program to MoE oversight, the single most frequent complaint from respondents in KIIs and FGDs was insufficient materials for ALP teachers and learners. In effect, 25 percent of teachers said that no student in their classes has a workbook, and 14.5 percent said that fewer than half of the students in the class do. To facilitate teacher training continuity, AQE staff worked with MoE to prepare county and school level staff as master trainers, tasked to conduct ALP facilitator training and school-based monitoring and supervision moving forward. It was also notable that some principals of schools with closed ALP centers stated that they were not informed of or given guidance to undertake the transition. EQUITY 4. What is the contribution of AQE to equitable transition to formal education, retention, and completion in Liberia? How did the age, gender, or location of a learner affect their trajectory? Administrative data starting from 2017 indicated an upward trajectory in student retention and passing rates over time. NORC estimates that between 2019-20 and 2020-21, 22 percent of ALP students transitioned to conventional school and 43 percent stayed in ALP, for a total retention rate of 65 percent. There were no major differences by gender for any of the outcomes, including school retention. Administrative and survey data indicated that the youngest students (6-9) are more likely to drop out of school than older students, especially when we compare students ages 6-9 with students 10-12. It is possible that this difference may be attributed to the weak focus on younger students’ education, more difficulty tolerating classroom overcrowding, or longer school days under the One School Approach. Students in urban schools were less likely to stay in school, possibly due to the opportunity cost of staying in school versus pursuing job opportunities in urban areas, compared to rural areas. It is also important to note that shortages of chairs present an equity problem, because they exclude students living with a disability, who might not be able to sit on the ground or stand for long hours during the school day. 5. What strategies for AQE delivery could be used to serve learners affected by an emergency like COVID? The Ministry of Education has launched programming that disseminates lessons over radio -based on AQE materials- and is planning to expand further into radio and television. However, it was clear that this approach excludes a significant number of students who live in communities without consistent electricity. The quality of the ALP materials and usefulness of the students’ workbooks were important for continued learning during school closures because of the COVID-19 pandemic. The data showed that some students did not have their own workbooks, which prevented them from progressing academically when schools were closed. Meanwhile, those who had workbooks were able to continue learning. x | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG The data indicate that teachers think a program like AQE could help students recover lost learning. Almost all teachers like the AQE curriculum and think that is a good alternative to make up for COVID￾related learning loss, as it is for them simpler and shorter to prepare and to teach, and for the students simpler to grasp. Having a condensed or compressed curriculum ready is a great resource and we should take advantage of it. However, using accelerated education to bring the students up to speed is not straight forward and adaptations and clear guidance to teachers should be considered when using it as catch-up program to help students recover the losses in knowledge and skills. COSTS 6. What are unit costs for AQE program participant access or completion? NORC computed the costs incurred by the AQE program and funded by USAID. These costs include all the expenses, and all the (monetized) contributions received by the program as implemented by EDC. The total AQE cost per student enrolled at least once in the program was 137 USD for implementation, 160 USD if it also includes the costs of development, and $283 when it includes management and operations costs. This refers to the maximum enrollments seen in the program, where the same student could have been enrolled in the program for more than one year and therefore shows the total cost of AQE as implemented by EDC per enrolled student-year. The total average costs were slightly higher when we only consider unique students that accessed the AQE. For example, the total implementation costs were 157 USD for each (unique) student that participated in the program. Finally, NORC computed the average AQE costs for each year approved by the students. One student could have approved, one, two or three levels of the program. In this case, the total cost of implementation was 305 USD per student per year approved, the costs of implementation and development were 356 USD, and the full costs including management and operations expenses reached 630 USD per student per ALP level approved. Different cost categories are included in the AQE costs per student computed above. Parental engagement was the cost component with the largest weight in the total AQE costs. Almost of quarter of the AQE costs were devoted to involving communities to support the ALP. Cost of training teachers were 23.9 percent of the total costs, followed by teaching and learning materials to which AQE allocated 22.6 percent of the full costs. Similar amounts were spent on teacher stipends and on school infrastructure and furniture, 13.9 and 13.8 percent respectively. Learning assessments are less than one percent of the full costs. 7. What would be the cost to the MoE to continue to scale up the AQE model? NORC estimated the MoE costs of running an ALP center for ten years under different scenarios. The simulated ALP center initially enrolls 150 students and 35 students every year for the following 9 years, reaching a total of 465 unique students and 830 estimated student-years at the center. Running such a center that initially has three trained facilitators for ten years costs 25,403 USD in total or 30.6 USD per student-year. These costs include all the teaching and learning materials needed –the main cost driver-, and training for a replacement facilitator. If facilitators were not available and training had to be provided for three people in the first year, the ten-years cost would slightly increase to $26,273 or $31.7 per student-year. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | xi Under the above scenario, ALP facilitators do not receive any extra compensation. In some instances, the absence of additional compensation for ALP facilitators seems to reflect a more efficient use of the teaching staff time in the MoE payroll. However, the change was not well-received by facilitators, and had important implications for the quality of the program. Volunteers who were not on the government payroll left and the dedication of those who continued teaching seemed to have suffered. The student￾teacher ratios have increased, and now conventional and ALP systems share teachers and classrooms in many schools, which affects the type and quality of education. Therefore, in the last scenario we simulated, we included training for three facilitators and assumed that they are paid stipends. This reconfiguration clearly increases the costs of running an ALP center, but it allows schools to use the facilities during the morning and afternoons, doubling the capacity while using the existing infrastructure, enabling the implementation of the original conceptualization of the program. If each facilitator is paid $50 per month -as EDC used to pay- during 10 months per year, the cost of the ALP center for ten years would increase to $41,798 or $50.4 per student-year which seems to be in line with the GoL budget per student. IMPLICATIONS Our findings indicate that the transition to MoE management generated concerns about the long-term sustainability of the ALP and potential student learning outcomes. There are clear differences between the resource availability when the program was under EDC management compared to MoE management. The loss of external funding and change in ALP classroom format in some cases, seems to strain teachers’ ability to attend to the learning needs of both the conventional and ALP students simultaneously. Not surprisingly, the loss of the stipend for teaching staff caused losses of facilitators and created some frustration among the remaining teachers, increasing teachers’ workload and potentially reducing the teaching capacity for both conventional and ALP students in some schools. Especially the inadequate numbers of workbooks, increased classroom size and insufficient seating for each student, are likely to negatively affect ALP and conventional students. However, at the same time, there are core elements to the design and content of the ALP learning materials that students and teachers deemed successful and should therefore be prioritized when considering the long-term implementation of the program. The ALP materials were well-received by all stakeholders and appear to be extremely beneficial to support lost learning strategies in the event of emergencies like COVID-19. In the future, the MoE will be responsible for training new facilitators and will have to consider teacher satisfaction and retention if the program proceeds without any type of additional compensation. Student-teacher ratios were impacted by teacher attendance issues that resulted from the lack of compensation for volunteer teachers. Our results on teacher burnout due to workload and lack of compensation are priority areas of improvement to ensure that all students can equally benefit from the ALP. Though concerns over student retention were minimal, teacher retention concerns were raised by multiple stakeholders, and ultimately will have an impact on the quality of education under the ALP over time. The various program configurations assessed in the cost analysis of scaling the program reflect these priority considerations. The most expensive simulated scenario, which includes teaching and learning materials, student assessments, ALP teacher training, and teacher stipends of $50 monthly for ten months per year, costs an average $50.4 per students enrolled per year which seems to be in line with the GoL budget per student. xii | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG SUMMARY RECOMMENDATIONS Based on findings from this evaluation, NORC offers the following recommendations for the Ministry of Education: 1. Age-related placement for students: Place students in ALP or conventional school according to their age. Currently there are many over-aged students in conventional school which compromises the very purpose of having an ALP 2. ALP and conventional school teaching structure and timing: In schools where the infrastructure or the number of teachers is insufficient to implement separate conventional school programs and ALP, work with principals and teachers to explore options to separate the sessions, such as conventional school in the mornings and ALP in the afternoons. 3. Classroom teaching support: If in certain locations, it is necessary to combine ALP and conventional students, explore options for support facilitators working alongside the main teacher. 4. Classroom seating. In classrooms that do not have sufficient chairs, consider budget revisions to ensure students have a classroom environment conducive to learning. 5. Printed workbooks. Prioritize budget allocation for student workbooks to ensure each student has their own workbook. 6. Transition support: Coordinate closely with school principals and staff and provide clear guidance and support to help schools transition to a new ALP model. 7. Consider less expensive designs of workbooks: Explore the possibility of reducing costs by having student materials divided in a workbook component that can only be used once and a “textbook” component that can last several years and be reused by different students. 8. Teacher support and compensation: Consult teachers and principals to develop a strategy to address teacher workload, potential burn-out and retention. 9. Remote monitoring. To maintain oversight of the quality of the program, explore options for remote monitoring methods so officials tasked with overseeing and monitoring ALP classrooms can cut down on transportation costs and travel time. 10. Train new facilitators and principals. Develop processes and resources to ensure new facilitators and principals are well-trained to deliver the ALP learning materials. 11. Radio and TV education platform. Continue expanding radio and television programs to support accelerated learning lessons and supplement this strategy with other context-appropriate education outreach methods to include all students, especially those who live in the most resource￾deprived areas. 12. Feeding program. Consider allocating a portion of the budget to fund a school feeding or nutrition program as an MOE budget priority. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 1 INTRODUCTION NORC at the University of Chicago, through the USAID Reading and Access Evaluation Contract, has been charged with conducting an external evaluation of the Liberia Accelerated Quality Education for Liberian Children Activity (AQE). This evaluation focuses on three areas of interest: 1) Sustainability of the AQE program model –originally implemented by the Education Development Center (EDC)- as it is transferred for further implementation to the Liberian Ministry of Education (MoE), including an assessment of any modifications introduced to the original program; 2) Equity in the provision of education to out of school children and youth; and 3) Costs of the AQE program and estimates of the costs of scaling it up by the Government of Liberia To conduct this evaluation, NORC employed quantitative and qualitative methods. The evaluation team used previously collected information such as AQE program costs records and AQE monitoring and evaluation data and conducted new data collection. BACKGROUND 1.1 THE AQE MODEL Liberia has a large fraction of over-age learners enrolled in primary school1 and a large number of out of school children and youth2. Implemented by Education Development Center, Inc. (EDC), AQE was a $33.9 million, four-year (March 2017–August 2021) activity established to increase access to education for approximately 48,000 eight-to-fifteen-year-old out-of-school children in six Liberian counties (Bong, Grand Bassa, Nimba, Lofa, Montserrado and Margibi). AQE aimed to provide these learners with safe, high-quality learning environments, using a curriculum that consolidates six years of primary school into three levels and, upon graduation, offers a pathway to the formal junior secondary level and/or further training opportunities. The AQE program had the following objectives 3: - Contribute to USAID/Liberia’s Country Development Cooperation Strategy (CDCS) Development Objective 4, “Better-Educated Liberians”; - Meet its targets under Goal 3 of the 2011 USAID Education Strategy, “increased equitable access to education in crisis and conflict environments”, achieving direct outcomes in education access for out-of-school children; - Provide relevant, flexible and quality education opportunities for out-of-school children in Liberia, giving them opportunities to pursue further education, training or employment; - Provide technical assistance and build partnerships with key Ministry of Education (MoE) counterparts and host country systems; particularly to institutionalize a nation-wide legitimate and credible accelerated education program; 1 NORC (2017) USAID/Liberia, Read Liberia Impact Evaluation, Baseline Report. 2 UNESCO, UIS. http://uis.unesco.org/en/country/lr 3 EDC (2019) USAID/Liberia, Accelerated Quality Education for Liberian Children Y3 ANNUAL and Q4 quarterly progress report 2 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG - Strengthen the policies, systems, and resources that are available to the MoE, particularly in the area of accelerated education; - Foster positive gender norms, with a special focus on girls and female teachers, who are least likely to have access to education in Liberia; and, - Promote stability during a key transition in post-war and post-Ebola Liberia through engagement of out-of-school children in activities that advance conflict resolution and key life skills. AQE included the following components: Accelerated Education (AE) Center accreditation – The Liberian MoE is responsible for the registration, accreditation and licensing of all alternative education programs including AQE and sets standards to maintain and regulate the national registries of these programs and their personnel. AE center monitoring and supervision – The following are the Program Quality Standards (PQS) that are used to assess the implementation of alternative education programs such as AQE: • Curriculum Content and Instructional Time. • Teaching and Learning Materials and Instructional Methodologies. • Teaching and Learning Environment. • Assessing Learners’ Progress. • Facilitator Recruitment, Training, and Support. • Learner Recruitment, Enrollment, Retention, and Support. • Community Links and Support to Alternative Education. • Management of Alternative Education Programs. • Monitoring and Evaluation. Standards for AE facilitators4 – • AE facilitators must have a minimum of a C certificate (Primary Teacher’s Certificate); this requirement is updated in some cases to a B certificate (Junior High Teaching Certificates) • AE facilitator training – All facilitators are required to attend a MoE-approved pre￾service training and regular competency based in-service trainings. • AE facilitator remuneration – AE facilitators have all the benefits and recognition that regular teachers do. AE assessment of learner progress and learning certifications – Learners in the AE program are required to complete periodic certifications to progress to the next level of AE AE teaching and learning materials – Materials for the AE program are required to undergo MoE review. This set of components is denominated the Minimum AQE Package. 4 Most facilitators were teachers already working at conventional schools. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 3 EDC followed a predetermined set of criteria to select sites for AQE. First, some factors--such as being a private school or already have an accelerated learning program (ALP) –made sites ineligible. Subsequent criteria restricting eligibility were requirements around the availability of basic infrastructure, presence of female teachers, and students’ progression from primary to secondary education. Additionally, proximity to other priority schools was considered. After a site survey, sites were examined against additional criteria including demonstrated need for ALP, community commitment to schooling, time and space to hold ALP classes, availability of teachers for ALP, and more. Based on these criteria, a total of 130 sites were visited for the first mapping activity in 2017 in Bong, Grand Bassa, and Montserrado counties, from which 100 sites would then be selected for implementation. The second site mapping activity, conducted in July 2018, covered Lofa, Margibi, and Nimba counties. EDC surveyed 138 sites and selected 91 for implementation. The third and final site mapping took place in February 2019 and covered all six existing AQE counties: Bong, Grand Bassa, Lofa, Margibi, Montserrado, and Nimba, with AQE using the same selection criteria as in previous years. Ultimately, 138 sites were surveyed and 80 were selected for implementation. See Table 1, below, for the full sample of schools surveyed and selected. Table 1: Number of schools surveyed and selected. County Surveyed Selected Cohort 1 Montserrado 48 40 Bong 42 30 Grand Bassa 40 30 Total 130 100 Cohort 2 Lofa 43 32 Margibi 26 19 Nimba 69 40 Total 138 91 Cohort 3 Bong 26 21 Grand Bassa 24 6 Lofa 25 13 Margibi 23 9 Montserrado 20 14 Nimba 20 17 Total 138 80 Grand total 406 271 1.2 TRANSITION TO THE ONE SCHOOL MODEL While EDC was implementing the AQE activity, accelerated learning classes were held in schools during the afternoon. Usually, regular schoolteachers acted as ALP facilitators and for this they received a 4 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG stipend (USD 50 per month) as overtime compensation, in addition to their regular pay. 5 In other cases, ALP facilitators were volunteers who also received the same stipend. Paying stipends is beyond the budget plans of the Government of Liberia (GoL), and therefore the implementation of the program under the GoL was different from implementation under EDC. In this new phase, a “One School Model” was implemented, in which teachers/facilitators teach both conventional school and ALP during the morning shift, and no stipend is provided. Teachers were trained to work under both models. Integrating the ALP into the conventional school (CS) system was contingent on both the demand for ALP (number of out of school overaged students) and the availability of school resources, namely classroom space and teacher time.6 In the 2019-2020 school year, the MoE, with EDC support, implemented the Alternative Learning Program-Conventional School Pilot (ALP-CSP) to explore how the integration of AQE into conventional school could work under GoL implementation. This was a pilot program fielded in twelve schools to explore models for this integration. The results of this pilot indicated that there are four potential scenarios for ALP integration. The first scenario is the full ALP-CS provision, in which all levels of ALP and all CS grades would be offered. This requires schools to have enough classrooms (at least 9, one for each grade of CS and 3 for each level of the ALP), enough teacher time and sufficient demand for the ALP. There are also mixed alternatives, in which schools could have only grades 3 to 6 of CS and level 1 of ALP. In this case both on-age and overaged grade 1 and 2 students attend the ALP Level 1 and the rest attend CS (model 2a). Another alternative is to have only grades 5 and 6 of CS and ALP levels 1 and 2. In this case, on-age grade 1 to 4 students and overaged students would attend ALP (model 2b) and the rest would attend CS. The last scenario is a model for cases where overaged students are so prevalent that the school would provide ALP education only. Table 2 summarizes the different ALP-CS integration models. Table 2: ALP-CS integration models CS ALP 1. Full ALP-CS Provision grades 1 to 6 Levels 1 to 3 2a: Partial ALP-CS Provision grades 3 to 6 Level 1 2b: Partial ALP-CS Provision grades 5 to 6 Levels 1 & 2 3: Full ALP Provision None Levels 1 to 3 These different models were shared by EDC with the MoE and have been communicated to the schools. The MoE decided that schools need to assess their demands for ALP and available resources to select the integration model that best fits their situation.7 5 Teachers with a C Certificate receive a monthly wage (after taxes) of 18,000 Liberian dollars (~USD 120), and teachers with a bachelor’s degree in Primary Education receive 51,000 Liberian dollars (~350 USD). Most teachers have a C Certificate of lower levels of education. 6 The criterion to assess teacher time availability is that teachers should be teaching 25 periods every week (each period is 45 minutes), so anyone teaching less than that has available time under his/her contract. 7 In principle any school, even one without and EDC’s AQE center, could follow an ALP-CS integration model. However, most likely only schools with a working accelerated education center, provided by EDC or another organization, will do so. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 5 RESEARCH OBJECTIVES AND QUESTIONS The main objective of this study was to analyze the transition of the accelerated learning program from the EDC-provided AQE model to the GoL-provided ALP-CS model. This assessment included analyzing what elements of the program have been sustained and which have changed. A secondary objective was to analyze to what extent the AQE model (as implemented by EDC since 2017) has contributed to equal access to education, in terms of transition to formal education, retention, and completion, and exploring if the AQE strategies could be used to support continued student learning in crises, such as the one created by the COVID-19 pandemic. Finally, the study also included cost analyses of AQE and estimations of the fiscal costs of implementing ALP-CS at scale. The research questions for this study, presented below, relate to program sustainability, equity and cost. Sustainability 1. What are the AQE elements that have been transitioned to and sustained by the MoE of Liberia? 2. What are the AQE elements that were not transitioned? What are the implications for program success when some elements of the AQE are not adopted by GoL, in particular, related to facilitators’ compensations and current workload? 3. How successful were AQE’s strategies to ensure sustainability and institutionalization related to the curriculum, AQE models, and teacher training approaches? Equity 4. What is the contribution of AQE to equitable transition to formal education, retention, and completion in Liberia? How did the age, gender, or location of a learner affect their trajectory? 5. What strategies for AQE delivery could be used to serve learners affected by an emergency like COVID? Cost 6. What are unit costs for AQE program participant access or completion? 7. What would be the cost to the MoE to continue to scale up the AQE model? METHODS 1.1 RESEARCH DESIGN NORC drew on methods associated with a realist evaluation approach for intervention process evaluations to help identify how the intervention worked for whom. We used an intervention framework based on the AQE transition model to assess how the revised intervention model worked for different students under which circumstances. To assess the implications of the transition to the new model, triangulated different stakeholder perceptions on the implementation and current outcomes of the transition to the revised school model. To inform the evaluation design, we developed a conceptual framework (depicted in Figure 1), which identifies: 1) the primary and secondary evaluation objectives; 2) the main AQE program transition elements to be evaluated stated as objectives, minimum package and learning content; and 3) projected sustainability and equity outcomes. This mixed methods evaluation design aimed to analyze the 6 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG transition processes, challenges and strengths leading to the potential sustainability and equity of the ALP following the handover from the EDC intervention design to the model managed by the Ministry of Education. Keeping in mind the program aim, to meet the education needs of young Liberians who have not accessed the formal primary school system, we examined the specific program objectives of the ALP transition process (Figure 1). This evaluation framework informed the study design, the study instruments, the coding framework, and analyses and our interpretation of the findings and recommendations. Figure 1. Evaluation framework for the Transition Process of the AQE-ALP from EDC to MoE. For details on the conceptual framework used to inform the design of questionnaires and qualitative research instruments, see Annex I. 1.2 DATA SOURCES AND DATA COLLECTION: NORC, in partnership with the local data collection subcontractor, The Khana Group (TKG), collected both quantitative and qualitative data for this evaluation. TKG is a research firm that works in 17 countries in Sub-Saharan Africa and has an office in Liberia. TKG has extensive experience in the International Development field working for USAID, DFID, and other agencies. QUALITATIVE DATA SOURCES AND COLLECTION Qualitative data came from four sources: Key Informant Interviews (KIIs), Focus Group Discussions (FGDs), Semi-Structured Interviews (SSIs) and document review. The NORC evaluation team worked with TKG to coordinate and conduct the KIIs, SSIs and FGDs. TKG staff received a 3-day qualitative data collection training in June 2021 covering interviewing USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 7 methods, ethics and confidentiality, security procedures, COVID-19 precautions, data management and storage, and participatory techniques for focus groups with children. Interviews and focus group discussions were conducted in English between July 12th, 2021, and August 6th, 2021, and were recorded. Then, the recordings were transcribed by TKG staff. NORC used a Youth Participatory Research approach for the focus groups with ALP students, who were between the ages of 11 and 18 years old. This approach used interactive exercises like games to create an environment meant to put youth at ease and encourage them to speak openly amongst their peers. Interviews took an average of one hour, ranging from 20 minutes to 2 hours, and focus group discussions took an average of one hour. Interview protocols can be found in Annex II. Documents reviewed included: • The Harmonized Alternative Education Policy of Liberia published in September 2017 • The AQE Midterm Evaluation, Operational Research Plan, and Quarterly Reports for FY19 and FY20 • Monitoring and evaluation documents from EDC, including the ALP Checklist Survey, ALP Functionality Rubric, and Classroom Observation Tool • The Operational Research Report FY20 for the ALP Conventional School Pilot (ALPCS) published in October 2020 • Documents about the AQE Model, including frameworks and presentations • Project tools such as the Learner Eligibility Guidelines and Accelerated Education Guide to the Principals QUANTITATIVE DATA SOURCES AND COLLECTION NORC used both administrative and survey data as sources of quantitative information. Administrative data included EDC’s records on students’ enrollment and assessments. Survey data, collected by TKG, included: • School observation surveys with principals • Teachers and facilitators survey • Follow-up survey of AQE graduates (with parents) A four-day training was conducted in June 2021, during which enumerators were trained in administering the three surveys, appropriate interview methods, ethics and confidentiality, COVID-19 protocols, enumerator roles and responsibilities, sampling procedures, best practices for tablet-based data collection, and data quality assurance. TKG then programmed and tested the survey questionnaires using SurveyCTO software, after which NORC also tested the forms on the server. All tools were piloted and adjusted, as needed. Supervisors oversaw field management (supplying equipment to field teams, verifying completed tests, and ensuring delivery), as well as liaising with the central office management. They ensured that data security procedures were followed and that all completed questionnaires were delivered to the central office securely to safeguard confidentiality. 8 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG During fieldwork, data were uploaded periodically to the server. NORC conducted regular data quality checks and logged issues in a fieldwork log for review by the fieldwork manager to make necessary adjustments and corrections during data collection. The integrity of data was safeguarded through password-protected tablets/devices through data collection, and quality assurance was followed as needed through mechanisms such as backchecks, tracking interview times, and more. ETHICS AND COVID-19 PRECAUTIONS NORC and TKG ensured that all information shared by respondents was confidential. Personal identifying information was removed from all data before analysis. All data were shared using a Secure File Transfer Protocol system which encrypts and protects the security of files. NORC also received approval from the University of Liberia PIRE Institutional Review Board and the NORC Institutional Review Board to conduct this research. All work under this evaluation was conducted following health protocols consistent with Centre for Disease Control (CDC) and Government of Liberia guidelines. Please find details about COVID-19 protocols in Annex IV. QUALITATIVE SAMPLE In total, 87 interviews and focus group discussions were conducted. For qualitative data collection, NORC selected two schools with an ALP center –one urban and one rural- in each target county, for a total of 12 schools. In each school, we conducted two KIIs with ALP facilitators and one KII with the principal. Three interviews with ALP facilitators could not be conducted due to limitations such as school closures and difficulties accessing some rural schools due to dangerous roads in the rainy season. Despite these challenges, 87 out of 90 planned KIIs and FGDs were completed. In the communities where these schools were located, the evaluation team organized one FGD with parents of ALP students including Parent Teacher Association (PTA) members, one FGD with students, and one KII with a community leader. In addition, there were 18 additional KIIs conducted with AQE staff, USAID staff, County Education Officers and MoE officers. All focus groups were separated by gender. In addition to the KIIs and FGDs, the qualitative data collection team conducted three additional, short￾duration SSIs with principals of schools in Bong County (2) and Montserrado County (1) that closed their ALP programs, to find out the reasons why the ALP centers closed, the current status of their former ALP students and their expectations for a future ALP center at their schools. For the full qualitative sample by respondent category, please see Table 3 below. Table 3: Qualitative Data Collected per Respondent Category: Respondent Category Number of KIIs and FGDs Planned Number of KIIs, SSIs and FGDs Conducted AQE teachers/facilitators 24 KIIs 21 KIIs School Principals 12 KIIs 12 KIIs and 3 SSIs USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 9 Respondent Category Number of KIIs and FGDs Planned Number of KIIs, SSIs and FGDs Conducted EDC staff; MoE Officials; County Education Officials; and USAID staff 4 EDC Staff KIIs 4 USAID staff KIIs 6 County Education Officer KIIs 4 MoE Officer KIIs 4 EDC Staff KIIs 4 USAID staff KIIs 6 County Education Officer KIIs 4 MoE Officer KIIs Parents ALP Students 12 FGDs 12 FGDs 12 FGDs 12 FGDs Community Leaders 12 KIIs 12 KIIs QUANTITATIVE SAMPLE The quantitative data collection team interviewed principals, parents, and teachers. Principals of 90 schools where EDC supported AQE centers were surveyed. To sample these schools, NORC used administrative data from EDC. There were 281 schools in the enrollment data file provided by EDC. NORC dropped 19 schools with no data on student enrollment, for a final sample frame of 262 schools. Using this sample frame, 90 schools were randomly selected, stratifying the sample by cohort and county. Along with the principal survey, the quantitative data collection team conducted school observations in these 90 schools. Observations included analyzing the student rosters to record the number of students by grade and gender and searching for a predetermined list of students from the AQE program to verify their current enrollment status. In each school the evaluation team searched for a list of 10 students, for a total of 900 students. The evaluation team also interviewed parents of a subset of these 900 students. The objective of these interviews was to determine if the student had graduated, transferred to another school, or dropped out of school. We only interviewed parents of students that they could not find in the rosters and that were potential graduates, that is, were in level 3 in 2018-2019 or in 2019-2020. NORC assumed that all students that were not potential graduates and were not in the rosters dropped out of school. The data collection team attempted interviewing parents of 179 students and conducted 143 interviews. They could not find the parents of 36 students so for these students their enrollment status could not be determined. This implies that out of the 900 sampled students, NORC were not able to establish the enrollment status of 4 percent, which should not affect the overall results. Finally, in the same 90 schools where principal interviews and school observations were conducted, two ALP facilitators were also interviewed. In schools where there were fewer than two facilitators, NORC supplemented the sample with regular teachers. As a comparison group NORC also surveyed two teachers in each of 30 schools located in the same six counties where AQE was fielded, but that were not part of the AQE program. Therefore, the target sample size for the teacher/facilitator survey was 240, of which 218 interviews were conducted. Fewer teachers were interviewed than had been originally planned, because in some schools there was only one teacher or facilitator. Table 4 summarizes the quantitative data collected. 10 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Table 4: Quantitative Data Collected per Respondent Category Respondent Category Number of Interviews Planned Number of Interviews Conducted Principals/school observations 90 90 Parents 179 143 Facilitators and teachers 240 218 1.3 DATA ANALYSIS: QUALITATIVE DATA ANALYSIS For the qualitative components of the study, NORC triangulated analysis across the participants, listed in Table 3 above. The analysis primarily aimed to assess two outcome dimensions of the transition: sustainability and equity, but simultaneously explored the acceptability and quality of the new model after the transition. NORC’s qualitative analysts coded all transcripts using a deductive coding scheme in Dedoose qualitative analysis software. The coding scheme was based on the conceptual research framework and then was further developed using an iterative testing process to ensure inter-rater reliability. The coding allowed for the extraction of relevant excerpts by theme and evaluation question. These excerpts were analyzed to find congruencies and divergences among respondents and to provide information to answer the evaluation questions. QUANTITATIVE DATA ANALYSIS SUSTAINABILITY To answer the research questions on sustainability, the principal survey included questions on the extent to which the school was following the minimum package, now that accelerated education is supported by GoL and not EDC. This survey included questions on monitoring and evaluation: facilitator recruitment; training and compensation; learning assessments; and teaching and learning resources. NORC analyzed these data to determine the extent to which different ALP components have been sustained. EQUITY NORC analyzed retention rates and other outcomes for the AQE program between 2017-2018 and 2019-2020 using EDC’s administrative data, and also analyzed retention rates between school years 2019-2020 and 2020-2021. For this analysis, at the end of school year 2020-2021, NORC surveyed the enrollment status of a sample of ALP students in AQE schools. The student sample frame included the students that sat for the 2018-19 completion assessment and the students that started the ALP program during the school year 2019-20. Given the available data, this is the most complete account of the students who were enrolled in school year 2019-20. The sampling scheme follows a two-stage design. First, 90 AQE schools were sampled (these are the same 90 schools surveyed to answer the sustainability questions). Second, in each sampled school 10 ALP students were sampled (900 students in total). One of the key objectives of the survey was to determine what ALP students were doing post-graduation, however only 6 percent of the sample frame USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 11 were in Level 3 in 2019-2020, so NORC oversampled these students. Sampling weights were used to recover population parameters. When students could not be found in school rosters, parents were interviewed to inform what these students were doing in 2020-2021. With these data, NORC calculated the current enrollment status of the sampled students to determine retention rates and transition to conventional school. NORC analyzed these data by gender, age and rural/urban, to document the implications for equity of the ALP program. COSTS NORC received expenditure data from USAID/Liberia and from the AQE implementer, EDC. These data cover the period March 2nd, 2017, to March 31st, 2021. AQE reports include information about in￾kind contributions received by the program which is also used in estimates of costs. Finally, conversations and correspondence between the evaluation team and EDC provided additional details about the implementation of program components and their associated costs. 1.4 LIMITATIONS There are several limitations to this study, which include: • Most respondents in interviews and focus group discussions spoke Liberian English, and this may have minimally influenced researcher understanding of the transcripts. Some quotes were changed in minor ways to improve readability (changing grammar) for an audience that is unfamiliar with the Liberian dialect of English. • In the interviews with respondents from the MoE and EDC, it was clear that they felt a sense of ownership over the program, which may be reflected in their statements and could introduce some bias to the findings. • There were some challenges using administrative data from EDC, among which was that the completion assessment data for 2020 did not include student ID numbers, so it was not possible to link them to other records. FINDINGS In this section NORC presents the findings of the study, organizing them around the evaluation questions. 2.1. WHAT ARE THE AQE ELEMENTS THAT HAVE BEEN TRANSITIONED TO AND SUSTAINED BY THE MOE OF LIBERIA? 2.2. WHAT ARE THE AQE ELEMENTS THAT WERE NOT TRANSITIONED? WHAT ARE THE IMPLICATIONS FOR PROGRAM SUCCESS THAT SOME ELEMENTS OF THE AQE WILL NOT BE ADOPTED BY GOL, IN PARTICULAR RELATED TO FACILITATORS’ COMPENSATION AND WORKLOAD? To determine the elements of the EDC run AQE program that transitioned to the government-run ALP program, NORC asked school principals if they still have ALP students in their schools, and for those 12 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG that do, asked questions to assess what elements of the AQE minimum package are still part of the ALP program. Table 5 shows the percentage of schools that offer each grade and level, by whether the school offers conventional, ALP, or both. Seventeen of the 90 surveyed schools (19 percent) closed their ALP program and have only conventional school grades. Eighteen schools (20 percent) have all their students enrolled in ALP. Finally, 55 schools (61 percent) have both conventional grades and ALP levels open. Table 5. Surveyed of schools by whether they offer conventional, ALP or both SCHOOLS % Conventional only 17 19 ALP only 18 20 Both 55 61 Total 90 100 ALP CENTER CLOSURES AND OPERATIONS The survey asked principals of schools that do not have an ALP program anymore about the reasons why the program is no longer in operation. As Figure 2 shows below, there is a wide variety of reasons for closures. A lack of facilitators and constrained physical space were mentioned frequently. Principals also referenced the unwillingness of facilitators to teach in the integrated ALP-CS system and parents’ preference for conventional school. Other principals said that it was hard for students to attend school in general. To explore the full range of reasons for closures, NORC conducted additional SSIs with principals, described below. Figure 2. Reasons for shutting down the ALP program (N=16) 31% 38% 31% 25% 13% 13% Other Not enough teachers/facilitators Not enough classrooms Lack of other resources Facilitators did not want to teach both systems Students/parents prefer conventional school During qualitative fieldwork, the data collection team conducted brief, informal SSIs with principals at two schools in Bong County and one in Montserrado County to discuss reasons why the ALP centers closed, the status of their former ALP students and their expectations for a future ALP center at their schools. Reasons for ALP center closures described in this qualitative data included: a. No prior communication from either AQE or MoE to school principals about what is expected for ALP-CS system. Schools did not know how to handle the transition. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 13 b. No stipends for facilitators. Some facilitators were still owed payments for their previous instruction at the time of the interviews. c. No new learning materials were provided to distribute to students. d. No seating capacity: Under the ALP-CS model after the transition, schools hold classes for both conventional school grades and ALP levels during the morning shift and in many cases, there are not enough seats or space. In one school in Montserrado County, the MoE was using space to run vocational training on the campus, and the school did not have any other space to house the ALP. Most students who were previously enrolled at these ALP centers transitioned into conventional school at the same schools. Those remaining either moved to other schools or were taken out of school since parents could not afford the conventional school fees. At each school, principals expressed disappointment over the ALP center closure, but were also hopeful that the centers could open again in the future. The following were listed as key to resuming the ALP classes in these schools: a. Instructions from the MoE about what needs to be done during the transition; b. Community and parent engagement to take ownership of the ALP, mobilize students and encourage enrollment; PTA involvement and effort from community leadership are needed; c. Renovation of an old building in order to shift the vocational school and make space for ALP sections (Montserrado County); d. Blackboards to aid instruction; e. Distribution of teaching and learning materials to the schools; and, f. Support to the GoL for i) teacher training, ii) capacity building and professional development and iii) monitoring and evaluation of classes to ensure activities are being duly carried out. For ALP operations that are still functioning, Table 6 shows the fraction of principals that say that ALP classes are pooled with classes from the conventional system, for schools that run both ALP and the conventional system. Depending on the ALP level, between 17 and 26 percent of ALP are shared with the conventional system. Table 6: Percentage of schools where ALP classes are pooled with classes from the conventional system LEVEL 1 LEVEL 2 LEVEL 3 % 19.1 17.1 25.8 Schools 47 41 31 Note: For each level, the number of schools is lower than the total number of schools where there are both ALP and conventional system (55), because not all have all the three ALP levels. Keeping two separate basic education systems within a single school is likely more costly than running only one. One of the key advantages of maintaining separate teaching modes is to allow schools to cater to the specific needs of on-age versus overaged children as needed. However, within schools that currently run both systems, this division is rarely observed. Table 7shows the mean age for students in conventional and ALP in schools that run both systems. Notably, on average, students in Level 1 are less than one year older than students in grades 1 and 2, students in Level 2 are also less than a year older than students in grades 3 and 4, and on average, students in Level 3 are even a bit younger than students in grades 5 and 6. Results for the median are similar. The ALP enrolls overaged students between 8 and 14 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 15 as it is meant to do, but the issue is that conventional school also has overaged students enrolled. A study by Menendez and Monroy-Toborda (2017) confirms this finding. Across ninety schools, their findings show that towards the end of the school year the average age of grade 2 students was 12.5. Overaged students from conventional schools were never transferred to the ALP and there were no instructions to do so under the AQE implementation of the program or under the transition of program to the GoL. Table 7: Mean and median student age by grade/level and system for schools with both systems (n=47) CONVENTIONAL ALP MEAN MEDIAN MEAN MEDIAN Grades 1-2 / Level 1 10.9 11 11.5 11 Grades 3-4 / Level 2 11.8 11 12.4 13 Grades 5-6 / Level 3 15.7 16 15.3 15 Note: In each school we asked for the ages of students in one grade and one ALP level only, for brevity. Therefore, the results for each grade/level are derived from fewer schools than 55. ONE SCHOOL APPROACH MODEL STRUCTURAL CHANGES FROM THE ONE SCHOOL APPROACH: COMBINING CLASSES The structural changes that resulted from combining ALP and conventional sessions had mixed consequences for students’ engagement, experiences, and the quality of their education. Qualitative data revealed some differences in the One School Approach between counties and schools, but in every school this approach involved having conventional and ALP students in school at the same time during the day, i.e., in the same classroom as conventional students. For example, instead of having conventional students come to school from morning to early afternoon and then having ALP students come from mid-afternoon to evening, all students under the One School Approach are in school from morning to early afternoon. While some schools tried to separate ALP students by having them sit separately from conventional students in the same room, often, ALP and conventional students were seated together. Since having conventional and ALP students in school at the same time increased the number of students per classroom, some schools had to find solutions such as using auditoriums to provide enough space. Teachers in classrooms with both conventional and ALP students most frequently described combining the ALP and conventional curricula, since they could not simultaneously teach different curricula to different sets of students. They explained that since the ALP materials are well aligned with the conventional curriculum, they sometimes preferred to use the ALP curriculum for both conventional and ALP students; the ALP curriculum is also more modular, requires less lesson planning, and gives more thorough explanations of the material. Since these teachers were also given training on the ALP curriculum, they felt comfortable using it. Therefore, most stated that while they mixed conventional and ALP materials, the ratio skewed more towards ALP. On the one hand, many respondents across participant categories praised the shift from separate afternoon ALP sessions to combined sessions that started in the mornings, because students were less USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 15 likely to be tired by afternoon and choose to skip class. On the other hand, combining ALP and conventional sessions strained the capacity of school infrastructure and other resources. Interview and focus group respondents indicated that there were frequent problems related to large class sizes, unbalanced student-teacher ratios, and lack of space, adequate teaching and learning materials, or sufficient seating for students. BENEFITS OF COMBINING CLASSES Respondents suggested that when there were classes in the afternoon, students would be exhausted from farming or doing chores during the morning and thus struggled to concentrate in the heat of the afternoon, as a facilitator explained: The children used to come in two shifts, and when the morning session left [and] the afternoon ALP session came, they [the ALP students] were too tired because of the sun. (Lofa County ALP Facilitator, KII) Other benefits of switching to one morning session are that the ALP students feel better when they are in sync with their peers and therefore there is better attendance, as described by an ALP teacher in Montserrado county: They come in more than they were in the afternoon on a daily basis now. Because they see their other brother and sister coming in the morning too, so everybody dress[es] to come. They are encouraged to come in the morning. (Montserrado County, ALP teacher, KII) Parents in every county overwhelmingly preferred the timing of the morning sessions. They said that when ALP had a separate afternoon session, they had to be at their farm or work and could not be home to make sure that their children went to class. In contrast, when their children left for school in the mornings, parents could supervise their morning routines and ensure their attendance. A further reported benefit of combining ALP and conventional classes is the additional time that ALP students have with teachers. For example, teachers from Lofa County explained that the separate afternoon sessions for ALP had been rushed, and they appreciated having more time after the transition: The first time, the learning was so interesting, it was very important but the only thing the problem was having the time was very short compared to this time. And for now, we have up to 3 o’clock and wherein children have the opportunities to express themselves in asking questions. (Lofa county, ALP Teacher, KII) CHALLENGES WITH COMBINING CLASSES As suggested above, one downside to the structural changes under the One School Approach is that it increased class sizes, often beyond the seating capacity of classrooms. A principal and a community leader complained about these problems: [We have] about 60 to 50 students in one class […] before when ALP was here, we used to separate the classes. […] but now the classes are combined so it makes the thing narrow for me. (Nimba County, Principal KII) 16 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG The building, we have over 1000 students that are here, then incoming, ALP people coming …in the morning hour, it bring[s] some little problems, the building is not hosting. I think they usually have their ALP class in the morning hour in the auditorium and now, the classes cannot hold everybody... (Bong County, Community Leader KII) Additionally, some respondents described stigma against overage students being in the same class as their younger peers. An ALP teacher in Bong County said: The smaller ones used to laugh at them. Yeah, they used to play fun out of them say ‘you too big for the class’. (Bong County, ALP Teacher KII). This is also a problem in conventional schools where overaged students share the class with young children. Finally, since the ALP and conventional students were combined under the One School Approach, respondents from nearly every category and across counties complained about not having sufficient chairs for students and said that this was a major issue affecting attendance. Interviews with ALP teachers, students, and parents each repeatedly commented on the absence of very basic equipment for a school: My son can always talk about chair business, he says no chair in the class so that’s why he can complain all the time […] if he don’t come on time, and see the place occupied, he will not get seat to sit down. (Bong County, Parents, FGD) All I say is everyone should have their workbook and chairs at least till the level three phase of work, because there is no door, it’s just open, no chairs for them. (Montserrado County, ALP Teacher, KII) [Interviewer: What kind of hard time can you face here as an ALP student?] That chair business. Yeah, we can suffer for chair here bad way. (Montserrado County, ALP Students, FGD) AQE did construction work to rehabilitate classrooms in disrepair and fix latrines and provided school furniture. However, under the One School Approach, all the students use the facilities during a single shift creating a shortage of classroom space and seats. NUTRITIONAL NEEDS AND MEAL PROVISION Although feeding or nutritional support was not part of the ALP program or specifically assessed as part of the evaluation, some respondents, especially parents and students in the focus groups, recommended that the ALP establishes a meal program. The absence of food provision for students was recognized as affecting their attendance and ability to concentrate. In one focus groups with ALP students in Grand Bassa county, a student explained: There is no food so, from 7am to 2pm see [with] those kinds of hours, 7 good hours... So, sometimes I can be hungry, we come to school to come and sit down in class I can be hungry so that’s what make it sometimes I can miss day. (Grand Bassa County, ALP Students, FGD) Parents in a focus group in Bong County called for some sort of nutrition program: USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 17 They [students] say they're feeling bad […] They not feeding the children […] now, they ain’t, they ain’t getting the things they used to get in the evening. (Bong County, Parents, FGD) During the interviews, it was not uncommon for people to mention that links between poverty and drop-out are also related to hunger, suggesting the need for meal plans of some sort. Community leaders in Montserrado and Bong Counties described this relationship: Money, is not only attached to them getting book bags, getting copy books and other stuff. What really can keep kids in school, especially in this society that we find ourselves in, is the first little breakfast that they will have in the morning to eat, so when parents can’t even afford to provide that, you know a child will not just go to school hungry. (Montserrado County, Community Leader, KII) Some of those people, they don’t have resources to support the children, with mainly food and all those things. No learning, no learning, you can't learn, empty bag can't stand. You will not come to school if you will not get food to eat, so you need to get on the hustle. (Bong County, Community Leader, KII) 2.2.B. MINIMUM PACKAGE MONITORING AND SUPERVISION The principal survey included questions to assess which elements of the minimum package remain part of the ALP. Only the 73 schools that have ALP students were asked these questions. To supplement the principals’ responses, NORC also asked ALP facilitators questions about aspects of the minimum package that are relevant for them (e.g., teachers’ recruitment, training, and compensation). ALP centers are supposed to follow certain guidelines to monitor and supervise the implementation of the ALP. These guidelines include disseminating information relevant to the ALP and keeping attendance records. As Table 8 shows, 92 percent of schools have ALP classes integrated in the school timetable, 56 percent have the daily schedule of ALP classes posted on school bulletin board, 86 percent keep student attendance records, 38 percent have the ground rules for ALP classes in the school bulletin board and/or in classrooms, and 95 percent are receiving support supervision from the District Education Officers (DEOs) or County Education Officers (CEOs). Table 8: Monitoring and supervision in schools with ALP centers (n=73) MONITORING AND SUPERVISION PERCENTAGE OF SCHOOLS WITH ALP Are ALP classes integrated in the school timetable? 92 Is the daily schedule of ALP classes posted on school bulletin board? 56 Does the school have ALP attendance records? 86 Are ground rules for ALP classes on view in the school bulletin board and/or in classrooms? 38 Is the ALP-CSP receiving support supervision from the DEO or CEO (by phone/by site visit)? 95 18 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG As explained above, part of the minimum package is that the MoE certifies all ALP centers. However, currently the MoE is not doing this. INSTITUTIONAL SUPPORT FOR ALP TEACHERS Formally, monitoring and evaluation of ALP is done by principals, Accelerated Basic Education Supervisors (ABES), DEOs, and CEOs. However, there also appears to be informal monitoring by parents and community leaders in some counties. The level of engagement from parents was highly variable between counties, as described in the section on community involvement below. At the administrative level, before the transition, EDC prepared monitoring tools for larger scale oversight of the ALP program in partnership with the Ministry’s Bureau of Planning, Research and Development and the MoE Supervisor of Monitoring and Evaluation. These tools synthesized processes for monitoring data that flow upwards. In these processes, Accelerated Education Supervisors, District Education Officers and County Education Officers visit schools regularly and receive information from principals. They then report to the national coordinator for the ALP program in the Ministry of Education. In interviews with ALP teachers, they said that monitoring from higher level education officials was infrequent, in contrast to consistent and frequent support from school principals. One teacher in Montserrado county, when asked if the principal, any local education officer, or any MoE official had supported them responded: Only my principal. The principal has been with us, and she is with us in the process. Sometime when some of us are facing little problems, we are not on the campus, she will come and replace us to carry on the teaching. (Montserrado County, ALP Teacher, KII) Respondents in Grand Bassa, Margibi, Lofa, and Montserrado counties also mentioned principals frequently monitoring ALP classes and substitute teaching for absent teachers. Education officials above the school level who were tasked with monitoring the ALP described logistical challenges that prevented them from fully completing their circuits. In particular, they called for more motorbikes for the Accelerated Education Supervisors and vehicles for the County Education Officers. An EDC staff member and an Accelerated Education Supervisor explained this problem: If he [Accelerated Education Supervisor] wants to visit a classroom in Montserrado there is no motorbike to go. If there’s a motor bike, there’s no fuel to go. If he wants to sleep in Lofa to do proper inspection there is no lodging around for him, simply the Ministry does not have sufficient money to run. So, because of that, you can do only so much. (EDC staff member, KII) I am Monitoring, but it is not in full […] When I say it’s not in full, I don’t have the capacity and the ability to go way into […] distant places that we would not just easily venture without some level of support. (County Education Official, KII) The limited resources to support monitoring have led to problems with reporting data to the highest levels. A MoE official, when asked how often they get reports from Accelerated Education Supervisors, responded: “That’s another challenge too, because they too been complaining, there’s nothing to work USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 19 with, and xyz, so, it’s been challenged with us when it comes to reporting.” (Ministry of Education Official, KII) PARENT AND COMMUNITY INVOLVEMENT The role of parents and Parent Teacher Associations (PTAs) differed not only by county but also by community within counties. In some communities, parents helped with monitoring the ALP program, provided material support to teachers, or even paid stipends to volunteer teachers after the transition. In other communities, however, parents in focus groups stated that they did not provide any kind of support. After the transition, some communities began using PTA fees or other fundraising projects to provide classroom materials or compensate volunteer teachers. For example, in Nimba county, respondents reported that one PTA took on extra collective farming projects to raise money for their community’s school. However, this kind of material support was not provided everywhere. In Lofa county, parents in a focus group for one school said that they don’t provide any material support for ALP teachers, but parents in a different school in Lofa raised money to cover stipends for volunteer teachers. A parent from this focus group explained, “at least, those that are not on that government salary, can be helped […] AQE used to pay the teachers, the afternoon school. Now, they’re not paying them, and we’re paying them.” (Lofa County, Parents, FGD) Communities with more organized PTAs also had them support the monitoring of the ALP. Community leaders in PTAs in Nimba and Montserrado County described giving intensive support to ALP teachers that included assisting with monitoring: If you ask for the record book, you’ll see my signature in it, for every time I visit them, I will sign, and the purpose of my visit. Sometimes I will go and even inspect the teacher roll book, inspect their lesson book, sometimes I sit in the class, and monitor them as they teach. (Montserrado County Community Leader, KII) I am the secretary for the PTA […] with the ALP, I think I'm the supervisor. When the monitors are not here, I'll look after the teachers and the students and then for whatsoever problem they have I go and solve it for them. Like the stationary, placing of whatsoever teaching materials, textbooks, and things, I have it kept and deliver it to teachers and students. (Nimba County Community Leader, KII) FACILITATOR RECRUITMENT, TRAINING, AND COMPENSATION ACCELERATED EDUCATION FACILITATORS To assess which aspects of the AQE program have been maintained related to facilitators’ recruitment, training, and compensation, NORC surveyed two facilitators/teacher in each of the 90 schools where principal surveys were conducted. As a comparison group, two teachers were surveyed in each of 30 schools that were located in the same six counties in which AQE was fielded, but that were not part of the AQE program. In total, NORC surveyed 218 facilitators/teachers, of which 29 facilitators teach ALP 20 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG only, 87 teachers teach conventional school only, and 102 teacher/facilitators teach both systems.8 Table 9 shows descriptive statistics by teacher type. Thirty-one percent of ALP facilitators are women, which is higher than the figure for conventional school teachers (20 percent) and those that teach both systems (17 percent). In theory, facilitators need at least a C-Certificate to teacher ALP, however 20 percent have a high school degree or less. Table 9: Descriptive statistics by teacher type ALP (n=29) Conventional (n=87) ALP and Conventional (n=102) Women (%) 31 20 17 Education (%) High School and Lower 13.8 17.2 22.5 C Certificate 51.8 61.0 58.9 B Certificate 10.3 5.0 6.9 AA Certificate 3.4 5.8 5.9 Some College 7.0 8.0 5.0 Bachelor's Degree 10.3 3.4 1.0 Other 3.4 - - Avg years of experience 13.0 13.1 11.9 Only ALP facilitators were asked questions on recruitment and training, so the underlying sample for the analyses on these two subjects is only the 131 facilitators that either teach only ALP (29) or both ALP and conventional system (102). Questions on teacher compensation and burnout were asked to all 218 surveyed teachers/facilitators, so the analyses produced results for all types of teachers/facilitators. FACILITATOR RECRUITMENT Table 10 shows that most of 131 ALP facilitators interviewed had completed a formal recruitment process, including an interview and/or a competence assessment to start teaching in the ALP program. Because the government has overseen the program for only one year, it is difficult to assess whether the current recruitment process replicates what was done between 2017 and 2020. In fact, only one of the surveyed facilitators interviewed started working this year. Table 10. Recruitment process for ALP facilitators (%) RECRUITMENT MECHANISM PERCENTAGE OF TEACHERS Teachers who sat for an interview 89 Teachers who sat for written competence assessment 91 TRAINING 8 All facilitators teaching ALP only are from the 90 AQE schools, 47 of the 87 teachers teaching conventional system only are from the comparison schools and the remaining 40 are from the AQE schools, and all but one of the 101 teacher/facilitators teach both systems are from the AQE schools. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 21 Of the 131 ALP facilitators interviewed only 11 (8 percent) did not receive ALP training in the last five years. As Figure 3 shows, the fraction of trained teachers increased between 2016-2017 and 2019-2020, reflecting the introduction of new cohorts of facilitators over time, and declined between 2019-2020 and 2020-2021 from 61 percent to 24 percent. In principle, this doesn’t necessarily mean that training is not being delivered appropriately now that the government is running the ALP program. Facilitators are supposed to be trained over multiple sessions during a year, but once they finish, they do not need to attend more training sessions, and are supported through coaching sessions instead. In this sense, in schools that were formerly AQE, not all teachers need to receive training each year, only teachers that have never been trained do (e.g., novice teachers). Therefore, more important than how many teachers are trained each year is how many teachers have not received any training at all. We found that eight percent of ALP facilitators have not received training in the last five years. How this figure evolves in the coming years will reflect whether the pace of government training for ALP facilitators is adequate. Figure 3: Fraction of teachers that receive ALP training by year 5% 23% 46% 61% 24% 8% 0% 10% 20% 30% 40% 50% 60% 70% 2016_17 2017_18 2018_19 2019_20 2020_21 No training Facilitators’ views on training sessions were generally positive. As Table 11 shows, 99 percent said they learned new things and 100 percent said the training was useful. Fifty-three percent said that training should be longer, and 98 percent said they were applying in class they had learned in training. Table 11. ALP Facilitator Opinions on ALP Trainings (n=120) QUESTIONS ABOUT ALP TRAINING PERCENTAGE YES Did you learn new things? 99% Did you find it useful? 100% Do you think the training was long enough? 53% Are you implementing the ALP approach in your ALP classes? 98% In terms of coaching, ninety percent of ALP facilitators reported receiving feedback about their teaching after being observed in class, as Table 12 shows. All respondents found this feedback either very useful 22 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG or useful. Results show that teachers were observed with regularity. Figure 4 shows that most teachers were observed either monthly or more frequently than that. Observations were conducted by the school principal in most cases, and also by AQE staff. Table 12. Teachers who were observed during lessons in this academic year (n=128) COACHING PERCENTAGE OF TEACHERS Teachers who received feedback on their teaching 90% Figure 4: Frequency of teacher observations (n=115) 2% 3% 6% 10% 78% 00% 20% 40% 60% 80% 100% Once a year Once per semester Once every two months Once a month More often than once a month Many teachers expressed that they could benefit from refresher trainings, especially when it came to managing their classrooms after the transition. As one facilitator noted: The ALP teacher training, like for me I am a teacher for ALP and that has made my mind brighter, how to present in the classroom, how to draw children mind whenever you are in the classroom and the way, our training too. The last time there was a group that came and some, I make my recommendation that if the program having to extend, I believe that you can call for 2 weeks or 3 weeks, teacher training to refresh teacher mind, that will be good. (Lofa County, ALP Facilitator, KII) Having previously participated in regular trainings, teachers expressed that they were able to continuously improve on their teaching, not only for the ALP students, but for teaching the conventional classes as well. An ALP facilitator and a principal in Margibi County and Grand Bassa counties stated: When USAID was in power, there was regular workshop, that has to do with the training of the teachers and the administrators concerning the program. That was good […]. Sometimes when we forget what we need to do, after the workshop we pick up […]. But since government took over, nothing. (Margibi County, ALP Facilitator, KII) The teachers training was very important because it was helpful, it helped our teachers to know exactly what to do at a particular time, what not to do, not only for the ALP but it also help them to know how to take care of the conventional school, what lesson to be taught, what lesson not to be taught, what lesson to be planned, I mean so the teachers training is very good and very, very important for our teachers. (Grand Bassa County, Principal, KII) USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 23 COMPENSATION Between 2017 and 2020 ALP facilitators received a stipend of 50 USD per month for their participation in the AQE program. As explained in section 1.2, for school year 2020-2021 the government suspended this payment and also had teachers/facilitators teach both ALP and conventional school. Table 13 shows the percentage of teachers/facilitators that are being compensated, categorized by whether they teach ALP, conventional or both. Only 13.8 percent of teachers teaching ALP exclusively are being compensated. These funds may be coming from the parent-teacher associations as described in the subsection on parental and community involvement in section 2.1. In contrast, 71.3 percent of teachers who exclusively teach conventional school are being compensated. Finally, 75.5 of teachers teaching both systems are being compensated for their work as conventional teachers, but only 9.8 percent are being compensated for their work as ALP facilitators. Table 13. Percentage of teachers/facilitators that are compensated TEACHER TYPE COMPENSATED AS: N CONVENTIONAL ALP ALP only N/A 13.8% 29 Conventional only 71.3% N/A 87 ALP/Conventional 75.5% 9.8% 102 The fact that the GoL is not compensating ALP facilitators like AQE used to creates several challenges for program implementation. Teachers or facilitators teaching both systems are likely to feel overworked and less satisfied with their jobs. Those who received compensation from AQE previously are likely to feel that now they are doing some work for free. A school principal in Grand Bassa county stated: It's a serious problem because the teachers feel that they not getting anything except their regular salary, and they complain that the program is an extra load on them. (Grand Bassa County Principal, KII) One respondent from EDC indicated that Ministry officials believed that the program could operate without the stipend for those teachers who were on the government payroll. However, a Ministry official interviewed for this study expressed concern that the withdrawal of the stipends might “demotivate our teachers” and added that he hoped the ALP program could be sustained until the government has the capacity to provide the necessary resources. The cut-off of stipends has posed significant challenges for teachers who teach both conventional school and separate ALP classes. Although principals and ministry officials do their best to encourage teachers to continue teaching despite the extra workload, teachers expressed dissatisfaction with the expectations to wait until they receive compensation. An ALP facilitator in Nimba county stated: We all are human beings. We ask questions, we say, ‘will it-’ because in our last year, our last semester, ALP use to give us a stipend, so when we asked the question, we say ‘oh, honorable, will the ALP still continue their stipend?’ They said no. We say, ‘but then we cannot run the ALP, because the ALP program, is a program by itself that we, we are entitled to something.’ 24 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG But now, they say only government and when you are on government payroll, only government. But last year we had volunteering teachers that were teaching and working in the ALP. So, mostly some of them are feeling somehow reluctant, but still, we are giving them, telling them, encouraging them. We say ‘oh, maybe one day your chance will come.” (Nimba County ALP Facilitator, KII) While the MoE made efforts to foster a smooth transition from the EDC, there can be little doubt that the loss of the stipend had substantial effects on staffing. As a senior advisor observed: “…a lot of [schools] fell off because of the stipend among other things”. It also appears that a significant proportion of the ALP staffing losses during after-school hours were ‘volunteer teachers’, who left when they learned they would not be compensated. In response to the staffing loss, the MoE strategized to keep the ALP going using teachers who were on the payroll and integrating the ALP into the conventional school hours. But, understandably, even these teachers remained frustrated at losing the stipend for their afternoon teaching. At the same time, according to a MoE official, the Ministry “retired a lot of teachers and now they are trying to bring … young teachers onto the government payroll”, which she admitted was a challenge that became apparent because of the ALP. One MoE official explained that many have pondered the problem of sustainability without the stipends: Those small, small stipends will be halted immediately after the program closes. So, does government have the resources to be able to continue? This is a question that we are all working around to see how we can answer. (Ministry of Education Official, KII) In Margibi county, there are concerns for how the schools will manage when students progress to higher levels and there are not enough teachers on government payroll: The enrollment for this year is not like what we have for the regular program when the NGO was over it. Because when the NGO was over it, we had level one and level two. The level two teacher was a volunteer teacher that the NGO was paying and even giving them stipend. But since they left, we couldn’t maintain the level two teacher because we don’t have the stipend to give them, so we only, we maintain the government paid teacher, because it's just his salaries that government give. (Margibi County, Principal, KII) TEACHER BURNOUT We asked teachers about their workload measured in teaching hours per week. The first row in Table 14 shows the average number of reported teaching hours. Facilitators only teaching ALP teach 17.9 hours per week on average, while the figure for those only teaching conventional school is 22.4 hours per week, and for those teaching both systems it is 22.9 hours per week. Teachers are supposed to teach almost 20 hours per week (25 periods of 45 minutes each), so as benchmark for overworking Table 14 also shows the fraction of teachers working more than 25 hours per week. The percentage of staff that works more than 25 hours per week is 40.4 percent for those that teach both systems, 38.8 for those teaching conventional school only, and 24.1 percent for those teaching ALP only. These results indicate that there are several teachers teaching more hours than they are supposed to. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 25 Table 14. Hours per week worked by type of teacher/facilitator ALP CONVENTIONAL BOTH Average hours worked per week 17.9 22.4 22.9 % Who work more than 25 hours per week 24.1 36.8 40.4 The teacher survey included the Maslach Burnout Inventory (MBI) for Educators Survey (Maslach et al, 1996). The MBI is utilized in educational research to measure degrees of educator burnout. It draws on three burnout scales – emotional exhaustion, depersonalization, and personal accomplishment. We translated the instrument to Liberian English, pre-tested and adapted to secure proper understanding of the items Table 15 shows the possible score ranges for the three MBI subscales and the classification in low, average, and high depending on the scores. Table 15. Maslach Burnout Inventory (MBI) scores classification MBI SUBSCALES LOW AVERAGE HIGH POSSIBLE SCORE RANGE Emotional Exhaustion ≤13 14-23 ≥24 0-63 Depersonalization ≤2 3-8 ≥9 0-35 Personal accomplishment ≥43 42-36 ≤35 0-56 Source: Maslach, Jackson, and Leiter (1997). Figure 5 shows the average emotional exhaustion score by type of teachers. Emotional exhaustion is one of the primary contributing factors to burnout, with emotionally exhausted teachers feeling fatigued and unable to engage with people. This subscale also assesses feelings of being emotionally overextended by one’s work, with higher scores indicating higher levels of exhaustion. Data collected for this study indicate low and average levels of emotional exhaustion across teachers teaching ALP, conventional, and both systems. 26 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Figure 5: Maslach Scale - Levels of Emotional Exhaustion Amongst Teachers 14.28 12.91 14.09 0 10 20 30 40 50 60 ALP (n=29) Conventional (n=87) ALP/Conventional (n=102) Figure 6 shows the depersonalization scale of the Maslach inventory, which measures impersonal responses, cynicism, and disengaged feelings towards one’s students. All teachers interviewed report consistently low levels of depersonalization, suggesting that they care about their students instead of feeling disengaged. Figure 6: Maslach Scale - Levels of Depersonalization Amongst Teachers Figure 7 shows the personal accomplishment scale, measuring feelings of competence with one’s work, and successful achievement with students. Teachers displayed relatively high levels of feelings of personal accomplishment. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 27 Figure 7: Maslach Scale - Levels of Personal Accomplishment Amongst Teachers 41.07 41.62 41.86 0 10 20 30 40 50 ALP (n=29) Conventional (n=87) ALP/Conventional (n=102) There are no major differences on the burnout scales between types of teachers. Teachers report relatively low levels of feelings of exhaustion and depersonalization, but also report average or low￾average levels of personal accomplishment. It is useful to consider these findings alongside teacher’s opinions and perceptions discussed further below. With changes in workload post-transition, teachers express that although they have adapted and are able to manage, the previous AQE structure was more conducive to teaching and student learning. Some schools have found ways to adapt to the changing workloads by implementing rotational teaching or adapting their curricula to accommodate all students. First, an individual teacher was tasked to a certain class that you do all the four courses. First, we were doing Mathematics, English, Science and Social Studies. If you are tasked to be with level 1 or level 2, you do all of the courses but this time around now, we are doing something like rotational thing. For me, I can teach the science from level 2 and level 3… it makes it easier. (Nimba County ALP Facilitator, KII) Teachers’ experiences vary depending on the availability of ALP workbooks and whether the ALP students are separated from conventional students. In one school in Grand Bassa county, a teacher reported that, since they do not currently have enough ALP workbooks for their students, they must develop the curriculum themselves: I research and get everything. So, I had to struggle and make research. Overnight I can't sleep, I have to make research from the curriculum. I have to get the material the children need. I am forced to get it for them…Since there is no material [ALP workbooks], it is easier now that I can make it. [The ALP curriculum] was easy, it was the easiest one. The change is you have to make your own research and prepare the notes… we are used to it now, it's not difficult again because we are used to it until if our material comes, then we know how to handle it. (Grand Bassa, ALP Facilitator, KII) 28 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Regardless, teachers who cover both the ALP curriculum and the conventional curriculum are still encumbered by an increased workload, without the encouragement of the stipends to support their efforts. One ALP teacher/facilitator expressed discouragement after the stipends were discontinued: [The MoE] should double their speed. They should come in with their stipend, we will need stipend and at least it should be little encouraging. They should make it little encouraging so that the teacher will not feel weak to come to work. Once the working man is encouraged or happy, he/she will be able to do the work professionally…More specifically, I'm a volunteer, so I'm feeling weak. (Montserrado County ALP Facilitator, KII) Parents and students also reported concerns about teachers’ attendance and capacity to teach following the transition, alluding to issues with teacher retention given the changes in workload and compensation. During a focus group discussion with parents in Bong County, several parents stated that when their “children go to campus, there is no instructor,” so they are forced to return home early. Students at another school in Bong County reported that their teacher is sometimes absent for long periods of time because he is fatigued from teaching two sessions (ALP and conventional classes) and splits his time between two schools. Students have noted changes in the quality of their instruction due to teachers experiencing fatigue from factors such as lack of compensation for additional workload and changing expectations after the transition. Students at a school in Lofa County exclaimed: The problem we are facing in the ALP some of our teachers said they can get tired, so our student among ourselves we can take, then we be reading in the book. We would be solving question from the book asking the other students to answer and answering some question in the book asking the other students to answer the other question in the book. (Lofa County, FGD with ALP Students) Students and teachers alike worry that the number of teachers presently available are not enough to ensure the sustainability of the program. There is a need to contract more teachers on government payroll, so that teachers are compensated for their work. In Bong County—one county where several schools are struggling with teacher attendance and retention—the PTA head and parents in the focus groups stated: Some of the things that make the parent too unhappy is that the government have some instructors who they downsized. Because, if you pension (retire) any worker, you have to employ [a replacement]. When pension is going on you have to employ [replacements], but the government put some people down, some volunteer teachers who use to help, said that “they too were not on payroll”, they left, they left the school, the teaching field. So, for that reason the school, the parents are finding it difficult to (stutters) send their children to school. (Bong County, PTA Head, KII) As for me I don’t feel fine, for my children to just go to school and come back because nobody is there to teach them, it doesn’t go down well with me really. It not supposed to be that way.’ (Bong County, Parents FGD) In some schools, the number of teachers is not sufficient to cover all levels, placing an additional burden on current teachers. Combined with the resource limitations, teachers struggle to adequately teach their students compared to the previous AQE structure: USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 29 The difficulty is that the children are many. First, they were not many, I say because I had I think about 17 last year that graduated from the ALP department from level 3 but now we have almost about 50 some more here and now no books, nothing and again nothing over there. The teaching aid, they are not giving us any other thing. I think they only came here and gave us guide, to guide the children and guide the teaching. (Nimba County ALP Facilitator, KII) They ask the ALP students in line with the conventional students to be able to pay at least the minimum amount to help the volunteer teachers because government was not able to employ teachers that could be able to teach the entire school System. (Lofa County, ALP Facilitator, KII) RECORD KEEPING Ninety-eight percent of ALP facilitators kept attendance records of ALP students, as Table 16 shows. Three-fourths of ALP teachers schedule time in their workday for lesson planning, with seventy-one percent of ALP teachers developing these plans daily or weekly. Teachers that did not develop lesson plans regularly explained that previously, they had lesson plans provided for them by USAID. Table 16. Attendance and lesson planning % Teachers who keep attendance records of students in ALP classes 98% Teachers who have scheduled time in the day for lesson planning 75% Teachers who develop lesson plans daily or weekly 71% Results also indicate that 95 percent of facilitators reported that they received guides. These teachers frequently utilized the guides to aid in lesson planning and preparation, as noted in Figure 8 below. Figure 8: Frequency of teacher utilization of guides (n=125) 92.8 3.2 4 Daily Twice a week Weekly 30 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG ASSESSMENT OF LEARNER PROGRESS AND LEARNING CERTIFICATIONS New students are required to sit for a placement assessment, unless they have never been in school, in which case they are automatically assigned to level 1. Enrolled students are also required to sit for end￾of-unit and end-of year or completion assessments. Students need to pass these assessments to progress to the next level. NORC asked principals of schools with ALP students how many new students sat for the placement assessment (excluding students that have never attended school), and how many students sat for end-of￾unit and completion assessments. Figure 9 shows the findings. Thirty-six percent of principals said that they have all or most of their new students sit for a placement assessment, 16 percent said about half of new students sit for a placement assessment and 48 percent said that either a few or none do. In contrast, end-of-unit and completion assessments are more prevalent. In the case of end-of-unit assessments, 70 percent of principals said that all or most students sit for an end-of-unit assessment, 19 percent said about that about half of students do so, and only 11 percent said that either a few or none do. Prevalence for completion assessments is a bit higher than for the end-of unit-assessments. Figure 9: Assessment prevalence according to teachers (%) (N=73) 36 16 48 70 19 11 71 25 4 All/most About half A few/none Placement (new students) End-of Unit Completion Note: Sample size for completion assessment is 72 due to item-specific missing data. The survey questions provided respondents with a 7-point scale but for ease of exposition we collapsed the seven response options into three. It is perhaps not surprising that placement assessments are not as prevalent as completion or end-of￾unit assessments. It is possible that even if students have been in school before, it was too long ago or they never passed a grade, so in many cases, principals may place students in level 1 without a placement assessment. End-of-unit and completion assessments, on the other hand, are high stakes assessments that determine level progress. TEACHING AND LEARNING MATERIALS While this transition evaluation was not designed to assess the quality of the teaching materials, several participants commented on the good quality of certain teaching modules, including an EDC staff member USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 31 who indicated that the modules for levels one and two offer particularly strong literacy and numeracy teaching, as they have been tested multiple times. Similarly, a local education official with supervisorial functions stated that the workbooks are “very good, very good, these materials are used by our instructors in the classrooms…I know exactly what the school supposed to be doing at what time and then for what level.” (Nimba County, Local Education Official, KII) However, despite respondents praising the quality of the materials, the insufficiency of their distribution was a major challenge for teachers/facilitators and for students. SHORTAGE OF MATERIALS AFTER THE TRANSITION When asked about the impact of the transition to MoE oversight, the single most frequent complaint from respondents in KIIs and FGDs was the shortage of materials provided to ALP teachers and learners. Quantitative data confirms that students do not have appropriate learning resources in a significant proportion of schools. As Figure 10 shows, 35 percent of teachers said that all ALP students have their own workbook, 14.5 percent said that most students do, 11 percent said that about half the students have a workbook, 14.5 percent said fewer than half and 25 percent said that none of the students have a workbook. Lofa is the county where students are least likely to have their own workbook, according to the surveyed teachers. Figure 10. Percentage of teachers who report that students have their own workbook 35.1% 15% 10.7% 15% 25.1% 0 5 10 15 20 25 30 35 40 All of them Most of them About half of them Less than half of them None of them EDC previously provided teacher guides, and most importantly, they supplied individual learner workbooks for each student, as well as other teaching materials like pencils, posters, and teaching aids to the schools. Now, as one principal described, all material support seems to have been cut off: The difficulties we have is that the materials are not sufficient with the current [ALP model]. The NGO, used to bring us materials that has to do with the children’s books, even pencils and 32 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG copybooks that we will use, readers. But [after the transition] we don’t have any of it, we only managing [with] what we got last year that’s left on our shelf... (Margibi County Principal, KII) In particular, students and teachers explained that without individual student workbooks for ALP learners, the quality of their education declined, including their academic progress and participation in classrooms. Without student workbooks, teachers had to write out exercises on chalkboards for students to copy, which took extra time and made classroom activities less interactive. In addition, student workbooks were important for homework assignments, and their absence resulted in students being unable to do work at home. The difficulties of not receiving workbooks were described by many respondents in ALP schools, such as this teacher: Mainly the children’s learner workbook, that’s where I really got a problem because it’s tedious. While we on the lesson, they [are] supposed to be going along with you... What they learn in the school, what you taught them, when they [go] home, they still [see] it in their learner workbook and [do] their homework. (Montserrado County ALP Teacher, KII) One principal explained: The teacher guide will go along with the learners’ workbook because those lesson that are in the teacher guide, as you [are] teaching, you tell the learner to refer to page this, page that, but now is not so because we don’t have learner workbook. (Bong County Principal, KII) This was also echoed in FGDs; in a focus group discussion with parents in Montserrado county, participants complained about the shortage of learner workbooks and how this prevents their children from studying at home. Additionally, in most of the focus groups with students, when they were asked to compare the ALP program before and after the transition, they discussed how books were no longer being provided, and how difficult it was to share a limited number of workbooks in groups. Yet, one Ministry of Education official when praising the plans for sustainability of this program, stated that AQE had “turned over a lot of teaching materials to us, both in soft copy and hard copies…so we learn a lesson now…from the onset, we start talking sustainability.” (MoE Official, KII) 2.3. AQE’S STRATEGIES TO ENSURE SUSTAINABILITY AND INSTITUTIONALIZATION INCLUDING CURRICULUM, AQE MODELS, AND TEACHER TRAINING APPROACHES. SUCCESS FACTORS FOR MINISTRY OF EDUCATION CAPACITY The Ministry of Education saw great success leveraging their technical experts for curriculum designs and revisions, so that the ALP curriculum is well-aligned with the conventional curriculum. One effective strategy that the Ministry of Education has used to plan ALP implementation is establishing a Technical Working Group drawn from several departments in the Ministry, including Gender, Health and Safety. This Technical Working Group includes people from the Bureau of Basic and Secondary Education, the Bureau of Planning, and other related bureaus. Within the Technical Working Group, there are groups that focus on planning, on girls’ education, on community engagement, on monitoring and evaluation, and on guidance and counseling. An official from the Ministry of Education explained: USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 33 We knew that this project had implications for all other bureaus, and we thought that the way to work is to form a holistic team, so that everybody is talking the same thing. (MoE Official, KII) CHALLENGES FOR MINISTRY OF EDUCATION CAPACITY Qualitative findings suggest that the MoE has made substantial efforts to transfer the program to the current single-classroom integrated format. An EDC representative also explained that to transition the program leadership to the MoE, EDC’s aim was to “ensure that we did it together with the Ministry of Education officials, we took them alongside us.’ While the MoE made efforts to foster a smooth handover from EDC, there can be little doubt that the loss of the previous teacher stipend had substantial effects on staffing. As a senior advisor observed: “A lot of [schools] fell off because of the stipend among, other things”. (EDC Staff, KII) It also appears that many of the ALP staffing losses during after-school hours were ‘volunteer teachers’, who left when they learned they would not be compensated. In response these staffing losses, as indicated previously, the MoE strategized to keep the ALP program going by merging classes and using teachers who were on the payroll. But, understandably, even these teachers remained frustrated at losing the stipend for their afternoon teaching. Many respondents discussed the Ministry of Education’s limited resources as the primary barrier to implementation moving forward. One senior staff member at EDC suggested that strategies and tools have been put in place for the MoE to manage and monitor the program, while also recognizing the likely limitations, explaining: They have the toolkit; they have the framework. If they have the books, they could implement it. Now, I want to be frank, maybe not to the level of quality that we are looking for but remember this is the ministry taking over at scale. So, we are not looking for that quality; but we are looking for the highest quality possible. But we are looking for a sense of ownership and we can do this. (EDC staff, KII) A local education official also expressed approval of the program, questioning why such a good program was not rolled out in more schools and suggesting the importance of sufficient government leadership and funding: You have a government that is supposed to be responsible for this, so try to engage your leaders to see a budgetary allotment is made for the improvement of the program. It can’t be just the USA is doing everything… (Nimba County, Local Education Official, KII) At the same time, a MoE official noted that the Ministry has “limited resources” and “competing priorities”, which makes it difficult to concentrate on the ALP program compared to the attention it received when USAID was involved. This official added that there were hopes that they might be able to attract further resources, perhaps by speaking to USAID again. Respondents indicated that this was not just an issue of total funds, but also an issue of financial administration capacity. An EDC staff member explained that allocating funds to educational programs like ALP face obstacles related to administration and the centralization of government structures: 34 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG In the institutional capacity assessments are being carried out every year was the fact that the financial domain was always one of the weakest. And it was one of the weakest not because the counties didn’t have financial functionaries there, but because they had no budget, or because the budget was centrally controlled, and they didn’t necessarily have that decentralized capacity to deal with that. (EDC Official, KII) SCALABILITY, REPLICABILITY, AND THE FUTURE OF THE ALP PROGRAM The ALP model has been well-received by participants, and the quality of the curriculum was praised by respondents in the focus group discussions and interviews for this study. Most of the problems that have surfaced with this program during the transition period relate to constraints on funding and insufficient teaching and learning materials, not with the quality of the AQE model itself. Thus, respondents are keen to find ways to scale up the model moving forward, especially given the negative impact that the COVID-19 pandemic has had on student enrollment. An EDC official spoke of plans to scale up the ALP model in other counties of Liberia: We are going to scale this up. And we are going to take what we learnt from AQE and the model from AQE and we are going to bring it to other counties […] you had the sense among the County Education Officers who were representing the counties and so on, [of] an articulation of the positive energies that they have developed […] from 12 pilot schools to 190 schools that's not a bad scale up, even though we've conducted functionality assessments which show that some schools are struggling and others are satisfactory, but they are implementing. (EDC Official, KII) An official from the MoE discussed plans to find more funding to continue and scale up the program, and praised the quality of the model: The AQE did extremely well […] the government is even looking for partners and donors and making use of our little domestic resources to be able to ensure the transition of most of the out of school kids into [conventional school], reaching the AQE program to conventional schools [that] match their different age ranges. (Ministry of Education Official, KII) This issue of funding is at the core of the scalability of the ALP program. One respondent from USAID cautioned that it will be difficult to establish the ALP program in new schools as it takes more resources to start the program in new sites than to continue it: Without a really strong, robust infusion of resources and training I think that’s a challenge for the ministries to think about as well what can they do to recreate that level of investment early on to set a foundation for the school to create the one school approach that is appropriate for them. (USAID Officer, KII) An idea proposed by an EDC official to cut down on the costs of the program is to redesign the teacher guides and student workbooks for a 3–5-year reusable context. They explained that yearly printing of these materials is too costly for the MoE to take on moving forward, but it is more feasible to pay these costs at 3- or 5-year intervals. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 35 One under-leveraged resource that could help support the scale-up of the ALP program is parents and PTAs. Several respondents called for more systematic coordination between local education officials and PTAs. One EDC official explained the enthusiasm and support for ALP from parents: Listening to the community, to the PTA’s feedback, they really have ownership of the ALP […] ALP had brought changes into the community, they really like it. So based on that, on the feedback I heard from the communities, the module can be repeated. […] parents are all really wishing that ALP continues, not in those selective, selected schools, but ALP should be a country wide, because they have seen the benefit of it. (EDC Official, KII) This being said, the program cannot depend on PTAs to provide compensation for teachers moving forward. A local education official in Nimba county made the case that the only sustainable funding of the program must come from the government: The only promising future is budgetary allotment for all the program at the national level. Once somebody is on payroll and being paid as accelerated learning program instructors, that sustains the program. […] We are not there yet at the community level to be able to support the program. I won’t want to lie to anybody. (Nimba County, Local Education Official, KII) Another factor for the scalability of the ALP is technology. The MoE is moving forward with plans to expand teaching by radio and television, even to the extent of owning their own stations, and they also discussed using video-calls for remote monitoring to cut down on transportation costs for local education officials. However, all these strategies, while effective, are constrained by inconsistent or absent electricity in some rural communities in Liberia. Thus, leaning too far into these technology-based solutions will lead to uneven benefits of the ALP program, excluding students in rural households. 2.4. WHAT IS THE CONTRIBUTION OF AQE TO EQUITABLE TRANSITION TO FORMAL EDUCATION, RETENTION, AND COMPLETION IN LIBERIA? HOW DID THE AGE, GENDER, LOCATION, OF A LEARNER AFFECT THEIR TRAJECTORY? In this section NORC discusses retention and other indicators of the ALP program. First, administrative data is used to discuss results between 2017-18 and 2019-20, when the AQE program was in place, and second, data from the follow-up survey is used to discuss results between 2019-20 and 2020-21, when the One-School model was in place. ADMINISTRATIVE DATA – AQE MODEL NORC used EDC administrative data to construct a panel of AQE students during the three years that the program was in place. Annex V describes how this dataset was constructed. A total of 47,670 students were enrolled between school years 2017-18 and 2019-20.9 Table 17 shows that 38,901 students were enrolled for one year only, 8,145 were enrolled for two years, and 624 were enrolled for all three years. 9 Each year, but especially in 2020, there were students that sat for the completion assessments but do not show up in the enrollment data. As we explain in Annex V, these students are not considered in the analysis. 36 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Table 17. AQE students, by years of enrollment STUDENTS One year 38901 Two years 8145 Three years 624 Total 47670 Table 18 shows student characteristics. Females represent roughly half of students, although their participation declined a bit in school year 2019-2020. Ages are roughly uniformly distributed up to age 15, especially at the beginning of the program. About a third of students attended a school located in an urban area. In the first year, almost all students were attending ALP level 1; over the years the participation of students in level 2 and 3 grew, although even by 2019-2020 the fraction of students in level 3 was just 7 percent; furthermore, as late as 2019-2020 still two thirds of ALP students were in level 1. This includes cohort 2 and 3 schools, where naturally level 1 students should be the majority, but even if analysis is restricted to cohort 1 schools only (not shown), the figure does not change much. This means that even after two years of having the ALP program running, most ALP students are new children enrolling in level 1. Table 18. Students’ characteristics 2017-18 2018-19 2019-20 Female 0.49 0.49 0.47 Age (years) 6-9 0.31 0.19 0.20 10-12 0.36 0.37 0.41 13-15 0.32 0.39 0.35 16-19 0.00 0.05 0.05 School located in urban area 0.35 0.29 0.24 ALP Level Level 1 0.97 0.82 0.67 Level 2 0.03 0.14 0.26 Level 3 0.00 0.04 0.07 N 8599 21389 27075 Using these data, findings are organized into three outcomes of interest: assessment completion, assessment approval, and next year retention. The results are shown in Table 19. Completion refers to the number of students enrolled in a given year that sat for the completion assessment. Approval (pass) corresponds to the number of students that passed the completion assessment. Retention refers to the USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 37 number of students enrolled in a given year that were also enrolled in the following year. In 2017-2018, 23 percent of the AQE students sat for the completion assessment, 8 percent passed it, and 23 percent were retained. These indicators improved in the following years. The results for 2018-2019 show that 46 percent of the AQE students sat for the completion assessment, 32 percent passed it, and 35 percent were retained into the following year. Finally, in 2019-2020, 67 percent of students sat for the completion assessment and 46 percent pass the assessment. The administrative data does not include information about the number of students that were retained in 2020-2021 but presented below are the percentages calculated from follow-up survey data. Table 19. Completion, passing assessments, and retention for AQE students COUNTS AS % OF ENROLLED 2017-18 (101 schools) Enrolled 8,599 100 Completion 2,020 23 Passed 647 8 Retained next year 1,946 23 2018-19 (198 schools) Enrolled 21,389 100 Completion 9,924 46 Passed 6,871 32 Retained next year 7,447 35 2019-20 (263 schools) Enrolled 27,075 100 Completion 18,056 67 Passed 12,411 46 To analyze how students’ characteristics are correlated with the outcomes of interest, estimated linear probability models were used. Table 20 shows the results. The first column shows the association between student’s characteristics and ALP level completion. The second column shows results for passing the assessment; only students that sat for the assessment are included in this analysis. The third column shows the correlation between students’ characteristics and the probability of being re-enrolled the following year; in this case only students in levels 1 or 2 are included, for school years 2017-18 and 2018-19. The results indicate that females are 1.1 percentage points (pp) more likely to sit for the completion assessment than males, and although the difference is statistically significant, it is small relative to the average completion rate of 52 percent, shown at the bottom of the table. There is no statistically significant difference in passing rates by gender. In terms of retention, females are 1.3 pp less likely to be enrolled the following year, again the difference is statistically significant but small relative to the mean of the dependent variable of 31.3 percent. 38 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Table 20. Correlations between students’ characteristics and completion, passing and retention rates COMPLETION ASSESSMENT PASSING ASSESSMENT(A) NEXT YEAR RETENTION(B) Female 0.011* -0.003 -0.013* (0.00) (0.01) (0.01) Age (reference category: 6-9 years) 10-12 0.096*** 0.045*** 0.034*** (0.01) (0.01) (0.01) 13-15 0.047*** 0.097*** -0.028* (0.01) (0.01) (0.01) 16-19 0.330*** 0.090*** 0.018 (0.04) (0.02) (0.04) School located in urban area -0.044* 0.026 -0.063** (0.02) (0.03) (0.02) School year (reference category: 2017-2018) 2018-19 0.173*** 0.360*** 0.116*** (0.02) (0.04) (0.02) 2019-20 0.373*** 0.329*** N/A (0.02) (0.03) ALP level (reference category: Level 1) Level 2 0.142*** 0.036** 0.060*** (0.01) (0.01) (0.01) Level 3 0.247*** 0.085*** N/A (0.02) (0.02) N 57063 29694 29059 Dep. Variable mean 0.520 0.670 0.313 (a) Conditioned having sat for the assessment (b) Conditioned on levels 1 and 2 and school years 2017-18 and 2018-19 Standard errors clustered at the school level in parentheses. * p<0.05 ** p<0.01 *** p<0.001 Correlations were estimated between age ranges and the outcomes of interest. The left-out category comprises students between 6-9, so the results are compared to that age range. This means that, for example, students who are 10-12 years old are 9.6 percentage points (pp) more likely to have sat for the completion assessment than students who are 6-9 years old. The results indicate that age is USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 39 positively correlated with both sitting for the completion assessment and passing it, as all the coefficients for age for these two outcomes are positive and statistically significant. However, the results for retention are less univocal. We find that students who are 10-12 years old are 3.4 pp more likely to enroll the next year than students who are 6-9 years old, but students who are between 13 and 15 years old are 2.8 pp less likely to be enrolled for the next year than students who are 6-9 years old, and there is no difference between students who are 16-19 years old and students who are 6-9 years old. This suggests that, while older students may be more likely to write and pass the completion assessment than younger students, this advantage is not reflected in the probability of staying in the ALP system. There are a couple of factors that could explain this. First, it is possible that some students transitioned to conventional primary schools, so the analysis underestimates survival in the school system as a whole. This could explain why students who are 10-12 years old are more likely to be enrolled in ALP the next year than students who are 6-9 years old, as the latter are younger and could have transitioned to conventional primary schools after passing the level 1 completion assessment. However, it is unlikely that this is the reason why students who are 13-15 years old are less likely to enroll in ALP the next year than students who are 6- 9 years old, as the former are not good candidates to transition to primary school due to their age. A second potential explanation is that older students face higher opportunity costs of staying in school, and therefore are more likely to drop out of school altogether than younger students, even if they pass the completion assessment. In terms of location, students in urban schools are less likely to sit for the completion assessment and less likely to be enrolled the following year. Both results could be reflecting the better job opportunities in urban versus rural areas, so the opportunity cost of staying in school is higher for students in urban areas than in rural areas. The analysis also includes indicators for school year. The results show that the program improved over time in terms of completion, pass and retention rates. Relative to the results in 2017-18, completion rates are higher by 17.3 pp in 2018-19 and 37.3 pp in 2019-20, pass rates are higher by 36 pp in 2018-19 and 32.9 pp in 2019-20, and retention rates are higher by 6 pp in 2018-19 relative to 2017-18. This likely reflects improvements in program implementation, especially between 2017-18 and subsequent years. According to EDC staff, the 2017-18 completion and passing rates were low partly because in 2017-18 instruction time was less than a semester due to activity start up delays, while in 2018-19 instruction covered the full school year. In addition, other AQE interventions significantly increased in 2019. Finally, formative assessments, introduced in 2019, seem to have played a key role in increasing learner participation.10 Finally, analysis reveals that the probability of completing the assessment and passing it increases as students progress in the ALP levels. Probably the less endowed children (worse socioeconomic backgrounds and/or low ability) are unlikely to pass level 1, so those that do are more likely to sit for the completion assessment, pass it, and enroll the following year in the ALP program. 10 Source: Email from Apollo Nkwake, September 13, 2021. 40 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG FOLLOW-UP SURVEY – ONE SCHOOL MODEL NORC conducted a follow-up survey on a sample of 900 students at the end of school year 2020-2021 to document their enrollment status. The sample frame includes the students that started the ALP program during the school year 2019-20, and the students that sat for the 2018-19 end-of-year or completion assessment.11 Annex VI discusses how the sample frame was constructed as well as the weighting scheme used. Table 21 shows the distribution of students by ALP level in 2019-2020, disaggregating between the list of new students in 2019-2020 (placement), and the list of students that sat for an end-of-year assessment in 2018-2019 (completion).12 Table 17 shows that all placement students are in Levels 1 or 2 (so no students started ALP in level 3 in 2019-2020). As for completion students, about one-fourth were supposed to enroll in Level 1 (because they failed their level 1 completion assessment in 2018-2019), about 60 percent were supposed to enroll in Level 2, about 10 percent were supposed to enroll in Level 3, and less than 3 percent were classified as graduates, meaning that they passed their Level 3 completion assessment in 2018-2019. Only six percent of the 900 students were classified as Level 3 or graduate. Table 21. Surveyed students by level in 2019-2020 PLACEMENT COMPLETION TOTAL Level 1 416 102 518 Level 2 88 240 327 Level 3 0 43 43 Graduate 0 11 11 Total 504 396 900 Table 22 shows descriptive statistics for the sample of students by ALP level. Females make up roughly half of students across all levels. Students in Level 1 are 11.6 years old on average, and students in higher levels are between 13 and 14 years old, on average. About a fourth of the sample attends a school in an urban area. Table 22. Surveyed students’ descriptive statistics by level LEVEL 1 LEVEL 2 LEVEL 3 GRADUATE ALL Female 0.47 0.49 0.53 0.48 0.48 Age in 2019-2020 11.6 13.1 13.7 13.3 12.2 Urban 0.24 0.26 0.20 0.32 0.25 NORC reviewed school rosters to verify the enrollment status of this sample of students in July 2021. For students that were supposed to be in Level 3 or were graduates in 2019-2020 that were not found in school rosters, parents were interviewed. The objective of these parent surveys was to determine if 11 As an alternative we could have used the list of students that sat for the completion assessment in 2019-20, which includes a more recent account of children attending the ALP. The problem is that this list does not have student id numbers, so we could not link these data to the enrollment data to recover students’ gender and age. Because we wanted to analyze how gender and age are correlated with enrollment status in the follow-ups survey, we did not use these data as sample frame for the follow-up survey. 12 This distinction is important because “placement” students, who we know were enrolled at the beginning of 2019-2020, are perhaps more likely to be enrolled in 2020-2021 than “completion” students, who we last observed by the end of 2018-2019. In effect, we don’t know if completion students even enrolled in 2019-2020, and the level they are assigned to in Table 17 comes from the level they were supposed to enroll in 2019-2020 given their results in the 2018-2019 completion assessment. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 41 these former ALP students had graduated or not, were attending secondary school, or what they were doing. Table 23 shows the enrollment status of the sampled students. The sample is divided between students in levels 1 and 2, and students in level 3 and graduates. This division was made because it is natural to expect that students in level 1 and 2 in 2019-20 would continue in ALP or primary school in 2020-21, while students in level 3 and graduates are less likely to stay in these levels, even if they do not drop out of school. The results are weighted using sampling weights, so they represent the results for the larger population of students in the sample frame. It was not possible to determine the enrollment status of 10 Level 3 and graduate students.13 For students in level 1 and 2, 21 percent transitioned to conventional primary school and 45 percent are still in ALP, for a total of 66 percent still enrolled in school. For students in level 3 and graduates, 47 percent are in primary school, 23 percent are still in ALP, and a negligible fraction transitioned to secondary school, for a total of 70 percent still enrolled in school. Notably, 25 percent of level 3 and graduate students are not enrolled in school but graduated from primary school (or ALP), while only 3 percent are not in school and did not graduate. Table 23. Enrollment status in 2020-2021 by ALP level in 2019-2020 ENROLLED OUT-OF-SCHOOL PRIMARY ALP POST￾PRIMARY TOTAL PRIMARY/ALP GRADUATE? TOTAL YES NO Levels 1 and 2 (N=845) 0.21 0.45 N/A 0.66 N/A 0.34 0.34 Levels 3 and graduates (N=44) 0.47 0.23 0.03 0.70 0.25 0.03 0.28 Total (N=890) 0.22 0.43 0.00 0.66 0.01 0.33 0.34 To understand the ALP’s implications on equity, the correlations between enrollment and students’ gender, age, and school location (urban/rural) were analyzed. Regressions were used to simplify the analysis where the outcome variable is an indicator for whether the student is enrolled, regardless of whether s/he is enrolled in primary, ALP or post-secondary education, and the explanatory variables are the students’ gender, age, and school location (urban/rural). Note that this outcome is different than the enrollment one using administrative data (third column in Table 4), because with administrative data it is unknown if the students that are not enrolled in ALP transitioned to conventional school or dropout of school altogether, while with the survey data we do. Table 24 shows the results. In the first column students from all grades are pooled, in the second column only children that were in levels 1 or 2 in 2019-20 are included, and in the third column only children that were in level 3 or graduates in 2019-20 are included. The analysis did not show differences by gender for in group. Age seems to be positively but relatively weakly related with enrollment. Students between the ages of 10 and 12 are more likely to be enrolled in school than students who are 6-9 years old. Results for ages 13-15 and 16-19 are also positive, which means that they are more likely to be enrolled in schools than students who are 6-9 years old, but the parameters are not always statistically significant. Finally, there is no correlation between school location and enrollment, except for children in level 3 or graduates; findings show a 13 We only interviewed parents of students that we could not find in the rosters AND that were in Level 3 or graduates in 2019-2020. Therefore, any student that was in level 1 or 2 that we could not find in the rosters is classified as dropout with less education than primary or ALP. 42 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG negative and statistically significant correlation between schools located in urban areas and enrollment for that group. Table 24. Determinants of school enrollment ALL ALP LEVEL IN 2019-20 1-2 3 & GRADUATES Female 0.074 0.079 -0.018 (0.05) (0.05) (0.10) Age (reference category: 6-9 years) 10-12 0.218* 0.208* 0.302** (0.09) (0.09) (0.10) 13-15 0.166* 0.161 0.031 (0.08) (0.08) (0.14) 16-19 0.115 0.041 0.240* (0.13) (0.15) (0.10) School located in urban area 0.003 0.016 -0.338** (0.13) (0.13) (0.12) N 861 700 161 Note: Sample sizes are unweighted which is why they are different from what is shown in Table 23 Missing data on the outcome variable is imputed. Standard errors clustered at the school level in parentheses. * p<0.05 ** p<0.01 *** p<0.001 MARGINALIZED STUDENTS AND EQUITY CONSIDERATIONS Interviews and focus group discussions offered additional findings on the marginalization or exclusion of students from the ALP program because of their age, gender, location, or household income level. Additionally, there was no specific mention of this in the interviews or focus group discussions, but students with a disability tended to be excluded because of the limited seating capacity, i.e., shortage of chairs when the conventional and ALP classes were combined. In terms of age, it was clear that overage students felt stigmatized, which made them reluctant to participate in the ALP program. A principal in Grand Bassa County described this reluctance: The most difficult part of this program is to have the children coming every day because some of them are overgrown and don't really want to come to school, so we need to encourage them. (Grand Bassa County, Principal, KII) Female overaged students were marginalized and more likely to drop out because of community norms around marriage. One ALP teacher in Bong County explained, It [The ALP program] cannot delay the children to be grown up […] some of these girls, some of our children that are here, some of them you see they are 10 years, 15 years, 20 years before USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 43 you get to start them with ABC, sometimes some of them will not go to school, because as soon as they reach a certain level then they marry. (Bong County, ALP Teacher, KII) The location of students affected their ability to participate in the ALP program in two primary ways. During the pandemic, students who lived far away from schools were not able to travel to pick up materials as frequently as their peers, slowing their academic progress. Homework was especially difficult for these students if they also did not have their own workbooks and thus could not complete exercises on their own. Poverty was also a substantial factor in student enrollment, especially once fees were imposed after the transition. In a focus group with students in Bong County, they said: “Our parents them not get money, so we can't bring uniform […] We are suffering. [There is] no money for our parents them to buy, to pay the school fee.” (Bong County, ALP Students, FGD) One principal in Margibi county explained that even relatively low fees led to drops in enrollment for households living in poverty: Parents don’t have the money to pay the fees. Government says the children should pay 1000 dollars [$5.85 in USD] for registration to enter and that government school here, to get the 1000 dollars self, some parents can’t even afford it, then they got to buy the materials, so some parents can’t afford that. So, some of the children drop. (Margibi County, Principal, KII) It is also important to note that distance learning strategies employed during the pandemic that rely on radio or television can exclude children in communities that don’t have equipment or electricity. CHANGES IN COSTS OF ENROLLMENT IN ALP The transition has placed a greater economic burden on parents of overage students because of the loss of resources previously provided under the EDC-financed program. Prior to the transition, ALP students were given learning materials and could attend school at no cost. In Montserrado County, a PTA chair explained how this impacted the program: At present, the community, there are a lot of kids here in the community that should still be benefiting from that program. But because the program closed, most of them now are in the regular (Inaudible) government school. … some kids’ parents don’t even have the money to send them to school. (Montserrado County, Community Leader, KII) Across counties, findings indicate that enrollment and student retention was impacted by the burden of school fees following the transition to the Ministry of Education, as one principal explained the reduced number of students: Some of the parents say they don’t able to pay the PTA fees, they say the PTA fee is too much because last year when they came, they said free and we received lots of students but now after they left, they say we don’t have the compensation, the stipend they have for teacher they are no longer, so the parent should take the burden. So, you see most of the parent withdraw their children. (Lofa County Principal, KII). 44 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 2.5. WHAT STRATEGIES FOR AQE DELIVERY COULD BE USED TO SERVE LEARNERS AFFECTED BY AN EMERGENCY LIKE COVID? There is broad consensus about the negative impact that extensive school closures during COVID-19 had on learning. However, there are very few studies that measured the actual magnitude of learning losses, particularly in developing countries. Some studies using data from the pre COVID-19 period (Angrist et al., 2021, Kaffenberger, 2021, Gustafsson and Nuga, 2020, Soudien et al., 2021) estimate the possible learning losses. In contrast, Ardington, Wills, and Kotze (2021) calculated the actual reading learning losses in poor public schools in South Africa using longitudinal data. They estimated reading losses between 57 and 70 percent of a school year among grade 2 students and between 62 and 81 percent of a school year among grade 4 students. In Liberia the post-COVID-19 reading performance of grade 3 students after around three months of classes is at the level of the pre-pandemic grade 2 students towards the end of the school year (Menendez, Solovyeva and Hoadley, 2021). In addition, the number of students enrolled in the early primary grades declined compared to pre-COVID-19 enrolment (USAID 2021 and NORC-TKG 2021). Previous research, in Sierra Leone and Guinea for example, indicates that the most disadvantaged children are disproportionally affected and tend to be the most likely to dropout (Smith, 2021). This implies that our estimate of the average learner performance after COVID-19 is likely to be biased upwards. Could the AQE approach help students affected by COVID-19 or a similar crisis to recover? This study explored two ways in which accelerated education is potentially a good strategy. The first way considers if the AQE approach could be an alternative to help students overcome the learning losses caused by the extended school closures. The second way explores whether strategies employed by AQE to reach students during the COVID-19 school closure period could be helpful to reach students in similar situations. First, presented here is an analysis of whether an accelerated education program such as AQE could be used after a prolonged crisis that prevents normal school activities from offsetting learning losses. Quantitative and qualitative data indicate that teachers think a program like AQE could help students recover lost learning. Almost all -99 percent- of the teachers interviewed think that the AQE curriculum provides a good alternative to make up for COVID-related learning loss. Educators also strongly believed that the ALP curriculum was a good alternative for conventional schools to make up for this discrepancy in learning. ALP facilitators stated that the AQE curriculum simplifies both teaching and student learning, which is beneficial to conventional students as well as ALP students: They [the ALP curriculum] break it down to the level of the student. Like for instance if you are teaching measurement for example if you look at the ALP curriculum, the teacher guides that we are using now, the way they will teach you about measurement even the little one will understand what really they are talking about. But in the conventional school the same measurement you have to go and take another paper, you plan your lesson on it, then you come in the classroom and present it to them. But measurement in the ALP classroom they have their book with them and you using your teacher's guide and you are taught one on one, and they will enjoy teaching it. They enjoy the teaching that we do with them in the ALP teacher guide. (Montserrado County ALP Facilitator, KII) USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 45 However, using accelerated education to bring the students up to speed is not straight forward. In general, accelerated education programs have three critical components: i) a condensed curriculum; ii) interactive and learner-centered pedagogy; and iii) longer instruction time plus music/arts/sports. The condensed or compressed curriculum exists, and teachers have a positive attitude towards using it. However, the curriculum was created for over-age out-of-school children. As shown in Table 7, the average Liberian student in conventional schools is over-aged but there is a broad distribution of ages. There are some students of the expected age attending early grades of school for whom accelerated education may not be appropriate. Furthermore, students were in school before the COVID-19 related closures, and afterwards a fraction stayed engaged, and many returned to school once classes resumed. This is a different population from the out-of-school group; these students had some level of knowledge before schools closed, and although part of it could have been lost, this difference needs to be considered. Additionally, only AQE facilitators are trained in the appropriate pedagogy to deliver the accelerated education curriculum. Those facilitators found the training important and useful; using the AQE curriculum more broadly would require massive training of teachers. It is possible that at least part of the AQE curriculum could be used but it will require adaptation to use it in a catch-up program to help students recover the losses in knowledge and skills. The effects of COVID-19 are likely to be different across schools, and even across students enrolled in the same classroom. For example, principals interviewed for the Read Liberia Impact Evaluation conducted by NORC14 in 2021 report that the grade 2 teacher or someone else at the school sent homework to grade 2 students in around 90 percent of the schools. However just over half of the students reported receiving any homework from the schools. One-third of the grade 2 students in the sample reported never studying or doing schoolwork while schools were closed. Another third indicated studying rarely and the rest reported studying and doing schoolwork often or always during the schools’ closure period. This heterogeneity is a challenge for designing a catch-up strategy useful for all. Ideally, one would want teachers to be able assess and decide on the remediation strategy based on the situation in their classrooms. However, there is evidence that many teachers do not follow strong assessment practices (Menendez et al, 2021) and have deficiencies in content knowledge (MoE, Republic of Liberia, 2018) and therefore a differentiation approach would be very difficult without strong external support. What can be learned from the work of the ALP facilitators during the school closures? ALP facilitators used a variety of means to ensure the progression of learning during the COVID-19 pandemic (Table 25). One-fifth of facilitators surveyed conducted classes online, while others taught students in smaller groups. Homework was also assigned consistently by facilitators during the pandemic. To make up for the learning losses incurred by students during the COVID-19 pandemic, ALP facilitators scheduled additional learning hours once schools reopened, sometimes on weekends. An accelerated program could be a helpful addition to such extra time. The literature suggests that summer and after-school learning programs tend to be associated with learning gains. Unfortunately, in Liberia classes were not only unavailable during holidays or after hours, but many schools were closed several weeks before the end of the 2020-21 academic year. 14 The sample is representative of public schools in the districts of Bong, Grand Bassa, Lofa, Margibi, Montserrado, and Nimba, that had grade one and grade two classes and at least 20 students in the grade 2 class in 2017. 46 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Table 25. Teachers’ strategies to ensure learning continuation ALP CLASSES PERCENTAGE OF TEACHERS N Before COVID-19 related closure Taught classes online 20% 217 Taught class individually/in small groups 68% 217 Assigned homework 94% 217 After COVID-19 related closure Scheduled additional hours 56% 217 Thought ALP curriculum provided a good alternative to make up for COVID-related learning loss 99% 211 Note: Sample size <218 due to item-specific missing data. During the COVID-19 pandemic, teachers and ALP facilitators in most schools reported giving their students materials to study at home, but in a minority of schools, students were not able to continue their lessons at all. In the focus group with students from Bong County, students reported that during the pandemic, “We were just playing because they ain't give us nothing to study.” (ALP Students, Bong County, FGD) In spring of 2020, some schools started using a “library in a box” strategy to help improve student access to materials so that they could study on their own time. Most schools were able to successfully organize a process through which students studied at home, then they or their caregivers dropped off assignments and received new materials from teachers while social distancing, thus allowing lessons and assessments to continue. However, even within focus groups from the same school, only some students were able to participate in these processes and access materials. Participation depended on how far students lived from their schools, and the availability of their parents or caregivers to pick up materials for them. A principal stated: We get town criers to call those ALP students since indeed, people are not coming, we want parents, we don't want children to come and pack up together, let parents come and receive their children's work and carry it. […] some parents actually came, and they cooperated with us but some parents are living far out and do not turn up anyway. (Nimba County Principal, KII) Additionally, several respondents noted that if there were low rates of literacy among parents or community members, students often struggled to complete their assignments at home because they had no one to help them. An ALP teacher in Lofa County said that once schools reopened, they had to add extra sessions on Saturdays to help students catch up after the interruption of the pandemic. At the level of the Ministry of Education, one of the key strategies to mitigate the consequences of school shutdowns was teaching by radio. This strategy was used to reach ALP and conventional school students and had mixed success, but it catalyzed plans in the Ministry to purchase their own radio station that would cover the entire country and regularly play different lessons, plus a television program that would cover the early childhood levels for children who might have trouble following radio lessons. The MoE used the AQE curriculum as a tool to condense the topics and prepared over 900 lessons. These strategies are ready for other future crises, as described by a respondent from the Ministry of Education: USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 47 The Ministry of Education owning this program, owning its own radio and television program, owning its own studio so as to develop the different lessons and call them for onward airing on the radio and on the television, helps to reduce the cost, and helps to be able to instantly begin the process of airing lessons when there is an emergency. (Ministry of Education Official, KII) While teaching by radio or television can be an effective strategy in some places, respondents across categories noted that in many parts of the country, there might not be sufficient electricity or equipment to make it work. Thus, technology-based strategies alone could exclude students in particularly resource-poor households or rural areas, as several participants noted: I think I read that 1 in 5 students in Liberia have access to a radio and so if they don’t have a radio, they are probably not on a computer at home or the laptop. (USAID Official, KII) Sometimes, payment to the radio station has not been forth coming, and some, most of the parents do not have the radio in the first place, even if they have the radio, (lack of) electricity and batteries to run it are some of the problems. (Local Education Official in Nimba, KII). In Montserrado, maybe nearby areas, it was successful, but those hard-to-reach areas, the rural areas, I am sure it didn’t really take effect there, or it was not successful there, because looking at radio stations reaching to those area, having children sitting, listening to radio, something they have not done before, it was kind of difficult, you know, it was not very successful in those areas. (Ministry of Education Official, KII) Findings from the Read Liberia Impact Evaluation conducted by NORC indicate that in 2021, almost 29 percent of the grade 2 students in the sample15 have electricity at home, and almost 75 percent listen to the radio. The proportion is only 26.6 percent for television. Not surprisingly, the percentages are higher in urban areas than in rural areas. However, when asked if while the school were closed, they listened to any learning program on the radio, more than 75 percent responded never or rarely. One non-technology-based strategy that was effective during the pandemic was the use of the student workbooks that had been distributed to the pilot schools before the transition. Students who had their own workbooks could study on their own time with some teacher guidance but also completely on their own when schools were closed, as one teacher explained: That time when everybody had their learner workbook, so that when they come, they meet with their teacher here. One on one we go over it. Majority of them, we go over it with them and then we tell them what they will do and when they will bring it […] because they had their workbooks, so it was successful. (Montserrado County ALP Teacher, KII) In the focus group with students in Lofa County, students explained that even though they were not able to have assignments graded during the pandemic, they were able to continue studying using their ALP student workbooks and maintain their academic progress once schools reopened. Thus, printing and distributing workbooks to the students, in tandem with radio and television programs, could be 15 The sample is representative of public schools in the districts of Bong, Grand Bassa, Lofa, Margibi, Montserrado, and Nimba, that had grade one and grade two classes and at least 20 students in the grade 2 class in 2017. 48 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG effective in helping students continue their academic progression in future crises as well as during periods of stability. Finally, teachers were also asked to compare the number of students who returned to school since schools reopened after the pandemic. Figure 11 shows their responses, which confirm that many students have not returned to school -conventional or ALP- as mentioned above. This highlights the need for a quick response to crises to try to maintain students’ school-attachment, and for strong reenrollment campaigns as soon as schools reopen. Figure 11. Students who returned to school that reopened after COVID-19 1 15 3 1 6 0 1 15 39 16 4 7 5 0 16 49 21 8 6 2 0 0 10 20 30 40 50 60 All Most More than half About half Less than half A few None ALP (n=27) Conventional (n=86) ALP/Conventional (n=102) Note: Sample size <218 due to item-specific missing data. 2.6. WHAT ARE UNIT COSTS FOR AQE PROGRAM PARTICIPANT ACCESS OR COMPLETION? NORC received cost data from USAID/Liberia and from the AQE implementer, EDC for the period March 2nd, 2017, to March 31st, 2021. The data was reported quarterly and in US dollars. Following USAID cost reporting guidance, these expenditure data were organized in the following cost categories: Category 1: General Management and Operations Home Office Field Office Start-up & Close-out Category 2: Monitoring, Evaluation and Learning Assessment Monitoring Evaluation Learning Assessment Category 4: In-Service Teacher Training Development Implementation USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 49 Category 5: Teaching and Learning Materials Development Production and Distribution Stipends Category 6: System Strengthening (Policy/Capacity Development) Category 8: Parents/Community Engagement Development Implementation Category 9: Safe Schools and Infrastructure Construction, Furniture & Equipment Development costs in each category are non-recurrent costs needed to design or create the different components of the activity. For example, development costs under category 5 refer to the costs incurred to design different AQE materials such as students’ workbooks and teachers’ guides for 3 levels. Implementation costs, in contrast, are recurrent costs necessary to field the activity. These costs need to be incurred again if the activity is extended geographically or maintained over time. In addition to the expenses, we received information about in-kind contributions used by the AQE program. The contributions include: GoL staff time allocated to capacity building activities and facilitators training activities, office space for the implementer at the MoE used to coordinate activities, and PTA members time used for parent and community engagement activities. We converted the expenditures to real dollars on 2020-21, monetized the contributions, and add them to compute total costs that reached $ 31,575,738. We show the distribution of the complete program costs by category in Figure 12. Figure 12. Distribution of AQE costs by category (2017-2021) Note: we included AQE expenditures in real dollars of 2020-21 and monetized the contributions 39% 10% 10% 16% 7% 11% 6% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% General Management and Operations Monitoring, Evaluation and Learning Teacher Training Teaching and Learning Materials System Strengthening Parents and Community Engagement Safe Schools and Infrastructure Percentage of total costs 50 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG The costs associated with Category 1: general management and operations are general costs related to tasks that support all the activity components. Following USAID guidelines, those costs were distributed across the rest of the cost categories proportionally to their weight and NORC obtained “full costs” or costs of development and implementation plus costs related to operations, and management. This cost analysis is retrospective, based on the actual expenses incurred by AQE and funded by USAID during the program period, plus contributions received. It includes all costs necessary to run the AQE (including indirect costs and fees) according to USAID implementation guidelines and therefore reflects USAID’s cost perspective. This should not be used to estimate the cost of continuing this program under the MoE implementation. Those costs are different, and we present an estimation in the next section. Based on the AQE program information, NORC selected the cost categories/subcategories that were necessary to produce the outputs of interest in the cost questions. All categories were included except for Category 2-subcategories of monitoring and evaluation, and Category 6, system strengthening. Those components were not deemed as strictly necessary to produce the program outcomes included in the questions below. The total number of students in the intake and the enrollment data, as described in Annex VII, was included for the cost calculations. Using these cost and outcome data, we answer the following questions: What were the total AQE costs per student enrolled at least once in the program? Implementation 137 USD Development and implementation 160 USD Full costs: development, implementation, management, and operations 283 USD The students enrolled at least once could approve the ALP level or not, and a student could have enrolled more than one year. Therefore, this refers to the maximum enrollments seen in the program and shows the cost of AQE as implemented by EDC per student-year enrolled. What were the total AQE costs per unique student enrolled at least once in the program? Implementation 157 USD Development and implementation 183 USD Full costs: development, implementation, management, and operations 325 USD The above question only considers once each of the students reached by the program. Although a student could have been enrolled multiple years, we only consider her once. Therefore, this reflects the total average cost per unique student enrolled in the program. What were the AQE costs per approved student-year? USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 51 Implementation 305 USD Development and implementation 356 USD Full costs: development, implementation, management, and operations 630 USD Finally, NORC computed the average AQE costs for each year approved by the students. Here, again this is not referring to unique students. One student could have approved one, two or three levels of the program. Only some of the students enrolled in the program finish each year and approve the end of the year examination, therefore the costs per year each student approved is higher than the costs per student enrolled. Different cost categories are included in the costs per student computed above. Table 26 below shows the different weight of each cost category included in the full costs. Table 26. AQE Full Costs Categories AQE FULL COSTS IN 2020 USD % PER STUDENT ENROLLED AT LEAST ONCE PER UNIQUE STUDENT ENROLLED PER APPROVED STUDENT￾YEAR Learning assessments 0.9% $ 2.6 $ 3.0 $ 5.7 Teacher training 23.9% $ 67.6 $ 77.6 $ 150.6 Teaching and learning materials 22.6% $ 64.1 $ 73.5 $ 142.7 Teacher stipends 13.9% $ 39.4 $ 45.2 $ 87.7 Parent engagement 24.9% $ 70.4 $ 80.8 $ 156.9 Safe school and infrastructure/furniture 13.8% $ 38.9 $ 44.7 $ 86.7 TOTAL 100.0% $ 283 $ 325 $ 630 Parental engagement is the cost component with the largest weight in the total costs. Almost of quarter of the costs were devoted to involving communities to support the ALP. Cost of training teachers were 23.9 percent of the total costs, followed by teaching and learning materials to which AQE allocated 22.6 percent of the full costs. Similar amounts were spent on teacher stipends and on school infrastructure and furniture, 13.9 and 13.8 percent respectively. Learning assessments are less than one percent of the full costs. The weights for the different cost categories are slightly different for implementation costs. Table 27 below shows the details. If the focus is only on implementation costs, teacher training is the component with the highest incidence on the total costs of implementation (23.3 percent), followed by teaching and learning materials production and distribution (20.8 percent) and parents’ engagement (20.0 percent). Similar amounts of the implementing costs were allocated to teacher stipends and school infrastructure -17.5 and 17.3 percent respectively. Learning assessments took 1.1 percent to the implementing costs. 52 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Table 27. AQE Implementation Costs Categories AQE IMPLEMENTATION COSTS IN 2020 USD % PER STUDENT ENROLLED AT LEAST ONCE PER UNIQUE STUDENT ENROLLED PER APPROVED STUDENT￾YEAR Learning assessments 1.1% $ 1.6 $ 1.8 $ 3.5 Teacher training 23.3% $ 31.9 $ 36.6 $ 71.0 Teaching and learning materials 20.8% $ 28.4 $ 32.6 $ 63.4 Teacher stipends 17.5% $ 23.9 $ 27.4 $ 53.2 Parent engagement 20.0% $ 27.4 $ 31.5 $ 61.1 Safe school and infrastructure 17.3% $ 23.6 $ 27.1 $ 52.6 TOTAL 100.0% $ 137 $ 157 $ 305 All the calculation details can be found in Annex VII part of which is confidential and only available to USAID and EDC staff. 2.7 WHAT WOULD BE THE COST TO THE MOE TO CONTINUE TO SCALE UP THE AQE MODEL? To answer this question, the evaluation team simulated the costs of running an ALP center for ten years. We assumed that at the time of opening, 150 students would join Level 1of the ALP program and 35 new students would join in each of the following years, for a total of 465 unique students enrolled at least one year16. Each year, 40 percent of the students are promoted to the next ALP level (or graduate if they were in Level 3), 12 percent repeat the ALP level, 20 percent are transferred to conventional school and the rest (28 percent) drop out. This amounts to a total of 830 student-years. The simulated number of students in each year is shown in the top panel of Table 28. The second panel in the table shows 3 facilitators allocated to different ALP levels over time. Table 28. ALP center simulation – Number of students and facilitators YEAR TOTAL 1 2 3 4 5 6 7 8 9 10 Students L1 150 53 41 40 40 40 40 40 40 40 L2 60 28 20 18 18 18 18 18 18 L3 24 14 10 9 8 8 8 8 465 students Total 150 113 94 74 68 66 66 66 66 66 830 student-years Teachers L1 3 1 1 1 1 1 1 1 1 1 L2 2 1 1 1 1 1 1 1 1 L3 1 1 1 1 1 1 1 1 3 Teachers Total 3 3 3 3 3 3 3 3 3 3 16 These assumptions are based on the data from AQE Cohort 2 in 2019 when 144 students enrolled and 29 were received the following year. We rounded up given that COVID-19 increased the number of out of school children. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 53 Implementing the ALP requires several inputs: teaching and learning materials, students’ assessment materials, trained facilitators, and space at school. Using the information obtained from AQE NORC computed the cost of facilitators guides and workbooks. Facilitator guides and workbooks cost 8.72 USD each. In level 1, facilitators need three guides and students need three workbooks (literacy, numeracy and life skills/learning together), for levels 2 and 3 four guides and four workbooks are needed (literacy, numeracy, science, and social studies). We assumed that every year every student enrolled receives a full set of workbooks -3 or 4 depending on the level they are attending. In addition, each center receives three sets of level 1 teaching guides in the first year, two sets of level 2 teaching guides in the second year, and one set of level 3 guides in the third year. After that, guides are replaced every 3 years. These assumptions ensure that teaching and learning materials are always available. There are plenty of facilitator guides and each student has its own workbooks every year. The students’ assessment estimated costs are 0.25 USD each. One assessment is needed upon entrance into the program and one assessment is needed at the end of each year. The estimated costs of fully training a facilitator are 290 USD. Finally, it is assumed in this calculation that space and furniture are available and therefore investments in infrastructure or furniture are not included. NORC calculated the costs of the MoE running an ALP center such as the one simulated in Table 28, under different scenarios. Currently the MoE is running the program with trained facilitators that are in the payroll and does not offer any additional stipend to them, therefore this simulation starts with a simple scenario that considers the costs of the current situation. Scenario 1: Three trained facilitators are available. First, the costs of running an ALP center for ten years are calculated assuming that three trained facilitators are available. Also assumed is that the ALP center can work with those facilitators, but one facilitator replacement will be necessary in the ten-year cycle. The costs of running an ALP center with 830 student-years as described in Table 28 under this scenario would be 25,403 USD in total for the ten years or 30.6 USD per student-year. Scenario 2: Training is needed for three facilitators. If no trained facilitator is available at the center and training is needed, the costs of running the ALP Center for ten years would increase to 26,273 USD or 31.7 USD per student-year. This figure includes the cost of training three facilitators in Year 1 and one new facilitator in Year 10, to replace facilitators that retire or leave their position. Scenario 3: New ALP Center -Training is needed for three facilitators and stipends are offered to facilitators. In addition to the scenario 2 assumptions this alternative includes paying facilitators a stipend as used to be the case under the AQE implementation by EDC 2. This scenario is obviously more expensive but also allows to use the school facilities during the afternoons, doubling the capacity while using the existing infrastructure. If each facilitator is paid 50 USD per month -as EDC used to pay￾during 10 months per year, the ten years cost of the ALP center would increase to 41,798 USD or 50.4 USD per student-year. This includes a 1.75 USD fee per salary transaction. A lower salary of 35 USD per month, would imply a total cost of 37,298 USD to run the ALP Center for ten years or 45 USD per student-year. 54 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG More details about these calculations and assumptions used can be found in Annex VII. CONCLUSIONS AND RECOMMENDATIONS In this section we discuss the main conclusions by research questions, and present recommendations from this evaluation. SUSTAINABILITY 1. What are the AQE elements that have been transitioned to and sustained by the MOE of Liberia? 2. What are the AQE elements that were not transitioned? What are the implications for program success that some AQE elements will not be adopted by GOL, related to facilitators’ compensation and workload? Of the 90 AQE surveyed schools, 73 (81 percent) have ALP students. Although this indicates that the program was sustained in a large fraction of schools, generally in a new format, it is worth highlighting some issues that compromise the long-term sustainability of the ALP. First, while 95 percent of principals said that they were receiving support supervision from the DEO or CEO, qualitative data indicate that education officials who were tasked with monitoring the ALP faced logistical challenges that prevented them from fully completing their assignments. Second, the results indicate that 37 percent of ALP facilitators are working more than 25 hours per week. The group that most frequently indicated working more than 25 hours per week is the one that teaches both conventional school and ALP. In addition, not surprisingly, qualitative data indicate that the cease of AQE stipends was not well-received by facilitators. Some volunteer teachers who were not on the government payroll left the program once the stipend ceased, as this was their only compensation. Volunteer teachers who continued teaching have inconsistent attendance and struggle with the workload without financial compensation. This has had a negative impact on the student-teacher ratios. The MoE rolled out a “One School Approach” that restructured the program so that ALP and conventional students come to school at the same time and for the same amount of time. According to some parents, there are some benefits to this approach, especially for student attendance as ALP students were more likely to come to school if they left home in the mornings at the same time as their caregivers, rather than in the afternoons when they were unsupervised. However, the One School Approach also strained school infrastructure and resources, especially in relation to classroom space, seats and teaching time. Most teachers are currently teaching both conventional grades and ALP levels. Many teachers have classrooms with both conventional and ALP students, and most frequently used a combination of conventional and ALP materials. While this is a clear divergence from the original model, it was also a testament to the quality of the ALP materials that are now being used for both conventional and ALP teaching, demonstrating how well they are aligned with the conventional curriculum. Teachers like the ALP approach and materials. Under the current conditions, the “One School Approach” frequently results in the merging of conventional school and ALP. Among schools that offer ALP and conventional school, 19.1 percent of Level 1, 17.1 percent of Level 2 and 25.8 percent of Level 3 students share the class with conventional USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 55 students. There are several reasons for this: 1) the average age difference between students in ALP and conventional school is only around half a year for Levels 1 and 2 and students in Level 3 are even a bit younger than students in grades 5 and 6. The issue of mixing students of very different ages in the classrooms has not been solved; 2) in many schools, infrastructure is insufficient to have separate classrooms to run simultaneous ALP and conventional school classes; 3) many teachers have conventional school and ALP learners mixed in the same class and they use the same approach and materials to teach both groups. This dual classroom approach creates a severe distortion from the original idea of providing an Accelerated Learning Program for over-aged children and youth out of school while operating separate classrooms for conventional primary school students. Regarding teacher training, we found that only 8 percent of teachers had not received training. Most of the trained facilitators were trained while EDC was implementing the program. Facilitators are supposed to be trained over multiple sessions during a year, but once they finish, they do not need to attend more training sessions. It is too early to assess whether this program component survived the transition, as facilitators trained by EDC are not supposed to be retrained. Moving forward, the GoL’s responsibility is to train new facilitators only. 3. How successful were AQE’s strategies to ensure sustainability and institutionalization related to the curriculum, AQE models, and teacher training approaches? Overall, findings indicate that the ALP model has been well-received. In particular, the quality of the curriculum was praised by focus group and interview participants. Most programmatic problems were associated with funding constraints, availability of teaching and learning materials, and merging the classes, not with the quality of the original AQE model itself. Comments from study participants suggest that they are keen to find ways to sustain and scale up the model, especially given the negative impact of the COVID-19 pandemic on student enrollment and potential implications for possible future school closure. In terms of the efficacy of strategies to ensure sustainability and institutionalization, constraints on resources from the MoE severely undermined their success. Shortages of teaching and learning materials is one of the key challenges that schools are facing. When asked about the impact of the transition of the program to MoE oversight, the single most frequent complaint from respondents in KIIs and FGDs was insufficient materials for ALP teachers and learners. In effect, 25 percent of teachers said that none of the students in their classes have a workbook, and 14.5 percent said that fewer than half of the students in the class do. To facilitate teacher training continuity, AQE staff worked with MoE to prepare county and school level staff as master trainers, tasked to conduct ALP facilitator training and school-based monitoring and supervision moving forward. It was also notable that some principals of schools with closed ALP centers stated that they were not informed of or given guidance to undertake the transition. 56 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG EQUITY 5. What is the contribution of AQE to equitable transition to formal education, retention, and completion in Liberia? How did the age, gender, or location of a learner affect their trajectory? Administrative data from AQE show that in 2017-2018, 23 percent of the AQE students sat for the completion assessment, 8 percent passed it, and 23 percent were enrolled next year. These indicators improved in the following years. The results for 2018-2019 show that 46 percent of the AQE students sat for the completion assessment, 32 percent passed it, and 35 percent were retained into the following year. Finally, in 2019-2020, 67 percent of students sat for the completion assessment and 46 percent passed the assessment. These results could underestimate retention in the school system as a whole, as administrative data does not track students that transition to the conventional system. Using the follow￾up survey, NORC estimates that, between 2019-20 and 2020-21, 22 percent of ALP students transitioned to conventional school and 43 percent stayed in ALP, for a total retention rate of 65 percent. In terms of the contribution of AQE to equitable access to education, there are a few important conclusions that can be derived from the administrative and survey data. First, there are no major differences by gender for any of the outcomes. Using administrative data, NORC calculated correlations between gender and assessment completion, assessment approval, and retention, and found that correlations are small, albeit some of them are statistically significant (the samples for the administrative data are very large so it is not surprising that even small differences are statistically significant). Similarly, using survey data there were no major differences found by gender in school retention. The results indicate that the youngest students (ages 6-9) are more likely to drop out of school than older students, especially when we compare students ages 6-9 with students ages10-12. This is observed using administrative and survey data. It could be that households do not prioritize education of their younger children if they think they can join school later anyway, but that older children would go without an education if they don’t stay in school. Another potential explanation could be that younger children are less able to tolerate the shortages of chairs, poor individual teacher attention in overcrowded classrooms, or longer school days under the One School Approach, especially without school feeding programs. We also found that students in urban schools are less likely to stay in school. Administrative data show that there is a negative correlation between schools being in urban areas with assessment completion and with retention, and survey data show that students in Level 3 and ALP graduates are less likely to stay in school the following year if they were enrolled in a school located in an urban area. If job opportunities are better in urban than rural areas, it is possible that students in urban areas face a greater opportunity cost of staying in school than children in rural areas, and therefore tend to drop out of school more. It is also important to note that shortages of chairs present an equity problem, because they exclude students living with a disability, who might not be able to sit on the ground or stand for long hours during the school day. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 57 5. What strategies for AQE delivery could be used to serve learners affected by an emergency like COVID? The Ministry of Education has launched programming that disseminates lessons over radio and is planning to expand further into radio and television. However, it was clear that this approach excludes a significant number of students who live in communities without consistent electricity. The quality of the ALP materials and usefulness of the students’ workbooks were important for continued learning during school closures because of the COVID-19 pandemic. The data showed that some students did not have their own workbooks, which prevented them from progressing academically when schools were closed. Meanwhile, those who had workbooks were able to continue learning. The data indicates that teachers think a program like AQE could help students recover lost learning. Almost all teachers like the AQE curriculum and think that is a good alternative to make up for COVID￾related learning loss, as it is for them simpler and shorter to prepare and to teach, and for the students simpler to grasp. Having a condensed or compressed curriculum ready is a great resource and we should take advantage of it. However, using accelerated education to bring the students up to speed is not straight forward and adaptations and clear guidance to teachers should be considered when using it as catch-up program to help students recover the losses in knowledge and skills. COSTS 8. What are unit costs for AQE program participant access or completion? NORC computed the costs incurred by the AQE program and funded by USAID. These costs include all the expenses, and all the (monetized) contributions received by the program as implemented by EDC. The total AQE cost per student enrolled at least once in the program was $137 for implementation, $160 if it also includes the costs of development, and $283 when it includes management and operations costs. This refers to the maximum enrollments seen in the program, where the same student could have been enrolled in the program for more than one year and therefore shows the total cost of AQE as implemented by EDC per enrolled student-year. The total average costs were slightly higher when we only consider unique students that accessed the AQE. For example, the total implementation costs were $157 for each (unique) student that participated in the program. Finally, NORC computed the average AQE costs for each year approved by the students. One student could have approved, one, two or three levels of the program. In this case, the total cost of implementation was $305 per student per year approved, the costs of implementation and development were $356, and the full costs including management and operations expenses reached $630 per student per ALP level approved. Parental engagement is the cost component with the largest weight in the AQE total costs. Almost of quarter of the costs were devoted to involving communities to support the ALP. Cost of training teachers were 23.9 percent of the total costs, followed by teaching and learning materials to which AQE allocated 22.6 percent of the full costs. Similar amounts were spent on teacher stipends and on school infrastructure and furniture, 13.9 and 13.8 percent respectively. Learning assessments are less than one percent of the full costs. 58 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 9. What would be the cost to the MoE to continue to scale up the AQE model? NORC estimated the MoE costs of running an ALP center for ten years under different scenarios, by simulating an ALP center that initially enrolls 150 students and 35 students every year for the following 9 years, reaching a total of 465 unique students and 830 estimated student-years at the center. Running such a center that initially has three trained facilitators for ten years costs 25,403 USD or 30.6 USD per student-year. These costs include all the teaching and learning materials needed –the main cost driver-, and training for a replacement facilitator. If facilitators were not available and training had to be provided for three people in the first year, the ten-years cost would slightly increase to 26,273 USD or 31.7 USD per student-year. Many respondents discussed that the MoE does not have the same level of resources to devote to the ALP as USAID did. Currently the MoE is trying to run the ALP without any additional compensation to those that participate in the program. Therefore, facilitators’ compensation was not included in the calculation above. What was included were the costs of providing all the enrolled students with their own set of workbooks every year and providing facilitators with training and with teaching guides, replaced every three years. The estimated average annual expense under those assumptions is 2,627 USD per ALP simulated center. In some instances, the absence of additional compensation for ALP facilitators seems to reflect a more efficient use of the teaching staff time in the MoE payroll. However, as mentioned previously, the change was not well-received by facilitators, and had important implications for the quality of the program. Volunteers who were not on the government payroll left and the dedication of those who continued teaching seemed to have suffered. The student-teacher ratios have increased, and now conventional and ALP systems share teachers and classrooms, which affects the type and quality of education. Therefore, in the last scenario we simulated, we included training for three facilitators and assumed that they are paid stipends. This reconfiguration clearly increases the costs of running an ALP center, but it allows schools to use the facilities during the morning and afternoons, doubling the capacity while using the existing infrastructure, enabling the implementation of the original conceptualization of the program. If each facilitator is paid $50 per month -as EDC used to pay- during 10 months per year, the cost of the ALP center for ten years would increase to 41,798 USD or 50.4 USD per student-year. Lower stipends could be considered as well. Assuming a salary of 35 USD per month, NORC estimates a total cost of 37,298 USD to run the ALP Center for ten years or 45 USD per student-year. All simulated scenarios seem in line with the government of Liberia education budget of 50 USD per student per year. RECOMMENDATIONS In light of these findings, NORC presents the following recommendations for improved implementation of the ALP program. These recommendations are centered on improving the sustainability of the program post-transition, especially in terms of quality and costs, improving equity outcomes and taking steps to ensure sustainable programming and continued learning in any future crises and school closures. Transition Process - Recommendations for Effective Implementation by the Ministry of Education MoE needs to coordinate closely with school principals and staff and provide clear guidance and support to help schools transition to a new ALP model. A principal in Bong County where USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 59 the ALP center was forced to close explained that they did not receive adequate information about the transition and were still hoping that they could integrate an ALP component in their school in the future. The Ministry should aim to identify the schools that were uninformed or inadequately informed about the transition and those schools that would still benefit from an ALP program. The Ministry should coordinate closely with these schools to identify the most feasible and effective way to re-introduce or strengthen the ALP program. Place students in ALP or conventional school according to their age. Survey data show that in general, students are not distributed according to age in ALP and conventional school. NORC found that there are very small differences in students’ age profiles between students in the two systems. One of the main purposes of having an ALP is being able to separate on-age and overage children. MoE should work with principals to optimize student distribution. This includes placing students according to skills and age when they are first placed and as they progress, and transition to ALP students in CS that become over age. Explore whether some redesign of facilitators guides, and student workbooks can make them last longer and reduce costs. Almost all the cost of running the ALP is associated with materials printing and distributing expenses, as seen in the cost simulations. In these simulations, it is assumed that facilitator guides would last three years, although it might be possible to use them longer, perhaps five years or more. Re-using workbooks is more challenging. It is important for each student to have their own material to use and be able to take it home. However, it would be worth exploring the possibility of having student materials divided in a workbook component that can only be used once and a “textbook” component that can last several years and be reused by different students. Develop a strategy addressing overworked teachers. This strategy would include increasing the number of teaching staff and optimizing teacher allocation to redistribute hours between those teachers who are overworked and those who are working fewer than 20 hours per week. In schools where the infrastructure or the number of teachers is not enough to run conventional school and ALP separately during the mornings, explore the possibility of returning to offering conventional school in the mornings and ALP in the afternoons. Running the ALP center in the afternoons may require paying some additional compensation to the facilitators and may make the program more expensive but it will also ensure sufficient teacher attention to ALP students and provide the necessary space for each group of students without additional investment in infrastructure and furniture. Merging the programs in one classroom will not accomplish the goal of either the ALP or the conventional school and may diminish student performance in both groups. If in certain locations, it is necessary to maintain classrooms that combine ALP and conventional students, explore the possibility of having support facilitators working alongside the main teacher and provide more seating for students. As mentioned above, combining ALP and conventional school classes is far from ideal and NORC does not recommend it. However, if it is the only approach to educate children and youth, consider opportunities to provide additional forms of in-classroom teaching support, such as apprenticeships for teachers in training, recent education program graduates or appropriately skilled volunteer classroom facilitators. For example, school principals might work with the MoE and Graduate Education programs to identify how to facilitate the placement of teachers-in-training in classrooms, where appropriate and feasible. 60 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Additionally, where possible, principals should explore the option of part-time volunteer facilitators, who might be assigned specific support tasks. Additional seating was also requested, given the larger student classroom size in combined classrooms. The MoE should try to allocate a bit more of the education budget for seating and desk-space. More seating will respond to concerns by teachers, students, and parents about the damage to attendance and learning because of the seat shortage. Explore the possibility of employing remote methods for classroom monitoring and observations so that officials tasked with monitoring ALP classrooms can cut down on transportation costs and travelling time. Accelerated Education Supervisors, District Education Officials, and County Education Officials reported major logistical challenges in fulfilling their oversight responsibilities post-transition. Without sufficient funds for transportation and lodging, they reported that they were unable to travel in person to monitor ALP in their school circuits. Wherever it is possible to set up video-calling or video recording equipment in schools and education offices, this approach for remote observation could be cheaper and more efficient. Over the life of the activity, AQE has provided approximately 330 tablets to the education system. Tablets could be used for remote calls. WhatsApp Web for example allows tablets (or computers) to mirror phones and support video calls. Of course, this might not be possible in all areas of the country. Sustainability Outcomes - Recommendations for Improved Implementation moving Forward: Strengthen the conventional school system to reduce the demand for ALP. The ALP can only be sustainable if the conventional system serves on-age children so that ALP can focus on overage children. As they are currently functioning, the conventional system and ALP are basically two parallel systems serving the same children, in which both groups may be disadvantaged. This dual approach may explain why, even after three years of ALP, most ALP students are enrolled in level 1, as the conventional system keeps leaving students behind for the ALP to enroll. This model is highly inefficient. If on-age students enroll and stay in conventional school, then the numbers of overage children should start decreasing until it becomes marginal. To optimize the students’ age distributions in ALP and the conventional system, GoL should consider: • Working with principals and families so that the latter recognize the importance of on age enrollment, and the benefits of ALP for overage children. • Providing guidelines for principals to gradually redistribute students between the ALP and conventional systems according to age, starting by transitioning the most overage students in the conventional system to the ALP. Given the importance of optimizing age profiles between systems, we recommend that this issue is addressed before any expansion of the ALP is considered. Prioritize allocating resources to the printing of student workbooks, so that students can keep up with lessons even in cases of crisis or school closures, and to improve the quality of classes by making them more streamlined and interactive. Many respondents, especially ALP teachers and students, complained that without individual student workbooks teachers were forced to revert to less effective and less interactive teaching methods. This is because teachers had to write out the content of the curriculum on the board with their back turned to the students so that students could then copy it into their notes. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 61 This not only took extra time, it also worsened the experience of ALP students as they indicated in their focus group discussions. Importantly, the workbooks were fundamental to facilitate learning during school closures due to the pandemic. Thus, as much as possible, printing student workbooks should be a priority for the ALP program. Ensure that school principals continue to be trained in ALP methods, because they often substitute for absent teachers and are the primary monitors of ALP classrooms. Respondents described issues of teacher attendance and attrition in ALP schools. This was a major obstacle to implementation of the program and was especially a problem because they could not call on volunteer teachers who no longer received stipends after the transition. Respondents across counties and schools described principals teaching ALP classes when teachers are absent. Additionally, principals are central to the oversight structure for ALP classrooms; they monitor and observe classrooms themselves and report to Accelerated Basic Education Supervisors, District Education Officials, and County Education Officials. Secure resources so new facilitators are properly trained. It is too soon to determine whether the program is properly recruiting and training facilitators. Most of the surveyed teachers received ALP training, but they did so under AQE. Moving forward, the government needs to make sure incoming facilitators are properly trained. Even in the most basic cost simulations scenarios, NORC included the cost of facilitator training given that it is a fundamental component of the program. Equity Outcomes: Recommendations for Equitable Access to ALP Education including in Future Crises Continue expanding radio and television programs to support accelerated learning lessons but supplement this strategy with others to reach all students. Focusing on expanding radio and television programming alone will exclude a significant number of ALP students, especially those in more rural or remote communities. A hybrid approach that combines different strategies would help prepare the ALP for any future crises or school closures while also being inclusive. The availability of quality materials such as the ALP workbooks in the hands of the students can help learning while schools are closed. Furthermore, their use could be guided by radio and television lessons and/or supported with messages from the schools to students and parents. As mentioned, having a good quality, condensed curriculum ready is a great resource to make up for learning loss during times of crisis. However, plans to provide teacher guidance should be in place, given that using accelerated education to bring the students up to speed is not straightforward. There is a need to adapt to the student body characteristics, the amount of instructional time lost, and the local context, and teachers must receive support and guidance to decide on the best approach to use a condensed curriculum. Consider allocating funds for a school feeding or nutrition program as an MOE budget priority. Although it was beyond the scope of this evaluation to assess the nutritional needs of ALP or conventional school students, the subject of school feeding, and student hunger arose in discussions with students and parents. During future MoE budgeting and in discussions with potential donors, the MoE should consider prioritizing funding for school feeding, especially for disadvantaged students. A nutritional component will likely increase enrollment and retention as well as respond to concerns around equity. 62 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG REFERENCES Ardington, C., G. Wills, and J. Kotze, COVID-19 learning losses: Early grade reading in South Africa, International Journal of Educational Development, Volume 86, 2021, 102480, ISSN 0738-0593, https://doi.org/10.1016/j.ijedudev.2021.102480. Angrist,N, A. deBarros, R. Bhula, S. Chakera, C. Cummiskey, J. DeStefano, J. Floretta, M. Kaffenberger, B. Piper, J. Stern. Building back better to avert a learning catastrophe: estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery Int. J. Educ. Dev., 84 (2021), Article 102397, 10.1016/j.ijedudev.2021.102397 Kaffenberger, M. Modelling the long-run learning impact of the COVID-19 learning shock: actions to (more than) mitigate loss. Int. J. Educ. Dev., 81 (2021), Article 102326 10.1016/j.ijedudev.2020.102326 Gustafsson, M. and C. Nuga. How Is the COVID-19 Pandemic Affecting Educational Quality in South Africa? Evidence to Date and Future Risks NIDS-CRAM Insight Brief I (2020) C. Maslach, S. Jackson, M.P. Leiter. Maslach Burnout Inventory Educator Survey (1997) in Maslach Burnout Inventory Manual, 3rd Edition. C. Maslach, S. Jackson, R. Schwab. Maslach Burnout Inventory Educator Survey (1986) in Maslach Burnout Inventory Manual, 4th Edition. Menendez, Alicia and Sebastian Monroy-Toborda (2017), USAID/Liberia Read Liberia Impact Evaluation, Baseline Report. NORC at the University of Chicago Menendez, A., A. Solovyeva and U. Hoadley, Read Liberia Impact Evaluation, Endline Report. 2021. USAID. NORC at the University of Chicago. Ministry of Education, Republic of Liberia National Teacher Testing and Vetting Report 2018. NORC, The Khana Group Read Liberia Data Collection Report – April 23rd 2021 Smith, W., Consequences of school closure on access to education: lessons from the 2013-2016 Ebola pandemic. Int Rev. Educ., 67 (2021) pp 53-78 Soudien, C., V. Reddy, J. Harvey. The impact of covid-19 on a fragile education system: the case of South Africa (forthcoming) F.M. Reimers (Ed.), Primary and Secondary Education During Covid-19: Disruptions to Educational Opportunity During a Pandemic, Springer (2021) USAID. Year 3 Operational Research Study: Community Engagement Inputs and Student Enrollment and Attendance Pre and Post-COVID Endline Brief. March 1, 2021 USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 63 ANNEXES ANNEX I: EVALUATION DESIGN FRAMEWORK 64 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG ANNEX II: QUALITATIVE INSTRUMENTS KII with Community Leaders Thank you for your taking the time to meet today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the ALP program is working now. The work we are doing is paid for by United States development funding from USAID. As you know, the ALP program recently moved to another stage. Today, I would like to ask you a few questions about your views of the ALP program and the recent changes to education for children in your community. I am particularly interested in your honest views about how things are working for families in your community. Your participation is entirely free and there are no penalties for you if you choose not to participate. If there is any question you do not feel okay answering, you may choose to not respond to the question. You may also choose to stop participating at any time. You can also stop me to ask me any questions you might have while we are talking. All of your answers will be kept secret and nothing you say will be linked to your name. Your identity will be kept confidential, and not shared with anyone outside the research team. If you agree, we would like to record this conversation so that we can clearly and correctly take down what you say and we will delete the recording once we are done with our work. Do you have any questions for me about what I have said? If, after this discussion, you have questions you may contact [contact] of the Khana Group at [phone number] Do you agree to participate? May I begin recording? MODERATOR NOTES: Suggested probing questions are included with each question. Please probe the respondent when necessary, and encourage them to provide details and explanations when possible. QUESTIONS PROBES 1 Please introduce yourself and your responsibilities as a community leader. Years in this position? How many people/families are in your community? KII with Community Leaders Research aim: Understand the community perceptions of the implementation and sustainability of the ALP program and current and future ALP enrollment and benefits to the community USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 65 QUESTIONS PROBES 2 What do you know or hear about the current ALP at the schools in your community? How do you typically get information about the program at the schools? 3 Do you currently do anything related to the ALP program? In the past, did you ever have any role in the ALP? If no, is there any role you can imagine doing to support the ALP? If yes, please explain If yes, please explain. - Contacts related to program - Active role Other interactions, e.g, a parent him/herself 4 What do people in your community think about the ALP? What parts of the ALP do you think are working well for the families of children in the ALP program? What do you think may not be working so well? Why? What do you hear from the parents? From the students? 5 Are there children in your community who are not able to attend or choose not to participate in the ALP program? Why or why not? Do you know why the parents or children opted to participate or not to participate in the ALP program? 6 What do you believe is the future of the ALP in your community? Why? What role do you think community members might have in helping the schools maintain education programs like the ALP? 7 What are the most effective parts of the ALP? Least effective? Or, if no real knowledge of the ALP: What do you think are the best ways to support students who are behind in their learning? Why? 8 Can you please describe how your community dealt with school closings during COVID? If some measures were taken: How well did these measures work? How could they have been better? Do you think these strategies should be replicated in case of future emergencies? 9 If you could make any improvements to this program, what would be the two most important changes you would make? Why? 10 Is there anything else you would like to add related to the ALP? 66 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG KII with AQE/EDC Staff Thank you for your taking the time to meet today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the ALP program is working now. The work we are doing is paid for by United States development funding from USAID. As you know, the ALP program recently transitioned to a different format. Today, I would like to ask you a few questions about your opinions of this transition process and your overall role and responsibilities related to the ALP program. I am particularly interested in your honest opinions about the ALP move from the last stage to the where it is now and how this ALP model can be maintained over time. Your participation is entirely voluntary and there are no consequences for you if you choose not to participate. If there is any question you do not feel comfortable answering, you may choose to not respond to the question. You may also choose to stop participating at any time. You can also stop me to ask me any questions you might have while we are talking. All of your responses will be not have your name and nothing you say will be linked to your name. Your identity will be kept secret, and not shared with anyone outside the research team. With your approval, we would like to record this conversation so that we can accurately capture what you say and we will delete the recording once we are done with our work. Do you have any questions for me about what I have said? If, after this discussion, you have questions you may contact [contact] of the Khana Group at [phone number] Do you agree to participate? May I begin recording? MODERATOR NOTES: Suggested probing questions are included with each question. Please probe the respondent when necessary, and encourage them to provide details and explanations when possible. QUESTIONS PROBES 1 Current job, title? Overview of current responsibilities? Years in this position? 2 Can you please describe your role and responsibilities related to the ALP? What are your responsibilities now compared to the past? Can you describe the EDC’s participation in or management of the With whom did you coordinate most in Liberia? What were the most important aspects of this coordination? What were the weaknesses of the coordination? KII with AQE/EDC Staff Research aim: Understand the strategies for institutionalization of AQE, the management of the transition, and sustainability of the program. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 67 QUESTIONS PROBES transition from the previous model to the new model? 3 How would you say the ALP model is currently working for the teachers? What are some successes of the current ALP model for the teachers? Shortcomings for the teachers? How is the new model working for the students? What are some successes of the current ALP model for the students? Shortcomings for the students? How do you get most of your information about how the new ALP model is working? How are these shortcomings being addressed? 4 How do you view the sustainability of this current model? What aspects of the program make it sustainable? What aspects put the program at risk of failure? Is the curriculum replicable? Is it sustainable? Why? Do you think this current ALP model should continue or should it be changed? If changed, what should be changed? 5 Now, I would like to ask you about the planning for and the implementation for the ALP. Before the program began, what general assumptions did the EDC have about the ALP programming? For example, what did you anticipate from the local implementers, such as head teachers, ministry of education, or others? How did the plans align with what actually happened? Were there any assumptions that were particularly inaccurate? Which ones? Any that were particularly accurate? Which ones? And why? 6 What aspects of the teacher training were most important to make the ALP sustainable? What were the strengths of the materials or training? What aspects could have been improved? What will make these tools most relevant for use in the future? 7 Now, I’d like to ask you about the budget. Can you tell me how the intended program budget compared to the actual program budget? Do you think the funding was adequate to achieve the original intervention aims and the transition aims? If yes, what aspects were most well-funded? If no, why not? How is the current and upcoming funding for this program? Where is it coming from? Who will oversee the spending and reporting? 8 From your understanding of the program, how did EDC envision pulling How did the actual transition differ from these expectations? 68 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG QUESTIONS PROBES out of the program and the future of ALP? 9 Next, I would like to talk to you about the stipends for teachers. Did you know what the plan was for transitioning teachers from stipends to no stipends for the ALP program? What effect did any changes of payment to teachers have on the program? What factors determined how they are compensated? How do you think these changes will in stipends affect the program long-term? 10 What factors do you think most affect the way teachers’ performance under the current model? And compared to before? What would most help teachers to be effective in supporting learning for the ALP students? Time? Resources? Class size? Funding? Head teacher, other support? What types of support, training or resources might they need? 11 Can you please talk about what informed the model of the ALP program? Do you think this program is replicable in other schools in Liberia? Will EDC be drawing on this model for future education programming in other places? What are the strongest aspects that might be useful for replication? Is this the type of model that is feasible for scale-up? Why or why not? 12 In your opinion, what are the most effective and least effective parts of the ALP model? Why? 13 How did the ALP strategy change in response to COVID-19? What was EDC’s role in responding to COVID-19 impact on this program? Can these strategies be replicated in case of future emergencies? 14 If you could make any improvements to this program, what would be the two most important changes you would make? Curriculum? Teaching structure? Time in classroom? Class composition? 15 Is there anything else you would like to add about the program or its future implementation? KII with Principal/Head Teachers of ALP program schools Thank you for your taking the time to meet today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the ALP program is working now. The work we are doing is paid for by United States development funding from USAID. USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 69 As you know, the ALP program recently transitioned to a different format. Today, I would like to ask you a few questions about your opinions of this transition process, the current ALP program and your role or the work you do that is related to the ALP program. I am particularly interested in your honest opinions about how things are working for your teachers in this school and for the students and their families, and how the transition has affected your work. Your participation is entirely voluntary and there are no penalties for you or the educators at your school if you choose not to participate. If there is any question you do not feel comfortable answering, you may choose to not respond to the question. You may also choose to stop participating at any time. You can also stop me to ask me any questions you might have while we are talking. All of your responses will not have your name and nothing you say will be linked to your name. Your identity will be kept confidential, and not shared with anyone outside the research team. If you agree, we would like to record this conversation so that we can clearly and correctly take down what you say and we will delete the recording once we are done with our work. Do you have any questions for me about what I have said? If, after this discussion, you have questions you may contact [contact] of the Khana Group at [phone number] Do you agree to participate? May I begin recording? MODERATOR NOTES: Suggested probing questions are included with each question. Please probe the respondent when necessary, and encourage them to provide details and explanations when possible. KII with Head Teachers Research aim: Understand the impact of the transition on the school, teachers and students and their families 70 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG MAIN QUESTION PROBES 1 First, can you please introduce yourself and tell me about your main role in X School? a) How many years have you been in this position? b) What are your main responsibilities in X school? 2 Can you describe what is your current role related to the ALP or what you do for this particular program? a) About how much time does this take during an average week? b) Who else other than the teachers support the program? c) Most difficult part? 3 Now I’d like to ask you to tell me how this program transitioned from the previous separated ALP to the current ALP-CS model? a) What were the most important things or preparation that had to be done to change the structure or make this transition? b) Describe the teacher training? c) Describe the curriculum changes? d) How did your role change? Was it easier or more difficult than before? Why? 4 Now, I would like to ask you some questions about the current ALP program and how you would compare it to the way ALP classes were before? a) If changed/or same, why? b) Time of day? c) Number of students per classroom? d) Grade levels per classroom? e) Number of overage students? 5 Can you describe what the teachers are doing under the current ALP structure (in L1, L2, L3)? a) More work? less work? b) Is the number of available teachers sufficient? c) Teacher/student ratio? d) Availability of teachers by subject? e) Teacher workload? f) Describe teachers’ attitudes towards the ALP now. g) How has the change in the stipend affected the program? 6 How are the past ALP students participating under the current structure? a) Enrolment for the ALP program? b) Current student attendance? c) Reasons for high/or lower demand? d) Cost implications of demand? e) Attendance? If changed/or same, why? 7 Can you please describe the relationship you have with parents at this school? a) How often are you in contact with parents? b) How are they involved in the school? 8 How has student learning changed since the program changed? And how is their performance now compared to before? How were learning materials distributed to students when schools were closed? c) What responsibilities did teachers have during this time? d) Any students who did not receive learning materials? 9 What proportion of the ALP students do you expect will progress to conventional e) Why do you think certain students will progress and others will not? f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 71 MAIN QUESTION PROBES learning/secondary, vocational or technical school? f) How did students perform on the placement/end-of￾unit assessments? 10 Can you please describe how your school making up for lost learning due to school closings? a) Was this way successful? How could the approach have been better? 11 What are the best or most effective parts and least effective parts of the current ALP program? b) Why? 12 If you could make any improvements to this program, what would be the two most important changes you would make? Curriculum? Teachers training? School resources? 13 Is there anything else you would like to add? KII with Local Education Officials Thank you for your taking the time to meet today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the ALP program is working now. The work we are doing is paid for by United States development funding from USAID. As you know, the ALP program recently transitioned to a different format. Today, I would like to ask you a few questions about your opinions of this transition process and your overall role and responsibilities related to the ALP program. I am particularly interested in your honest opinions about how things are working for teachers and staff in the schools you oversee. Your participation is entirely voluntary and there are no penalties for you if you choose not to participate. If there is any question you do not feel comfortable answering, you may choose to not respond to the question. You may also choose to stop participating at any time. You can also stop me to ask me any questions you might have while we are talking. All of your responses will not have your name and nothing you say will be linked to your name. Your identity will be kept confidential, and not shared with anyone outside the research team. If you agree, we would like to record this conversation so that we can clearly and correctly take down what you say and we will delete the recording once we are done with our work. Do you have any questions for me about what I have said? If, after this discussion, you have questions you may contact [contact] of the Khana Group at [phone number] Do you agree to participate? May I begin recording? MODERATOR NOTES: Suggested probing questions are included with each question. Please probe the respondent when necessary, and encourage them to provide details and explanations when possible. 72 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG QUESTIONS PROBES 1 What is your current job title and your current responsibilities in this job? Years in this position? 2 Can you please describe your current role and responsibilities in related to the ALP-CS (or local terminology) model? How have your responsibilities changed from the previous model? Were you in this position when the ALP program first started? Can you specify the ways you have supported the program? 3 Can you tell me about how many of the ALP centers are running in this community? Are there as many now as there were before they changed the program? If no, why were some ALP centers closed and others not? 4 Now, I would like to ask you about your work related to the AQE? Before the program began, what did you understand your role to be in the AQE? How do these expectations align with the actual role you are doing? Are there more or less responsibilities than expected? 5 What are some successes of the new model? What are some shortcomings of the model? How have you addressed these shortcomings? Has this impacted the quality of the education? Since you are familiar with the operations at multiple schools in this area, I would like to ask you some questions about how the schools dealt with the transition. 6 Were there any schools that experienced more issues in the transition than others? Why? Can you describe how these schools were before the transition? How did this impact students? How did head teachers respond? 7 Can you tell me about how financial resources were allocated across the schools you oversee? Did all schools receive the same financial support during the transition? If no, why? Who determines how resources are allocated to each school? 8 Can you please talk about the compensation or stipends for teachers and staff for the AQE program? How were they compensated in the past and how will they be compensated in the future? Why? How did teachers and head teachers respond to these changes? Are there any other ways that teachers are supported besides salaries and stipends? 9 How did head teachers respond to the transition to the new model? Were the responses positive or negative? How did the transition impact them most? 10 Can you describe how MoE officials have been involved in the AQE model? How has this changed since the new ALP-CS model was introduced? Can you specify how they are involved? Any other groups who are involved? 11 What do you think most affects how teachers work now compared to before? Time? Resources? Class size? Funding? Head teacher, other support? KII with Local Education Officials Research aim: Understand the management of the transition and coordination of teachers and staff f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 73 QUESTIONS PROBES What do teachers need to be able to offer effective teaching on for the ALP students? 12 What are the most and least effective parts of the AQE model? Why? 13 Can you please describe how schools are making up for lost learning due to school closings due to COVID-19? Were they successful? How could they have been better? Can these strategies be replicated in case of future emergencies? 14 If you could make any improvements to this program, what would be the two most important changes you would make? Why? 15 Is there anything else you would like to add? KII with Ministry of Education Officials Thank you for your taking the time to meet today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the ALP program is working now. The work we are doing is paid for by United States development funding from USAID. As you know, the ALP program recently transitioned to a different format. Today, I would like to ask you a few questions about your opinions of this transition process and your overall role and responsibilities related to the ALP program. I am particularly interested in your honest opinions about how things are working for your teachers in this school and for the students and their families, and how the transition has affected your work. Your participation is entirely voluntary and there are no penalties for you if you choose not to participate. If there is any question you do not feel comfortable answering, you may choose to not respond to the question. You may also choose to stop participating at any time. You can also stop me to ask me any questions you might have while we are talking. All of your responses will not have your name and nothing you say will be linked to your name. Your identity will be kept confidential, and not shared with anyone outside the research team. If you agree, we would like to record this conversation so that we can clearly and correctly take down what you say and we will delete the recording once we are done with our work. Do you have any questions for me about what I have said? If, after this discussion, you have questions you may contact [contact] of the Khana Group at [phone number] Do you agree to participate? May I begin recording? MODERATOR NOTES: Suggested probing questions are included with each question. Please probe the respondent when necessary, and encourage them to provide details and explanations when possible. 74 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG QUESTIONS PROBES 1 Current job, title? Current responsibilities? Years in this position? 2 Can you please describe your current role and responsibilities related to the AQE model? Can you describe the MoE’s involvement in the transition from the previous model to the new model? Were you in this position when the ALP program first started? About how much of your time do you spent on the AQE? 3 How would you say the AQE model is currently working for the teachers and the students? What are some successes of the current AQE model? Shortcomings of the model? How do you get most of your information about how the AQE model is working? How is MoE addressing the shortcomings? 4 How well is the AQE curriculum aligned to the national curriculum currently? What do you think have been the effects of aligning the AQE curriculum to the national curriculum on teaching and learning? Advantages? Disadvantages? Impact for teachers? For students? Now, I would like to ask you about your work related to the AQE. 5 Before the program began, what did you understand the role of MoE would be in the AQE? What did you think you or other staff would be doing related to the AQE? How do these expectations align with the actual role you are doing? More or less responsibilities than expected? 6 Can you tell me whether you or the MoE had any involvement in the teacher training process or materials? If yes, what involvement? If no, why not? 7 Can you tell me how ALP teachers are being trained this school year (2020- 2021)? How many days of training are ALP teachers receiving? Is this residential or cluster training (Recall that a residential training involves travel to a slightly distant center and spending a few days at a different training location, and that a cluster training is shorter, and involves hosting teachers for training from a few schools in close proximity)? Who is delivering these training sessions, EDC, MOE, both? Now I would like to ask you questions about the AQE budget and funding. 8 And what about the budget? Can you tell me about how the program was supposed How is the current and upcoming funding for this program? Where is it coming from? KII with Ministry of Education Officials Research aim: Understand the strategies for institutionalization of AQE, and the management of the transition f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 75 QUESTIONS PROBES to be funded and how it was actually funded? Who will oversee the spending and reporting? 9 What do you think most affects the way teachers are expected to work now compared to before? What do teachers need to be able to offer effective teaching on for the ALP students? Time? Resources? Class size? Funding? Head teacher, other support? Describe what is good and what is bad. 10 Can you please tell me about the compensation or stipends for teachers and staff for the AQE program? How were they compensated in the past and how will they be compensated in the future? Why? How do you think this will affect the program? 11 Can you describe how any local education officials have been involved in the AQE model? Can you offer any examples? Or specify the ways local staff have supported the program? 12 Can you please describe the strategies MoE implemented to make up for lost learning due to COVID? Were they successful? How could they have been better? Can these strategies be replicated in case of future emergencies? 13 What are the most effective and least effective parts of the AQE model? Why? 14 If you could make any improvements to this program, what would be the two most important changes you would make? Curriculum? Teaching structure? Time in classroom? Class composition? 15 Is there anything else you would like to add about the program or its future implementation? KII with ALP Facilitators Thank you for your taking the time to meet today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the ALP program is working now. The work we are doing is paid for by United States development funding from USAID. As you know, the ALP program recently transitioned to a different format. Today, I would like to ask you a few questions about your opinions of this transition process, the current ALP program and your role or the work you do that is related to the ALP program. I am particularly interested in your honest opinions about how things are working for your teachers in this school and for the students and their families, and how the transition has affected your work. Your participation is entirely voluntary and there are no penalties for you or the educators at your school if you choose not to participate. If there is any question you do not feel comfortable answering, you may choose to not respond to the question. You may also choose to stop participating at any time. You can also stop me to ask me any questions you might have while we are talking. 76 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG All of your responses will be kept anonymous and nothing you say will be linked to your name. Your identity will be kept confidential, and not shared with anyone outside the research team. If you agree, we would like to record this conversation so that we can clearly and correctly take down what you say and we will delete the recording once we are done with our work. Do you have any questions for me about what I have said? Do you agree to participate? May I begin recording? MODERATOR NOTES: Suggested probing questions are included with each question. Please probe the respondent when necessary, and encourage them to provide details and explanations when possible. MAIN QUESTION PROBES 1 First, can you please introduce yourself and tell me which grade levels you teach? c) How many years have you been in this position? 2 Please can you describe the current structure of the ALP program? (clarify: structure refers to the general set-up of the program). d) Times you teach: hours/days e) Size of classroom f) Students in classroom (CS & ALP) g) Single or two different curricula? 3 How would you compare your current teaching ALP versus the previous program format? h) How do you think the current format affects the how you teach? i) Does it make it easier or harder? j) Versus previous CS teaching? 4 What do you think is working well and what is working poorly about having the classes together/at the same time? k) How is the structure affecting ALP student learning? l) How is this structure affecting your teaching? Now, I’d like to ask you some questions about the training and curriculum for the ALP? 5 What was most useful in the 'teachers' guides'? And the student workbooks? What parts did you think were not useful? Why? What parts could be improved? Why? m) What ALP materials are you using most now compared to before? 6 What do you think are the most useful parts of the curriculum for you? What are the most beneficial parts for the students? 7 Please describe what type of training or support you received to transition the ALP teaching to n) What kind of help or support do you and the other ALP or CS teachers give to one another? KII with ALP facilitators (EDC/GoL) Research aim: Describe the perceptions and experience of facilitators of the transition process and current ALP/AQE teaching model in their school. f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 77 MAIN QUESTION PROBES combined teaching (or make the changes from the past format of ALP to the current one)? o) What training or support do you think would have been useful? 8 What do you think most affects the way you are teaching now versus before? What do you think could help you teach the ALP students? p) Time? Resources? Class size? Funding? Head teacher, other support? q) Describe what is good and what is bad. Thank you, now I’d like to ask you about your students. 9 How has student attendance been since the transition? r) If changed/or same, why? 10 How has student participation been since the transition? s) If changed/or same, why? 11 What proportion of the ALP students do you expect will progress to conventional learning? t) Why do you think certain students will progress and others will not? Thank you, now I’d like to ask you about the school and your future teaching plans. 12 Can you describe how your principal or any local education officials or MoE officials have supported you as an ALP teacher? u) Can you specify the ways she/he has supported the program? 13 What are your future plans as a teacher in this school? Thanks. Now I’d like to ask you about the teaching situation during COVID. 14 Has COVID-19 made the implementation of the ALP more challenging for you? If yes, how? Has COVID affected your ability to be an ALP facilitator? If yes, how? 15 Can you please tell me about how you worked when schools were closed? v) What have you done/plan to do to make up for lost learning? w) Impact this has on students in your classrooms now? Finally, we are coming near the end of this interview and I’d like to ask you your opinion about the ALP and also what recommendations you might have for the program. 16 What are the best or most effective parts of the ALP program? x) Why? 17 What are the least important or least effective parts of the ALP program? y) Why? 18 If you could make any improvements to this program, what would be the two most important changes you would make? z) Curriculum? Teaching structure? Time in classroom? Class composition? KII with USAID Staff Thank you for your taking the time to meet today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the ALP program is working now. The work we are doing is paid for by United States development funding from USAID. As you know, the ALP program recently transitioned to a different format. Today, I would like to ask you a few questions about your opinions of this transition process and your overall role and 78 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG responsibilities related to the ALP program. I am particularly interested in your honest opinions about the transition and your views on the sustainability of the program. Your participation is entirely voluntary and there are no consequences for you if you choose not to participate. If there is any question you do not feel comfortable answering, you may choose to not respond to the question. You may also choose to stop participating at any time. You can also stop me to ask me any questions you might have while we are talking. All of your responses will be kept anonymous and nothing you say will be linked to your name. Your identity will be kept confidential, and not shared with anyone outside the research team. With your approval, we would like to record this conversation so that we can accurately capture what you say and we will delete the recording once we are done with our work. Do you have any questions for me about what I have said? If, after this discussion, you have questions you may contact [contact] of the Khana Group at [phone number] Do you agree to participate? May I begin recording? MODERATOR NOTES: Suggested probing questions are included with each question. Please probe the respondent when necessary, and encourage them to provide details and explanations when possible. QUESTIONS PROBES 1 What is your current job, title, and current responsibilities? Years in this position? 2 Can you please describe USAID’s role in implementing the AQE model? How has USAID’s involvement changed over time? Relationship with MoE? And EDC? 3 Before the program began, what did you understand USAID’s role to be in the AQE model? Do these expectations align with the actual role? Were there more or less responsibilities than expected? 4 Can you please talk about the transition from EDC management to MoE management? What did you think were going to be the main processes for a successful transition? How was USAID involved in the transition? Can you specify the ways USAID supported the transition? Were there any challenges? How did COVID-19 impact the AQE? 5 What were some successes of the transition? What were some shortcomings of the transition? How were these shortcomings addressed, if at all? KII with USAID staff Research aim: Understand the donor’s role and oversight of the transition, and sustainability of AQE model f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 79 QUESTIONS PROBES 6 Next, I would like to talk to you about the budget. Can you please tell me how the AQE was intended to be funded after the transition? Were there commitments made by the MoE/GoL to support the program? 7 Next, I would like to talk to you about the stipends for teachers. Did you know what the plan was for transitioning teachers from stipends to no stipends for the ALP program? What effect did any changes of payment to teachers have on the program? What factors determined how they are paid? How do you think changes will affect the program long￾term? 8 What do you believe is needed for the future success of the AQE model? What are USAID’s plans for future funding/involvement? Do you think this is a good model to replicate in other locations /scale to other schools in Liberia and other countries? Why or why not? 9 How did the AQE strategy change in response to COVID-19? What was USAID’s role in responding to COVID-19 impact on this program? Can these strategies be replicated in case of future emergencies? 10 What are the most effective and least effective parts of the AQE model? Why? 11 If you could make any improvements to this model, what would be the two most important changes you would make? Why? 12 Is there anything else you would like to add about the program or its future implementation? Focus Groups Discussion with Parents of Current ALP Students Moderator Instructions: At the beginning of the discussion, the moderator will assign all participants a number. This will facilitate confidentiality and facilitate transcribing discussions. Groups will be separated by gender. We need to indicate the parents/caregivers that we are referring to their children currently in ALP. 80 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Thank you for your attendance today. My name is [moderator name] and I work with the Khana Group, a research organization based in here in Liberia. We are working with a research group in the United States named NORC, which is at the University of Chicago in Illinois. We have been asked to talk to you about how the Accelerated Learning Program (ALP) is working now. The work we are doing is paid for by United States development funding from USAID. Today, I would like to ask you a few questions about your views of your child’s education in the ALP, and your child’s experiences in and out of school. This discussion will last approximately 90 minutes. Your participation is voluntary. There are no penalties for you or your child if you choose not to participate, nor will it bring you any additional benefits. If there is any question you do not feel okay answering, you may choose to not respond to the question. You may also choose to stop participating at any time. All of your answers will be kept secret and nothing you say will be linked to your name. Your identity will be kept confidential, and not shared with anyone outside the research team. If you agree, we would like to record this conversation so that we can clearly and correctly take down what you say. Your name will not be on the recording, and to protect each other, we will only to refer to each other by our first names. We will delete the recording once we are done with our work. Everyone will have an opportunity to share their views. Please respect everyone’s responses, even if they are different from yours. Please keep the discussion confidential and to not disclose who else participated in the sessions or what any specific person said. Do you have any questions? If you have questions after the discussion you may contact [contact] of the Khana Group at [phone number] Do you agree to participate? May we begin recording? TOPICS MAIN QUESTION PROBES 1 Thank you. First, I would like everyone to introduce themselves. Can we please go around and state your first name. MODERATOR NOTE: Moderator and note-taker should introduce themselves first. 2 How many children do you have in the ALP, and how long have they been in the program? ▪How old are your children in the ALP? How many children do you have in regular school? How old are they? Now, I would like to ask you some questions about your child who is in the ALP. MODERATOR NOTE: if parent has multiple children in the ALP, please ask them to only respond based on the child who has been part of the program the longest. f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 81 TOPICS MAIN QUESTION PROBES 3 How would you compare the way your children’s classes were before versus now? ▪How has the time of day for their classes changed? ▪How has the time spent at the school changed? How has the time spent on homework or after￾school revision changed? 4 Can you each talk a little about how you decided to send your children to the ALP? ▪What did you think might be good about enrolling in your child to the program? What did you think might be difficult? 5 How has your children’s attendance been over the past semester? Has it changed or stayed the same? Why? Thank you. Now I’d like to ask you to discuss how your children are doing in school. 6 Can you each tell me a bit about whether your children’s performance has changed recently or stayed about the same? ▪Have your children experienced any particular challenges that have affected her/his studies? Has she or he experienced any particular successes? 7 Can you each tell us about your child’s attitudes towards school and the ALP in particular? How do you think they feel about learning in the ALP program? Thanks for your thoughts. I’d like to ask you some questions about the options for your children to transfer to the conventional school format. 8 Do you think your child will transfer to the conventional school? ▪ If no to any future transfer, why not and what might they do instead? If yes, when do you think he will feel ready to change? Now I would like to ask you about parents’ role in the ALP. 9 Do any of you know what role any parents play in the ALP? If yes, what types of things do any parents do? e.g., buy materials, leadership, support teaching, transportation? 10 Do you now or have you ever participated or had a role in the ALP? ▪ If yes, what types of things do you do? If no, have you ever been asked to play a role in the ALP? Thank you, now I’d like to ask what you like and don’t like about the ALP program and any recommendations you might have. 11 Can you name one thing you like and one thing you don’t think is so good about the ALP program? Please explain why you think this? 12 If you could make any improvements to this program, what would be the two most important changes you would make For example, think about: Curriculum? Time in classroom? Amount of homework? Before we finish, I would like to ask everyone if they would be willing to have one of their current ALP students to participate in a similar activity. MODERATOR: READ ALOUD THE PARENTAL CONSENT STATEMENT AND OBTAIN INDIVIDUAL CONSENT FROM EACH PARENT. Thank you for your time. Is there anything else you would like to add about your student, parents’ role or about the ALP in general? 82 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Focus Groups with ALP Students Instructions: At the beginning of the discussion, the moderator will assign all participants a number. This will facilitate confidentiality and facilitate transcribing discussions. Number of students per group: maximum up to five students per group. Age range: Age 11 years and older. Gender: youth can be separated into groups of girls and groups of boys if appropriate. Equipment: 1) Three coins. One coin will be used for flipping by students. The two other coins will be placed one in front of each basket/bowl. Each student will take turns flipping the coin to determine from which basket/box/bowl they will choose a question. 2) Question cards: Write one question per card. Do not write the probes on the card. 3) Two baskets or bowls or boxes. TO PREPARE: Divide the interview questions evenly into two baskets/bowls/boxes. Be sure to put a ‘fun’ (in blue) questions at the top of each basket. MODERATOR ROLE: Rotating around to each student, each student will flip the coin and pick a card from the corresponding basket/box/bowl. One student will ask the selected question, then moderator will help encourage the other students to answer the same question. Each question may have probes that you can ask to facilitate the discussion. HI everyone. Thank you very much for coming today. My name is [moderator name] and I work with the Khana Group, and we are a research organization based in here in Liberia. And, this is my colleague, [introduce note-taker], who will be taking some notes so we can remember all the important ideas you tell us. So, who are we and why are we doing this? We are working with a group from the University of Chicago in the United States and we are trying to learn about how you and other students feel about the accelerated learning program [check how the youth refer to it]. This is why we are so grateful that you could join today, because we know that you are really the local experts about your school and specifically about the accelerated learning program [check how the youth refer to it]. The things we will ask you about today will help people try to make the program as good as it can be for you and especially for future students, so we hope you will share your ideas with us. We also hope some of this session will also be fun, too. Again, we want to thank you for taking this time to help us out. As we explained before, this get-together is not for your studies or for school and there won’t be any sort of test about anything we talk about. In fact, we won’t be telling other people about what you talked about. Your name will be kept secret—or anonymous. We just want to hear your and your classmates’ opinions. There are no right or wrong answers to anything we will ask. And, your participation is totally voluntary, which means you don’t have to participate if you don’t want and you can decide to stop participating whenever you want. Nothing will happen if you decide you don’t want to participate. Also, when we talk today, you can choose what things you want to answer and what you don’t. And, if you don’t understand something, it is probably because other people don’t either, so please stop us and ask. We think our discussion will last approximately 45 minutes to one hour. So that we can make sure we remember everything that people said, we would like to record this session. We will delete the recording once we are done with this work. If that is okay with you, can you raise your hand? f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 83 So, while this is not school, we do have some rules so that everyone gets to share their ideas. The first rule is that we have to avoid talking while other people are talking because we want to listen to their ideas. The second rule is to respect everyone’s responses, even if they are different from yours. We want you to help us lead the questions. So the last instructions are: When you flip the coin you will get the opportunity to draw a question from the basket that corresponds to that side. The person that has rolled the dice will get the first opportunity to answer the question, and then the rest of you will have the opportunity to answer or respond to what the others have said. Do you have any questions? Do you each agree to participate? May I begin recording? Thank you. First, I would like everyone to introduce themselves. Can you please go around and state your name, age, grade level and how long you have been part of the ALP? Now, if you are ready, we will start the activity. As you can see, we have two baskets and one coin. One basket is for when the coin lands on heads and the other is for when you land on tails [change heads/tails to words that are easy understood, if needed]. You will each get the opportunity to flip the coin. Depending on whether you land on heads or tails, you will select the basket that says heads or tails. In each basket, there are pieces of paper with a question on them, which you get to answer first and then ask the others. When you flip the coin, you will get the opportunity to draw a question either from the first basket or the second one. The person that has flipped the coin will answer the question first, and then the rest of you will have the opportunity to answer or respond to what the others have said. MODERATOR: Pause and ask the students if they understand the instructions before proceeding. MODERATOR: The following questions will be added to the basket and asked as they are picked out from the basket. One student will ask the main question, while you help encourage the other students to answer. Each question may have probes that you can ask to facilitate the discussion. Main question Probes QUESTION BANK to be stacked into each basket/box/bowl. The following ice-breakers will be stacked on the top of each basket: If you could have any superpower, what would it be and why? If you could trade places with any famous person for a day, who would that be, and why? What do you think is the funniest looking animal and why? If you could change your name, what name would you pick In your free time, what do you do for fun? QUESTION PROBES 84 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Main question Probes What do you like most about your accelerated learning program now that the classes are mixed versus before? Why? How is this different from before? Name something you don’t like about the accelerated learning program now that the classes are mixed versus before? Why? How is this different from before? In general, how many days per week do you go to school? ▪ How many hours a day do you spend at school? ▪ Has it changed since the classes were merged? ▪ What are some reasons why students are sometimes absent? What makes a good teacher for the ALP and what makes a teacher a bad ALP teacher? ▪ Were your teachers better, the same or worse before the classes were mixed? What difficulties do ALP students experience at school? ▪ What do you think can be done to fix these challenges? ▪ Is there someone you could talk to about these problems? When schools were closed during COVID, how did you spend your time instead of going to school? ▪ How did you learn during this time? ▪ Did you get any learning materials from anyone during this time? If yes, from whom? ▪ If no, did you want any? Do you think you will transfer to the regular school, secondary school, or vocational school? ▪ Why or why not? ▪ What other options do you have? If you could make any improvements to the ALP, what would be the two most important changes you would want? MODERATOR: Please close with these debriefing questions: Debriefing questions Question What do you think about this game and our discussion? Was anything about it difficult? Do you have any questions? Thank you for participating today. Is there anything else you would like to talk about while we are all here? f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 85 ANNEX III: QUANTITATIVE INSTRUMENTS Liberia AQE: Parent/Caregiver Questionnaire Start Date and Time (Autofill): Enumerator Name: INTERVIEW Liaison Name: ________________________________ Liaison Phone number: _____________________________ Date (DD/MM/YYYY) |__|__|/|__|__|/|__|__|__|__| DISTRICT: ________________________________ VILLAGE: ________________________________ NAME: _______________________ NAME: _________________________ SCHOOL CLUSTER ID: CODE |__|__|__| NAME: _________________________ SCHOOL ID: CODE |__|__|__|__| NAME: _________________________ STUDENT ID: CODE |__|__|__|__|__|__| NAME: _________________________ PARENT NAME: __________________________________________________ PARENT PHONE NUMBER: ________________________________ PARENT APPOINTMENT SET UP? Y / N DATE, TIME, AND LOCATION: _________________________________________________ GIS LOCATION OF HOUSEHOLD (IF GIS UNAVAILABLE) LATITUDE(N/S) |__| – DEGREES:|__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| LONGITUDE (E/W) |__| – DEGREES: |__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| VILLAGE/TOWNSHIP/DIRECTIONS TO HOUSE (RECORD REGARDLESS OF WHERE YOU INTERVIEW): ___________________________________________________________ 86 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG DISPOSITION CODES NO REPLACEMENT NEEDED REPLACEMENT NEEDED IF FINAL DISPOSITION 07 Parent/guardian absent for extended period (e.g. in hospital, working in another town, deceased) 08 No one at home after 3rd visit: replace 09 Parent/guardian did not consent to interview 10 Partial Complete/Interview Finished (interview was stopped and will not continue) 11 Other Non-Interview (specify in Notes) 01 Interview NO REPLACEMENT NEEDED—MUST VISIT LATER (ATTEMPTS #1 AND #2 ONLY) 02 Appointment no show 03 No one home—will visit again 04 Postponed/Rescheduled (interview was never started: postponed and a new time scheduled) 05 Temporary Refusal (interview was refused, Supervisor will follow-up) Section A: Introduction and Consent Enumerator Instructions: Enumerator, read the script below in its entirety. Hello, my name is [enumerator, please mention your name]. I am working with The Khana Group, a data collection firm based in Monrovia, on a study to try to understand how to improve accelerated learning in Liberia. This evaluation is funded by USAID. We would like to ask you a few questions about your children’s education, how they spend their time, as well as a few questions about your household. If you agree to participate in the survey, all the observations and answers that you provide will be kept confidential. We will write a report but neither you nor your children will be identified by name. To support future research, anonymized data from this study may be released to the public. However, all information that may be used to identify you or your children will be removed prior to public release. You are free to choose not to answer any question and you can stop the interview at any time, or ask me to repeat something you did not hear or understand. You can also withdraw from the survey at any time. Stopping the survey, not answering any question or answering the full survey will not impact you Attempt Date Time Enumerator Number Disposition code Notes #1 #2 #3 f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 87 negatively in any way, nor will it bring you any additional benefits. However, your honest answers to our questions will help us understand how to improve accelerated learning in Liberia. We greatly appreciate your cooperation. This interview will take about 30 minutes. Do you have any questions for me now? If you have any additional questions about the survey of methodology, you can contact the following person: [Representative from The Khana Group; contact information of representative] Do you agree to participate? [Enumerator: do not read the following questions up until A_1 aloud] consent. Does the parent/primary caregiver consent? Yes No [skip to end of questionnaireà] child_name. Enter child name: _________ A_1. Are you the parent/primary caregiver of [child_name]? 1 Yes 0 No -99 Refuse A_2. How old is [child_name]? ______ years A_3. What is [child_name]’s gender? 1 Male 2 Female A_4. What is your relationship with [child_name]? Mother Father Grandmother Grandfather Aunt Uncle Sister Brother Male Cousin Female Cousin OTHER Which? ______ 88 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Section B: Schooling B_1. Is [child_name] currently in school? 1 Yes 0 No -98 Don’t know -99 Refuse B_2. Why is [child_name] not in school? [PROG: If B_1=No] 1 Finished primary and doesn’t want to continue 2 Not selected to go to secondary school 3 Not doing well in school 4 Doesn’t like school/rather do something else 5 Safety concerns at school/in the way to school 6 Travel distance to school 7 Fees/Financial/Costs 8 Needs to work 9 Expelled from school 10 Marriage 11 Pregnancy 12 Has baby/nursing baby 13 She/he is ill 14 Illness in the household/death in the household 15 Needs to look after siblings 16 Other _____________________ -98 Don’t know -99 Refuse B_2a. What is the highest grade or level [child_name] has completed? [PROG: If B_1=No] 1 Grade 1 2 Grade 2 3 Grade 3 4 Grade 4 5 Grade 5 6 Grade 6 7 Grade 7 8 Grade 8 9 Grade 9 10 Grade 10 11 Grade 11 12 Grade 12 13 ALP 1 14 ALP 2 15 ALP 3 16 ABE 1 17 ABE 2 18 ABE 3 f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 89 19 Other ________ -98 Don’t know -99 Refuse B_3. In what type of program is [child_name] enrolled? [PROG: If B_1=Yes] 1 Regular or conventional school 2 ALP 3 ABE 4 Both Regular or conventional school and ALP 5 Other ________ -98 Don’t know -99 Refuse B_4. In what conventional school level is [child_name] currently enrolled? [PROG: If B_3=1 OR B_3=4] 20 Grade 1 21 Grade 2 22 Grade 3 23 Grade 4 24 Grade 5 25 Grade 6 26 Grade 7 27 Grade 8 28 Grade 9 29 Grade 10 30 Grade 11 31 Grade 12 -98 Don’t know -99 Refuse B_5. In what ALP level is [child_name] currently enrolled? [PROG: If B_3=2] 32 ALP 1 33 ALP 2 34 ALP 3 -98 Don’t know -99 Refuse B_6. In what ABE level is [child_name] currently enrolled? [PROG: If B_3=3] 35 ABE 1 36 ABE 2 37 ABE 3 -98 Don’t know -99 Refuse B_7. Was [child_name] enrolled in school last year, before the COVID-19 related school closures? 90 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 1 Yes 0 No -98 Don’t know -99 Refuse B_8. What program was [child_name] enrolled in last year before the COVID-19 related school closures? [PROG: If B_7=Yes] 1 Regular or conventional school 2 ALP 3 ABE 4 Both Regular or conventional school and ALP 5 Other Specify ________ -98 Don’t know -99 Refuse B_9. In what conventional school level was [child_name] enrolled last year? [PROG: If B_8=1 OR B_8=4] 1 Grade 1 2 Grade 2 3 Grade 3 4 Grade 4 5 Grade 5 6 Grade 6 7 Grade 7 8 Grade 8 9 Grade 9 10 Grade 10 11 Grade 11 12 Grade 12 -98 Don’t know -99 Refuse B_10. In what ALP level was [child_name] enrolled last year? [PROG: If B_8=2] 1 ALP 1 2 ALP 2 3 ALP 3 -98 Don’t know -99 Refuse B_11. Did [child_name] sit for a national exam to graduate from ALP? [PROG: If B_10=3] 1 Yes 0 No -98 Don’t know -99 Refuse f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 91 B_12. Did [child_name] graduated from the ALP program? [PROG: If B_10=3] 1 Yes 0 No -98 Don’t know -99 Refuse B_13. To what extent do you think that finishing the ALP has helped/or can help [child name] to deal with emergencies like COVID? [PROG: If B_12=Yes] 1 A great deal 2 A little 3 It doesn’t make much difference 4 It doesn’t help at all -98 Don’t know -99 Refuse B_14. Did [child_name] sit for a national exam to graduate from primary school? [PROG: If B_9=6] 1 Yes 0 No -98 Don’t know -99 Refuse B_15. Did [child_name] graduated from primary school? [PROG: If B_9=6] 1 Yes 0 No -98 Don’t know -99 Refuse B_16. In what ABE level was [child_name] enrolled last year? [PROG: If B_8=3] 4 ABE 1 5 ABE 2 6 ABE 3 -98 Don’t know -99 Refuse B_17. How many hours per day does [child’s name] attend school? [PROG: If B_1=Yes] 1 Less than 2 hours 2 Between 2 and 4 hours 3 More than 4 hours per day 4 Other ________ -98 Don’t know -99 Refuse 92 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG B_18. How does this compare with last year, while schools were open? [PROG: If B_17=1, 2, 3 OR 4 AND B_7=Yes] 1 More than last year 2 Less than last year 3 Same as last year -98 Don’t know -99 Refuse B_19. Is [child_name] planning to go back to school next year? [PROG: If B_1=No] 1 Yes 0 No -98 Don’t know -99 Refuse B_20. How does [child_name] spends most of his/her time during a weekday? [PROG: If B_1=No] 1 Resting, playing or doing nothing 2 Hanging around with friends 3 Looking for a paid job 4 Working for pay 5 Working in a family business/farm without pay 6 Caring for his/her child 7 Caring for a child or adult in the family 8 Housework 9 Attending secondary school 10 Attending vocational training/apprenticeship 11 Travelling 12 Ill or disabled and unable to work 13 Other Specify_________________ -98 Don’t know -99 Refuse Section C: Demographic/Household Information C_1. Do you work for pay or to generate income? 2 Yes 0 No -98 Don’t know -99 Refuse C_2. How many people, including yourself, all adults and all children, usually live in this household? [Enumerator, please enter number] ______ C_3. Do you know how to read and write a simple sentence in English? 3 Yes f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 93 1 No -98 Don’t know -99 Refuse C_4. Does your household have: [Enumerator, please select all that apply] 1 Potable water inside the home? 2 A toilet inside the home? 3 A refrigerator or freezer? 4 Windows with glass in them? 5 A radio 6 A television? 7 Internet access? 8 A bicycle? 9 A motorcycle? 10 A car, truck or boat for personal use? -98 Don’t know -99 Refuse C_5. Do you or does someone in your house own a cell phone? 1 Yes 0 No -98 Don’t know -99 Refuse C_6. Is this household connected to the electric grid? 1 Yes 0 No -98 Don’t know -99 Refuse Thank you very much for helping with this research by sharing your experience as a parent/primary caregiver. End time (Autofill) 94 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Liberia AQE: School Observation/Principal Questionnaire Start Date and Time (Auto fill): Enumerator Name: Enumerator Code: INTERVIEW Name: ________________________________ Phone number: _____________________________ Date (DD/MM/YYYY) |__|__|/|__|__|/|__|__|__|__| DISTRICT: ________________________________ VILLAGE: ________________________________ NAME: _______________________ NAME: _________________________ SCHOOL CLUSTER ID: CODE |__|__|__| NAME: _________________________ SCHOOL ID: CODE |__|__|__|__| NAME: _________________________ DATE, TIME, AND LOCATION: _________________________________________________ GIS LOCATION OF SCHOOL (IF GIS UNAVAILABLE) LATITUDE (N/S) |__| – DEGREES:|__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| LONGITUDE (E/W) |__| – DEGREES: |__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| VILLAGE/TOWNSHIP/DIRECTIONS TO SCHOOL (RECORD REGARDLESS OF WHERE YOU INTERVIEW): ___________________________________________________________ DISPOSITION CODES NO REPLACEMENT NEEDED REPLACEMENT NEEDED IF FINAL DISPOSITION 03 Principal did not consent to interview 04 Partial Complete/Interview Finished (interview was stopped and will not continue) 01 Interview f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 95 NO REPLACEMENT NEEDED—MUST VISIT LATER (ATTEMPTS #1 AND #2 ONLY) 02 Postponed/Rescheduled (interview was never started: postponed and a new time scheduled) 05 Other Non-Interview (specify in Notes) Section A: Introduction, Consent, and ALP Standards Enumerator Instructions: Enumerator, read the script below in its entirety. Hello, my name is [enumerator, please mention your name]. I am working with The Khana Group, a data collection firm based in Monrovia, on a study to try to understand how to improve accelerated learning in Liberia. This evaluation is funded by USAID. We would like to ask you a few questions about how this school works. You are free to choose not to answer any question and you can stop the interview at any time or ask me to repeat something you did not hear or understand. You can also withdraw from the survey at any time. Stopping the survey, not answering any question or answering the full survey will not impact you negatively in any way, nor will it bring you any additional benefits. However, your honest answers to our questions will help us understand better how to improve accelerated learning in Liberia. We greatly appreciate your cooperation. If you agree to participate in the survey, all the observations and answers that you provide will be kept confidential. We will write a report but your school will not be identified by name nor will you be identified individually by name. To support future research, anonymized data from this study may be released to the public. However, all information that may be used to identify you or your school will be removed prior to public release. This interview will take about 45 minutes. Do you have any questions for me now? If you have any additional questions about the survey or methodology, you can contact the following person: [Representative from The Khana Group; contact information of representative] Do you agree to participate? [Enumerator: do not read the following questions up until 1.1 aloud] Attempt Date Time Enumerator Number Disposition code Notes #1 #2 #3 96 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Consent. Does the principal consent? Yes □ No □ [skip to end of questionnaireà] principal_name. Enter principal name: _________ principal_gender. Enter principal’s gender: _______ [Enumerator: please read the following questions aloud] X_1. Does this school have any students in ALP classes, or are all students in the school in conventional school? 1 We have some students in ALP classes 2 All students are in conventional school X_2. Did this school have an ALP in school year 2019-2020? [Only if X_1=2] Yes No DKN X_3. When did it close? [Only if X_2=Yes] __________ (month and year) X_4. Why there are no students in ALP classes in this school anymore? [Enumerator: Do not read options, select all that apply] [Only if X_2=Yes] 1 The school does not have enough teachers/facilitators 2 The school does not have enough classrooms 3 The school does not have other resources required for ALP classes Which ones?_______________ 4 Facilitators did not want to teach both ALP and conventional school 5 The school does not have enough overaged students to fill ALP classes 6 Students/parents in this school prefer to be enrolled in conventional school 8 Other □ Specify_________________ -98 Don’t know □ -99 Refuse □ f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 97 Section A. Monitoring and supervision [Only if X_1=1] A_1. Are ALP classes integrated in the school timetable? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_2. Is the daily schedule of ALP classes posted on school bulletin board? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_3. Excluding the students that had never been in school and therefore are placed in ALP level 1, how many new ALP students did a placement assessment this school year? 1 All new students □ 2 Most new students □ 3 More than half new students □ 4 About half new students □ 5 Less than half new students □ 6 A few of the new students □ 7 None of the new students □ -98 Don’t know □ -99 Refuse □ A_4. Do you have records of these placement assessments? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_5. How many ALP students have done an End-of-Unit assessment for the last complete unit? 1 All □ 2 Most □ 3 More than half □ 4 About half □ 5 Less than half □ 6 A few □ 7 None □ -98 Don’t know □ -99 Refuse □ 98 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG A_6. Do you have records of these End-of-Unit assessments? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_7. How many ALP students did a Completion assessment at the end of last year/semester? 1 All □ 2 Most □ 3 More than half □ 4 About half □ 5 Less than half □ 6 A few □ 7 None □ -98 Don’t know □ -99 Refuse □ A_8. Do you have records of these Completion assessments? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_9. Does the school have ALP attendance records? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_10. Are ground rules for ALP classes on view in the school bulletin board and/or in classrooms? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_11. Is the school following a morning, afternoon or evening schedule for the ALP program? [Enumerator, select all that apply] 1 Morning program 2 Afternoon program 3 Evening program 8 Other □ Specify_________________ -99 Refuse □ f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 99 A_11. Is the ALP-CSP receiving support supervision from the DEO or CEO (by phone/ by site visit)? 1 Yes 0 No Skip to Section B -98 Don’t know □ -99 Refuse □ A_12 How frequently? 1 Once a month or at least once a month 2 Once every three months 3 Once every six months 4 Once a year 5 Less than once a year -98 Don’t know □ -99 Refuse □ Section B: ALP-CS Integration Module B_1. How many physical classrooms are in this school? Please include every room available even if they are used sporadically only or never used. _____ [Enumerator, please enter number] -98 Don’t know □ -99 Refuse □ B_2. Now, can you tell me how many classrooms are currently in use? Please count only the rooms where there is class throughout the school day._____ [Enumerator, please enter number] -98 Don’t know □ -99 Refuse □ B_3. How many full-time teachers/facilitators teach conventional grades only? _____ [Enumerator, please enter number] -98 Don’t know □ -99 Refuse □ B_4. How many part-time teachers/facilitators teach conventional grades only? _____ [Enumerator, please enter number] -98 Don’t know □ -99 Refuse □ B_5. How many full-time teachers/facilitators teach ALP only? _____ [Enumerator, please enter number] [Only if X=1] -98 Don’t know □ -99 Refuse □ 100 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG B_6. How many part-time teachers/facilitators teach ALP only? _____ [Enumerator, please enter number] [Only if X=1] -98 Don’t know □ -99 Refuse □ B_7. How many full-time teachers/facilitators teach both conventional grades and ALP? _____ [Enumerator, please enter number] [Only if X=1] -98 Don’t know □ -99 Refuse □ B_8. How many part-time teachers/facilitators teach both conventional grades and ALP? _____ [Enumerator, please enter number] [Only if X=1] -98 Don’t know □ -99 Refuse □ B_9. What conventional grades are taught at this school? [Enumerator, select all that apply] 1 Kindergarten □ 2 Grade 1 □ 3 Grade 2 □ 4 Grade 3 □ 5 Grade 4 □ 6 Grade 5 □ 7 Grade 6 □ 8 Other □ Specify_________________ -98 Don’t know □ -99 Refuse □ B_10. What ALP levels are taught at this school this year? [Enumerator, select all that apply] [Only if X=1] 1 ALP 1 □ 2 ALP 2 □ 3 ALP 3 □ B_11. Please answer the following questions for each of the grades and levels that are taught in this school B_12. How many hours per day are students in conventional school taught in: [Enumerator, please enter number] 1 Grades 1-2 _ _ 2 Grades 3-4 _ _ 3 Grades 5-6 _ _ -98 Don’t know f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 101 -99 Refuse B_13. How many hours per day are ALP students taught in: [Enumerator, please enter number] [Only if X=1] 1 ALP 1 _ _ 2 ALP 2 _ _ 3 ALP 3 _ _ -98 Don’t know -99 Refuse Section C: Enrollment Protocol Record student age distribution: 1. Enumerators will ask for the school rosters of grades 1-6 and ALP 1-3. For each school roster enumerators will record total number of students by class and gender. C.1 Grade/level C.3 Is this class sharing the classroom with other grade/levels? 1 Shared 2 Not shared C.4 Which grades/levels are these students pooled with? [PROG: D3=1] 1 Kindergarten 2 Grade 1 3 Grade 2 4 Grade 3 5 Grade 4 6 Grade 5 7 Grade 6 8 ALP 1 9 ALP 2 10 ALP 3 -98 Don’t know -99 Refuse C5. Does this class has its own facilitator/teacher or shared with the class? [PROG: D3=1] C6. Total boys C7. Total girls 2. PROG: The tablet will select a random conventional school grade and a random ALP level to record the number of students by age: 102 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG C8. Grade/level C9. Class (A, B, C,…) C10. Boys ages C11. Girls by age Total Total Grade X (1-6) Level Y (1- 3) f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 103 CX. Are there any grades in this school that have more than one classroom? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ Record enrollment of selected ALP students 1. Enumerators will be provided with lists of students who were enrolled in AQE between 2017 and 2020. For each school there will be a list of 10. These lists will be assembled by NORC, using data provided by EDC). 2. Enumerators will review registrars to record current enrollment status of each of the students in the list. 3. Some students in some schools probably graduate from ALP in 2020. Enumerators will see that these students are flagged “potential graduate”. Students that are “Potential graduate” should conduct a survey of the parent of the student if they can’t find the students in the suggested grade/levels. 4. For these students enumerators will consult with the head teacher to confirm students are not enrolled anymore. Enumerators will visit the households of the potential graduates that are not enrolled and conduct the parent survey. 5. For enrolled students, enumerators will record attendance the previous month, current grade/level, and do a roll call, as follows: Student name C12. Currently enrolled? (Y/N)(C_a_1, C_b_1, C_c_1) C13. In what grade/level is [name] enrolled? (C_a_2, C_b_2, C_c_2) C14. Numbers of days attended previous month (C_a_3, C_b_3, C_c_3) Enumerators: If there is no data on attendance write - 99 C15. Numbers of days students were supposed to attend school the previous month. C_a. C_b. C_c. Enumerator: After recoding age distribution and enrollment, please answer: C16. Enrollment records are available for all consulted classes? 1 Yes 104 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 2 Only for some classes 3 No enrollment records available C17. Enrollment records are discriminated by age? [PROG: If C16=1 OR C16==2] 1 Yes 2 No C18. Enrollment records are discriminated by sex? [PROG: If C16=1 OR C16==2] 1 Yes 2 No C19. Attendance records at the student level are available for all consulted classes? 1 Yes 2 Only for some classes 3 No attendance records at the student level are available Thank you very much for helping with this research by sharing your experience as a principal. End time (Autofill) f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 105 Liberia AQE: Teacher/Facilitator Questionnaire Start Date and Time (Autofill): Enumerator Name: Enumerator Code: DISTRICT : ________________________________ VILLAGE : ________________________________ NAME: _______________________ NAME: _________________________ SCHOOL CLUSTER ID: CODE |__|__|__| NAME: _________________________ SCHOOL ID: CODE |__|__|__|__| NAME: _________________________ DATE, TIME, AND LOCATION: _________________________________________________ GIS LOCATION OF SCHOOL (IF GIS UNAVAILABLE) LATITUDE (N/S) |__| – DEGREES: |__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| LONGITUDE (E/W) |__| – DEGREES: |__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| Section A: Introduction and Consent Enumerator Instructions: Enumerator, read the script below in its entirety. Hello, my name is [enumerator, please mention your name]. I am working with The Khana Group, a data collection firm based in Monrovia, on a study to try to understand how to improve accelerated learning in Liberia. This evaluation are funded by USAID. We would like to ask you a few questions about your background, your experience as a teacher and your experiences with accelerated education. If you agree to participate in the survey, all the observations and answers that you provide will be kept confidential. We will write a report but your school will not be identified by name nor will you be identified individually by name. To support future research, anonymized data from this study may be released to the public. However, all information that may be used to identify you or your school will be removed prior to public release. You are free to choose not to answer any question and you can stop the interview at any time or ask me to repeat something you did not hear or understand. You can also withdraw from the survey at any time. Stopping the survey, not answering any question or answering the full survey will not impact you negatively in any way, nor will it bring you any additional benefits. However, your honest answers to our 106 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG questions will help us understand better how to improve accelerated learning in Liberia. We greatly appreciate your cooperation. This interview will take about 30 minutes. Do you have any questions for me now? If you have any additional questions about the survey of methodology, you can contact the following person: [Representative from The Khana Group; contact information of representative] Do you agree to participate? [Enumerator: do not read the following questions up until A_1 aloud] consent. Does the teacher/facilitator consent? Yes □ No □ [skip to end of questionnaire🡪] teacher_name. Enter teacher name: _________ teacher_gender. Enter teacher’s gender: _______ [Enumerator: please read the following questions aloud] A_1. What type of teaching certificate do you have? 1 C Certificate □ 2 B Certificate □ 3 AA Certificate □ 4 Other □ Specify_________________ -98 Don’t know □ -99 Refuse □ A_2. What is your highest level of education? 1 Elementary □ 2 Junior High School □ 3 Senior High School □ 4 Associate □ 5 C Certificate □ 6 B Certificate □ 7 AA Certificate □ 8 Bachelor’s degree □ 9 Master’s degree □ 10 Other □ Specify_________________ -98 Don’t know □ -99 Refuse □ A_3. How many years of teaching experience do you have? ______ [Enumerator, please enter number] A_4. Do you live in the school premises? 1 Yes □ 0 No □ [if No, skip to 🡪 A_6] f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 107 -98 Don’t know □ -99 Refuse □ A_5. How long does it take to travel from home to school each day? _ _:_ _ (hr: min) A_6. Is there an Accelerated Learning Program (ALP) Center in this school? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ A_7. Are you an ALP facilitator or ALP teacher? 1 Yes □ 0 No □ [skip 🡪 Section F] -98 Don’t know □ -99 Refuse □ A_8. How many years of experience as an ALP facilitator do you have? ______ [Enumerator, please enter number] Section B: ALP Facilitator B_1. What level or levels of ALP do you teach in this school year? [Enumerator, select all that apply] 1 ALP 1 □ 2 ALP 2 □ 3 ALP 3 □ B_2. Do you keep an attendance record of students in the ALP classes? 1 Yes □ [skip 🡪 B_6] 0 No □ -98 Don’t know □ -99 Refuse □ B_3. Why don’t you keep attendance records? [Enumerator - Allow multiple responses, but do not read options] 1 It is too time consuming 2 School/principal does not require it 3 Too few students attend class anyway 4 I don’t know how 5 I don’t have time for this 6 Other _____ -98 Don’t know □ -99 Refuse □ B_4. Can you show me attendance records of your classes? 1. Respondent produces records □ [skip 🡪 B_6] 2. Respondent does not produce records □ -99 Refuse □ 108 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG B_5. Why can’t you produce them? [Enumerator - Allow multiple responses, but do not read options] 1 I lost them 2 I’m not allowed to show them to anyone 3 I left them at home 4 I can’t find them right now 5 Other _____ -98 Don’t know □ -99 Refuse □ B_6. How often do you develop lesson plans for the ALP classes? 1 Daily □ [skip to B_8] 2 Weekly □ [skip to B_8] 3 Bi-weekly □ [skip to B_8] 4 Monthly □ [skip to B_8] 5 Never 6 Other □ ______ -98 Don’t know □ -99 Refuse □ B_7. Why don’t you develop lesson plans? [Enumerator - Allow multiple responses, but do not read options] 1 It is too time consuming 2 School/principal does not require it 3 I don’t know how 4 I don’t have time for this 5 Other _____ -98 Don’t know □ -99 Refuse □ B_8. Can you show me the current lesson plan? 1. Respondent produces lesson plan □ [skip to B_10] 2. Respondent does not produce lesson plan □ -99 Refuse □ B_9. Why can’t you produce the lesson plan? [Enumerator - Allow multiple responses, but do not read options] 1 I lost it 2 I left them at home 3 I can’t find them right now 4 Other _____ -98 Don’t know □ -99 Refuse □ B_10. Do you have any scheduled time during the school day for lesson planning for the ALP classes? 1 Yes □ 0 No □ f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 109 -98 Don’t know □ -99 Refuse □ Section C. Recruitment [Enumerator, read the following] Now I’d like to ask you a few questions about the time when you became an ALP facilitator. C_1. Before you were an ALP facilitator, were you a teacher in this or other school? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ C_2. How was the process through which you became an ALP facilitator? Did you sit for an interview? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ C_3. Did you sit for written competence assessment? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ Section D: ALP Training D_1. Have you received any ALP training in the last 5 years? 1 Yes □ 0 No □ [Skip to D_11] -98 Don’t know □ [Skip to D_11] -99 Refuse □ [Skip to D_11] D_2. If yes, in which school year or years did you receive this training? [Enumerator, select all that apply] 1 2016-17 □ 2 2017-18 □ 3 2018-19 □ 4 2019-20 □ 5 2020-21 □ -98 Don’t know □ -99 Refuse □ [Enumerator, if respondent answers more than one option on D_2, read the following phrase: “The following questions are about the last time you received this training”] 110 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG D_3. Approximately how many days was this training? _____ [Enumerator, please enter number] D_4. Was this a residential or a cluster training? Recall that a residential training involves travel to a slightly distant center and spending a few days at a different training location, and that a cluster training is shorter, and involves hosting teachers for training from a few schools in close proximity. [Enumerator, select all that apply] 1 Residential □ 2 Cluster □ 3 Other _______________ -98 Don’t know □ -99 Refuse □ D_5. Who provided this training? 1 AQE/EDC program staff □ 2 Staff form an NGO other than EDC 3 Master trainer from the County Education Office 4 Ministry of Education staff □ 5 Other _______________ -98 Don’t know □ -99 Refuse □ Thinking about the training: D_6. Did you learn new things? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ D_7. Did you find it useful? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ D_8. Do you think the training was long enough? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ D_9. Are you implementing the ALP training approach in your ALP classes? 1 Yes □ [Skip to 🡪 D_11] f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 111 0 No □ -98 Don’t know □ [Skip to 🡪 D_11] -99 Refuse □ [Skip to D_11] D_10. Why not? [Enumerator, select all that apply] 1 I have to teach ALP and conventional school at the same time □ 2 Because the conventional system works better □ 3 Because it is too difficult □ 4 Because I need more training □ 5 Because I don’t have enough materials □ 6 Because I don’t have enough time □ 7 Because there are too many students □ 8 Because I have to teach more than one ALP level at the same time □ 6 Other _______________ -98 Don’t know □ -99 Refuse □ D_11. During this academic year, has the school principal or someone else observed you teaching in your classroom? 1 Yes □ 0 No □ [Skip to 🡪 Section E] -98 Don’t know □ [Skip to Section E] -99 Refuse □ [Skip to 🡪 Section E] D_12. How often does someone observe you teaching? 1 Once a year □ 2 Once per semester □ 3 Once every two months □ 4 Once a month □ 5 More often than once a month □ -98 Don’t know □ -99 Refuse □ D_13. Who conducted these classroom observations? [Enumerator: Select all that apply] 1 AQE/EDC staff □ 2 Staff form the County Education Office 3 Staff from an NGO other than EDC 4 Ministry of Education staff □ 5 School principal/principal deputy/vice-principal 6 Staff from a different school 7 Other _______________ -98 Don’t know □ -99 Refuse □ D_14. After the classroom observations, did you receive any feedback about your teaching? 112 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ D_15. How useful was the feedback for you? 1 Very useful 2 Useful 3 Fair 4 Not Useful at all -98 Don’t know -99 Refuse Section E: Instructional Materials E_1. Do you have a printed ALP teacher guide(s) or teacher manual to help guide your classes? 1 Yes □ 0 No □ [skip to 🡪 E_5] -98 Don’t know □ -99 Refuse □ E_2. How often do you use the teacher guide(s)? 1 Daily □ 2 Twice a week □ 3 Once a week □ 4 Rarely □ 5 Never □ -98 Don’t know □ -99 Refuse □ E_3. How much did the teacher guide help you to teach your classes? 1 A lot □ 2 Somewhat □ 3 A little □ 4 Not at all □ -98 Don’t know □ -99 Refuse □ E_4. Are your current ALP lessons on the correct week of the proposed timeline in the Teachers’ Manual? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ E_5. Which subjects are you currently teaching? [Enumerator: Select all that apply] f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 113 1 Language arts □ 2 Mathematics □ 3 Science □ 4 Social studies 5 Other □__________ -99 Refuse □ E_6. What proportion of students have their own ALP workbook [Enumerator, read options aloud] 1 All of them □ 2 Most of them □ 3 About half of them □ 4 Less than half of them □ 5 None of them □ -98 Don’t know □ -99 Refuse □ E_7. How often do you use these students’ workbooks in class? 1 Daily □ 2 A few times a week □ 3 At least once a week□ 4 Rarely □ 5 Never □ -98 Don’t know □ -99 Refuse □ E_8. What is your opinion about how useful these workbooks are for student learning? Would you say they are: [Enumerator, read options aloud] 1 Very useful □ 2 Somewhat useful □ 3 Not very useful □ 4 Not useful at all □ -98 Don’t know □ -99 Refuse □ E_9. More generally, how successful do you think is the ALP in preparing children for transition to regular primary school? [Enumerator, read options aloud] 1 Very successful □ 2 Somewhat successful □ 3 Somewhat unsuccessful □ 4 Unsuccessful □ -98 Don’t know □ -99 Refuse □ E_10. How successful do you think is the ALP in preparing children for transition to junior secondary school? [Enumerator, read options aloud] 114 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 1 Very successful □ 2 Somewhat successful □ 3 Somewhat unsuccessful □ 4 Unsuccessful □ -98 Don’t know □ -99 Refuse □ E_11. How successful do you think is the ALP in preparing children for transition to conventional or regular school ? [Enumerator, read options aloud] 1 Very successful □ 2 Somewhat successful □ 3 Somewhat unsuccessful □ 4 Unsuccessful □ -98 Don’t know □ -99 Refuse □ E_12. How successful do you think is the ALP in preparing children for transition to technical or vocational education? [Enumerator, read options aloud] 1 Very successful □ 2 Somewhat successful □ 3 Somewhat unsuccessful □ 4 Unsuccessful □ -98 Don’t know □ -99 Refuse □ E_13. What do you think should be done to improve the effectiveness of the ALP to help children transition to school? 1 Longer ALP school days □ 2 More facilitators □ 3 More teaching materials □ 4 Compensation for additional teaching time □ 5 More classrooms and other infrastructure □ 6 Make the curriculum more relevant for the children □ 7 Training for parents to enhance parental support □ 8 Other Which?_____________ -98 Don’t know □ -99 Refuse □ E_14. Do new students sit for a placement assessment in this school? 1 Yes □ [skip 🡪 E_16] 0 No □ -98 Don’t know □ -99 Refuse □ E_15. How are new students placed in the right ALP level (how it is decided to which level they are assigned)? [Enumerator, do not read options] f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 115 1 All new overaged students are placed in Level 1 □ 2 According to their age □ 3 A teacher or other school staff interviews the child □ 4 According to what their family says □ 5 Other _______ □ -98 Don’t know □ -99 Refuse □ E_16. Did students in your class or classes do an End-of-Unit assessment for the last complete unit? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ E_17. Did students in your class or classes do an End-of-Year or completion assessment for the last school year? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ E_18. How is it decided if an ALP student can be promoted to higher levels of ALP or can graduate from ALP or can transfer to regular school? [Enumerator, do not read options] 1 An average between End-of-Unit and End-of-Year assessments 2 According to their End-of-Unit 3 According to their End-of-Year 4 Students transition according to their age □ 5 A teacher or other school staff interviews the child □ 6 Teacher’s/Principal’s discretion □ 7 According to what their family says □ 8 Other _______ □ -98 Don’t know □ -99 Refuse □ Section F: COVID During the COVID-related school closures, did you: F_1. Teach online? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ F_2. Teach class to individual students or in small groups? 116 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ F_3. Assigned any homework during the school closed period? 1 Yes □ 0 No □ [skip 🡪 F_7] -98 Don’t know □ -99 Refuse □ F_4. How often was homework provided? 1 Daily 2 A few times per week 3 Weekly 4 Fortnightly 5 Monthly 6 Other____ -98 Don’t know □ -99 Refuse □ F_5. How did you send the homework to your students? 1 Sent to Parent through Email 2 Sent to Parent through WhatsApp 3 Collected from school 4 Advised to watch educational TV / Radio 5 Other____ -98 Don’t know □ -99 Refuse □ F_6. How many students at this school usually returned their homework during school closures? 1 All/Most, 2 More than half 3 About half 4 Less than half 5 None 6 Other____ 7 -98 Don’t know □ -99 Refuse □ F_7. Once schools reopen, did you schedule additional hours for students to catch up? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 117 F_8. Compared to the enrollment level in your class or classes before the pandemic, how many students have come back to school since schools reopened? 1 All □ 2 Most □ 3 More than half □ 4 About half □ 5 Less than half □ 6 A few □ 7 None □ -98 Don’t know □ -99 Refuse □ F_9. Given the recent school closures due to the COVID-19 pandemic, do you think that using the ALP curriculum could be a good alternative for regular/conventional school to make up for the learning loss? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ Section G: Conventional Teacher G_1. Are you a teacher of conventional grades (grades 1 to 6) in this school? 1 Yes □ 0 No □ [skip to 🡪 H_2] -99 Refuse □ G_2. Are you a volunteer teacher? 1 Yes □ 0 No □ -99 Refuse □ G_3. Will you continue to teach next year? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ Section H: ALP and Conventional Hours H_1. How many hours of conventional school classes do you teach in a given week this school year? _____ [Enumerator, please enter number] [PROG: G_1=Yes AND A_7. =No]] -98 Don’t know □ -99 Refuse □ 118 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG H_2. How many hours of ALP classes do you teach in a given week this school year? _____ [Enumerator, please enter number] [PROG: G_1=No AND A_7. =Yes] -98 Don’t know □ -99 Refuse □ H_3. In the classes you teach, are ALP and conventional school classes integrated or separated? [Enumerator, please read responses] [PROG: G_1=Yes AND A_7. =Yes] 1 ALP and conventional classes are always separated 2 ALP and conventional classes are sometimes separated and sometimes integrated 3 ALP and conventional classes are always integrated -98 Don’t know □ -99 Refuse □ H_1. How many hours of exclusively conventional school classes do you teach in a given week this school year? _____ [Enumerator, please enter number] [PROG: H_3=1 OR H_3=2] -98 Don’t know □ -99 Refuse □ H_2. How many hours of exclusively ALP classes do you teach in a given week this school year? _____ [Enumerator, please enter number] [PROG: H_3=1 OR H_3=2] -98 Don’t know □ -99 Refuse □ H_3. How many hours of combined conventional/ALP classes do you teach in a given week this school year? _____ [Enumerator, please enter number] [PROG: H_3=2 OR H_3=3] -98 Don’t know □ -99 Refuse □ Section I: Teacher burnout Section J: Compensation [Enumerator, please read the following: “Now I would like to ask you some questions about your compensation as a teacher or facilitator”] J_1. Do you receive a salary as a teacher of conventional grades in this school? 1 Yes □ 0 No □ [skip to 🡪 J_7] -98 Don’t know □ -99 Refuse □ J_2. How much do you receive per month? [Enumerator, please enter number]_ _ Liberian dollars f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 119 -98 Don’t know □ -99 Refuse □ J_7. Do you receive a salary as an ALP teacher or ALP facilitator in this school? 1 Yes □ 0 No □ -98 Don’t know □ -99 Refuse □ 7.8 How much do you receive per month? [Enumerator, please enter number]_ _ Liberian dollars -98 Don’t know □ -99 Refuse □ Thank you very much for helping with this research by sharing your experience as a teacher. End time (Autofill) 120 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Liberia AQE: TEACHER SURVEY CONTROL SHEET Start Date and Time (Auto fill): Enumerator Name: Enumerator Code: DISTRICT: ________________________________ VILLAGE: ________________________________ NAME: _______________________ NAME: _________________________ SCHOOL CLUSTER ID: CODE |__|__|__| NAME: _________________________ SCHOOL ID: CODE |__|__|__|__| NAME: _________________________ DATE, TIME, AND LOCATION: _________________________________________________ GIS LOCATION OF SCHOOL (IF GIS UNAVAILABLE) LATITUDE (N/S) |__| – DEGREES:|__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| LONGITUDE (E/W) |__| – DEGREES: |__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| VILLAGE/TOWNSHIP/DIRECTIONS TO SCHOOL (RECORD REGARDLESS OF WHERE YOU INTERVIEW): ___________________________________________________________ A. How many ALP facilitators were interviewed? 0 1 2 Skip to END B. Why fewer than 2 ALP facilitators were interviewed? [select all that apply] 1. Facilitator refusal 2. Principal refusal 3. Facilitator absent for extended period 4. Only one ALP facilitator in school 5. There are no ALP facilitators in this school 5. Other ____________________ C. How many conventional teachers were interviewed? 0 f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 121 1 2 Note: If only one facilitator was interviewed, enumerators should interview one conventional teacher; if no facilitator was interviewed, enumerators should interview 2 conventional teachers. This is why we ask different questions depending on how many interviews of each type of teacher were conducted. D. Why no conventional teacher was interviewed? [select all that apply] [PROG: Only if A=1 AND C=0] 1. Teacher refusal 2. Principal refusal 3. Teacher absent for extended period 4. Only one teacher in school 5. Other ____________________ E. Why fewer than two conventional teachers were interviewed? [select all that apply] [PROG: Only if A=0 AND C<2] 1. Teacher refusal 2. Principal refusal 3. Teacher absent for extended period 4. Only one teacher in school 5. Other ____________________ END 122 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG ANNEX IV: FACE-TO-FACE (F2F) SURVEY COVID – 19 RISK MITIGATION STRATEGY Overview In light of the COVID-19 pandemic, its effects on data collection activities and related activities that require face-to-face (F2F) interaction and potential risks for staff, contractors, and data providers, NORC is strengthening existing risk assessment and mitigation for staff, contractors, and data providers (e.g. survey respondents) for USAID-funded data collection activities. The information provided is this section is for informational purposes only. NORC does not, herein or otherwise, purport to offer medical advice with respect to COVID-19. For medical or scientific information, contractors should seek guidance from qualified medical and scientific experts (See Centre for Disease Control - https://www.cdc.gov/coronavirus/2019-ncov/index.html. Information on COVID-19 1. COVID-19 is a highly infectious virus transmitted through the air and off surfaces on which it has settled. The principal exposure pathways are inhaling the airborne virus or touching a virus￾contaminated surface and then touching one’s eyes, nose, or mouth. 2. To prevent exposure to the virus it is essential that pathways into the respiratory system and mucus membranes are blocked. There is some concern that exposure could potentially occur through contact with the eyes. 3. The virus can be transmitted before symptoms appear. As a result, unsuspecting carriers may drive infection rates. It is important to implement measures that reduce the risk of transmission, even among people who have no symptoms of the disease. 4. People with underlying medical conditions (such as immunodeficiency’s, asthma, diabetes, and heart disease) and those who are older are at greatest risk of serious complications. Risk Mitigation Strategy The best way to protect the workforce and the public is to prevent the potential for exposure to the virus wherever possible. In the absence of being able to totally prevent exposure, consultants should follow a hierarchy of controls to manage COVID-19 risks while carrying out the task at hand. These are listed below, from the most effective to least effective mitigation measures: 1. Wear personal Protective Equipment (PPE) to reduce exposure. Examples include: o Wearing a face mask in public spaces and during F2F interaction (while masks offer some protection to the wearer, the largest benefit is to prevent the spread to others; they are therefore most effective when worn by all the parties in the F2F interaction) o We will ensure face to face interaction is done 2 meters apart during all face-to-face data collection activities. The evaluation team recognizes that data collection will take place during the current coronavirus pandemic and, as such, extra precautions will be required. The evaluation team will provide greater oversight during the enumerator training. The trainings will be organized to minimize the risk to attendees and will include additional content on minimizing the risk of spreading the virus to communities. TKG will be required to hold the training at a venue that is preferably in the open to allow for better ventilation or remotely. To help ensure these best practices are followed, and to minimize the field teams’ exposure, TKG will be required to provide masks and soap to all enumerators and alcohol wipes for all tablets and voice recorders. TKG will also call ahead, or use other forms of inquiry, to households to determine if any f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 123 members are currently sick or have a cough prior to conducting a survey. Such households would be rescheduled for the end of the data collection or replaced. The same supplies will be provided to participants in key informant interviews and focus groups, and appropriate distancing measures will be utilized when these are being conducted. Additionally, masks will be provided for all participants. TKG will increase the frequency with which project lead check in with enumerators by phone, to ensure oversight on their work remains. Background recording will occur at any point in the survey and will be included in the consent section. However, neither would be aware of when the recording started or ended. TKG would then review these recordings to ensure all training protocols are being adhered to and to flag any points for follow up. Lastly, given the rapidly changing situations and directives from local and national governments, the evaluation team will maintain frequent communications with TKG and NORC to keep all apprised of progress and assess any changes necessary during the lead up and implementation of the enumerator training and field work. Due to COVID-19, the following protocols will be followed during data collection: The Khana Group will emphasize strong practices around harm reduction: practicing good hygiene, social distancing, wearing masks, and promoting an awareness of symptoms. Our priority is ensuring staff and local communities are not spreading and/or exposed to the virus during the video recording in the various schools. The Khana Group will also comply with local laws and requirements for conducting research in-country, including those related to COVID-19. In the event a local law or requirement conflicts with The Khana Group’s requirements, The Khana Group will contact its Technical Officials. In conclusion: • Individuals are required to inform project lead if and when sick. The enumerators report illness to the project lead. • Avoid touching anything in or around the schools/ classroom sites. • Enumerators will give each person a face mask at the beginning of the interview or discussion and encourage them to utilize it. Participants will be required to wear a mask for the duration of the interview. If the respondent cannot or is unwilling to wear a mask, then the respondent will not be allowed to participate. • Enumerators will wash or sanitize their hands after every interaction • Enumerators will practice social distancing, and conduct interviews and discussion outdoors whenever possible. Enumerators will be asked to report COVID-19 symptoms or exposure to suspected or confirmed cases of COVID-19 at any time during the video recording immediately to the Project Lead. If an enumerator gets in contact with a sick / confirmed COVID-19 case, such enumerator will be asked to self-isolate for 14 days as recommended by the government of Liberia. If the person comes down with severe symptoms, such person will be taken to the 14th military hospital where a test will be ordered, and they will be quarantined and provided with medical care. 124 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG ANNEX V: USING ADMINISTRATIVE DATA TO ANALYZE SURVIVAL IN THE AQE PROGRAM In this annex we discuss how we processed EDC’s administrative data to produce a panel of students to analyze survival in the AQE program, as discussed in section 4.4. EDC’s data include the list of students that enrolled at the beginning of the school year during school years 2017-18, 2018-19 and 2019-20; we call these the intake data. EDC also has the list of students that sat for the end-of-year or completion assessments; we call these the completion data. Assuming that sitting for a completion assessment is a valid proxy for enrolment, these data can be used to estimate inter-annual retention rates. For example, we can estimate the retention rate from year 2017-2018 to 2018-2019 by looking at how many of the students enrolled in 2017-2018 sat for the 2018-2019 completion assessment. In principle, all students that sat for a completion assessment should show up in the intake data, either for that same year or a previous year. However, this is not the case. As Table 29 shows, the fraction of students that sat for completion assessments that are not in the intake data increases over the years. In 2017-2018 161 students, equivalent to 7.4 percent of the students that sat for that year’s completion assessment, do not show up in the 2017-2018 enrollment data. In 2018-2019, 3,713 students that sat for that year’s completion assessment (27.3 percent) do not show up in the enrollment data of 2018-2019 or 2017-2018. And in 2020 that figure is 19,140 (50.2 percent). Table 29. Number of students in completion data not in intake data Students not in intake data Students in completion data % No. schools 2017-18 161 2164 7.4 74 2018-19 3713 13588 27.3 180 2019-20 19140 38115 50.2 261 For the analysis on completion, pass and retention rates discussed in section 4.4 we only use students that show up in the intake data and drop all the students that are in the completion data only. If we included students that show up in the completion data only, we would overestimate completion rates, unless we could be sure that all students that do not show up in the intake data, but were enrolled at some point, are accounted for in the completion data. A major caveat to these data is that the completion assessment data for 2019-2020 does not include a student id (as do the completion assessment data for 2018 and 2019 and the enrollment data for all three years). To complete the student panel and being able to estimate retention up to school year 2019-2020, we used the names and schools of the students that sat for the 2019-20 completion assessment to merge them to the directory of students with an id number (these include the list of all students that enrolled during school years 2017-18, 2018-19 and 2019-20, and the students that sat for a completion assessment in 2017-18 and 2018-19). Merging by name is problematic because spelling errors can affect the likelihood that we can identify all the students that are both in the enrollment data and in the 2019-2020 completion data. Although we used probabilistic matching to maximize the number of students that we can find, we cannot guarantee f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 125 that we identify all the students that are in both data sets (and because we are using probabilistic matching, we may be linking students that are not actually the same person; however, given that we are using schools in the merging process, this problem seems relatively minor). The main implication of this is that we may be underestimating retention from 2018-2019 to 2019-2020, because we fail to merge all the students form the completion to the intake data. 126 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG ANNEX VI: SAMPLING WEIGHTS OR THE FOLLOW-UP SURVEY To inform retention rates during the transition year, NORC surveyed the enrollment status of a sample of students in AQE schools. The student sample frame includes the students that sat for the 2018-19 completion assessment and the students that started the ALP program during the school year 2019-20. Given the available data, this is the most complete account of the students that were enrolled in school year 2019-20.17 The sampling scheme followed a two-stage design. First, we sampled 90 AQE schools. Second, in each sampled school we sampled 10 ALP students (900 students in total). This sample allows estimation of a promotion rate of 65 percent with 95 confidence interval and a 4.5 percent margin of error.18 There are 281 schools in the enrollment data file provided by EDC. We dropped 19 schools that have no students in completion 2018-19 data or the 2019-2020 enrollment data, for a final sample frame of 262 schools with 32,422 students. Schools are stratified by cohort and county, so there are 30 schools per cohort and 15 schools per county. In addition, 46 replacement schools are selected. We sample 10 students in each school. Because we are using the 2018-19 completion assessment data and the list of students that started the ALP program during the school year 2019-20, we can approximate whether each sampled student should still be in school (either conventional primary school or ALP), or if s/he may have graduated from ALP.19 For the students that are supposed to still be in school we used schools’ registrars to determine the school enrollment status of these children. For potential ALP graduates we first looked for them in the registrars (in case they failed to pass a level or grade), and if we cannot find them, we conducted a household interview with their caregiver to document whether the student is enrolled in school (ABE or secondary school), or what is s/he currently doing. We oversampled potential graduates because only 6 percent of students in the sample frame are potential graduates. Therefore, in each school we sampled up to 5 potential graduates. After that for each school more students were sampled until there were 10 sampled students in each school, for a total of 900 students. Four schools were replaced. In the 90 surveyed schools enrollment status was reviewed for 900 students, including 200 potential graduates. Sampling weights were constructed to recover population parameters. For potential graduates we calculated sampling weights for students i in school s and stratum r following: 17 An alternative approach would have been using the 2019-2020 completion assessment data as sample frame. However, these data do not have students’ gender or age, and because it also does not have student id number, we could not use these data to analyze how gender and age are correlated with retention in the follow-up survey. 18 This assumes a design effect of 1.9, associated to an intracluster correlation (ICC) of 0.1. The estimate for the ICC and expected promotion rates come from AQE data for school year 2019-2020 provided by EDC. 19 For example, if a student enrolled in ALP Level 2 in school year 2019-20, currently s/he should be in level 2 if s/he failed the 2019-20 completion assessment, or in ALP 3 if s/he passed. In addition, the student could had been transition to primary school (if the ALP center closed, for example) so this student could be in grade 3, 4 or 5 of the conventional primary school. f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 127 𝑤𝑟𝑠𝑖 = 𝑁𝑟 ∑ 𝑁𝑟𝑗𝟏[𝑆𝑟𝑗 = 1] 𝑗 𝑁𝑟𝑠 𝑛𝑟𝑠 where 𝑁𝑟 is the number of potential graduates in strata r, 𝑁𝑟𝑠 is the number of potential graduates in school s of strata r, 𝑆𝑟𝑗 is a dummy variable for whether school j of strata r was surveyed, and 𝑛𝑟𝑠 is the number of potential graduates surveyed in school s of strata r. An analogous formula was used to calculate weights for non-potential graduates. 128 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG ANNEX VII COST SIMULATION UNDER MINISTRY OF EDUCATION ALP IMPLEMENTATION We assumed that at the time of opening 150 students would join Level 1of the ALP program and 35 new students would join in each of the following years, for a total of 465 unique students enrolled at least one year. This assumption is based on the data from AQE Cohort 2 in 2019 when on average 144 students enrolled in ALP centers and 29 were received the following year. We rounded up given that COVID-19 increased the number of out of school children. We also assumed that each year, 40 percent of the students are promoted to the next ALP level (or graduate if they were in Level 3), twelve percent repeat the ALP level, twenty percent are transferred to conventional school and the rest (28 percent) drops out. This amounts to a total of 830 student-years. Again, these assumptions are based on data from the AQE program. The simulated number of students in each year is shown in the top panel of the table below . The second panel in the table shows 3 facilitators allocated to different ALP levels over time. ALP center simulation – Number of students and facilitators Year Total 1 2 3 4 5 6 7 8 9 10 Students L1 150 53 41 40 40 40 40 40 40 40 L2 60 28 20 18 18 18 18 18 18 L3 24 14 10 9 8 8 8 8 465 students Total 150 113 94 74 68 66 66 66 66 66 830 student-years Facilitators L1 3 1 1 1 1 1 1 1 1 1 L2 2 1 1 1 1 1 1 1 1 L3 1 1 1 1 1 1 1 1 3 facilitators Total 3 3 3 3 3 3 3 3 3 3 f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 129 Cost Simulation ALP Center – Three Facilitators and 830 student-years YEAR 1 2 3 4 5 6 7 8 9 10 TLMs & Assessments Workbooks and assessments L1 $ 3,999 1,413 1,103 1,065 1,061 1,060 1,060 1,060 1,060 1,060 L2 $ 2,108 998 701 646 637 635 635 635 635 L3 $ 843 500 340 299 291 289 289 289 Total 3,999 3,521 2,943 2,267 2,047 1,996 1,986 1,984 1,984 1,984 $24,712 Facilitators' guides L1 $ 78 26 26 26 L2 $ 70 35 35 L3 $ 35 35 35 78 70 35 26 35 35 26 35 35 26 $401 Training 1 replacement facilitator 290 Total 4,077 3,591 2,978 2,293 2,082 2,031 2,012 2,019 2,019 2,300 $25,403 $30.6 student-year Training 3 new facilitators training $ 870 $ 870 With training for 3 facilitators + replacement 4,947 3,591 2978 2,293 2,082 2,031 2,012 2,019 2,019 2,300 $26,273 $31.7 student-year SALARIES Facilitator salary $50*10 months plus $1.75 fee 1,553 1,553 1,553 1,553 1,553 1,553 1,553 1,553 1,553 1,553 $15,525 With training for 3 facilitators + replacement + $50 salary 6,500 5,143 4,531 3,845 3,634 3,584 3,565 3,572 3,572 3,853 $41,798 $ 50.4 student-year Facilitator salary $50*10 months plus $1.75 fee 1,103 1,103 1,103 1,103 1,103 1,103 1,103 1,103 1,103 1,103 $11,025 With training for 3 facilitators+replacement + $35 salary 6,050 4,693 4,081 3,395 3,184 3,134 3,115 3,122 3,122 3,403 $37,298 $ 45.0 student-year 130 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Facilitator guides and workbook are divided in two parts that cost $4.36 each. This cost is based on the direct costs of materials production and distribution, minus the amount devoted to tablets and to assessment materials, divided by the total number of materials rounded up to 185,000. In level 1, facilitators need three guides and students need three workbooks (literacy, numeracy, and life skills/learning together). For levels 2 and 3 four guides and four workbooks are needed (literacy, numeracy, science, and social studies). The students’ assessments estimated costs are $0.25 each. One assessment is need upon entrance into the program and one assessment is needed at the end of each year, therefore we assume 2 assessments in the first year and one in the following year. The cost of each assessment was estimated as a 3/50 proportion of each workbook part. The estimated costs of fully training a facilitator are $290 (the training is 10 days at $29 per day; based on the direct costs of training in 2018/19 and the number of people-days of training 33,140, which include teachers and principals, GoL staff, and PTA members); currently the MoE is running the program with trained facilitators that are in the payroll and does not offer any additional stipend. We assumed that one of those facilitators will need to be replaced. If a new ALP center opens, training of new facilitators will be necessary and therefore we costed training three facilitators in Year1 and kept one in Year 10, as replacement, AQE used to compensate facilitators at $50 per month for the stipend (overtime). There was a $1.75 fee for each transaction. When we include facilitators payment to consider the possibility of running the ALP in the afternoons, we offered two alternatives, paying $50 or $35. f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 131 ANNEX VIII: CONFLICT OF INTEREST (COI) FORMS 132 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 133 Name Cathy Zimmerman Title Senior Qualitative Expert/Senior Research Scientist Organization NORC at the University of Chicago Evaluation Position? Team Leader ☐ Team Member ☒ Evaluation Award Number (contract or other instrument) GS-10F-0033M/AID-OAA-M-13-00010, Tasking N027 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Liberia Accelerated Quality Education (AQE) Evaluation I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 7 October 2021 134 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 135 Name Ninar Taha Title Research Analyst Organization NORC at the University of Chicago Evaluation Position? Team Leader ☐ Team Member ☒ Evaluation Award Number (contract or other instrument) GS-10F-0033M/AID-OAA-M-13-00010, Tasking N027 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Liberia Accelerated Quality Education (AQE) Evaluation I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 10/07/2021 136 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG Name Mithila Iyer Title Research Assistant Organization NORC at the University of Chicago Evaluation Position? Team Leader ☐ Team Member ☒ Evaluation Award Number (contract or other instrument) GS-10F-0033M/AID-OAA-M-13-00010, Tasking N027 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Liberia Accelerated Quality Education (AQE) Evaluation I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 11/10/2021 f USAID.ORG EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) | 137 138 | EVALUATION OF THE ACCELERATED QUALITY EDUCATION FOR LIBERIAN CHILDREN ACTIVITY (AQE) USAID.ORG U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, DC 20523