February 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by CAMRIS International, Inc. EARLY GRADE READING PROGRAM PERFORMANCE EVALUATION Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 ii EARLY GRADE READING PROGRAM PERFORMANCE EVALUATION 2019 FINAL Report February 2020 Contract Number: AID-367-C-15-00001 DISCLAIMER This publication is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 iii TABLE OF CONTENTS Acronyms and Abbreviations vi Executive Summary 9 1 Evaluation Purpose 14 2 Project Background 14 3 Evaluation Questions And Methodology 18 3.1 Methodology 18 3.3 Data Collection, Management, and Analysis 22 3.4 Methodology Limitations 23 4 Findings 24 4.1Evaluation Question (EQ)1. To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? 26 4.1.1 Viability of EGRP: What Components are Working Well? 26 4.1.2 Viability of EGRP: What Components are Not Working So Well? 27 4.1.3 Viability of EGRP: Helping Factors in the School Environment 29 4.1.4 Viability of EGRP: Hindering Factors in the School/Education Environment 30 4.1.5 Sustainability of EGRP: Factors in the Environment That Can Help 33 4.1.6 Sustainability of EGRP: Factors in the Environment That Hinder 34 4.1.7 Scalability: Helping 36 4.1.8 Scalability: Hindering 37 4.2 EQ2. To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? 38 IR 1.1 Evidence-based EGR instructional materials designed, distributed, in use 39 IR1.2 In-service professional development for teachers in public schools on reading instruction and materials in use 39 IR1.3 Monitoring and coaching for teachers in early grade reading instruction. 40 IR 2.1 Early grade data collection and analysis systems 40 IR 2.2 Policies, standards, and benchmarks that support improved EGR 40 IR 2.3 Planning and management of financial, material and human resources 41 IR 2.4 Utilization of data for course correction during implementation 41 IR 2.5 Classroom-based and district-based EGRA processes 41 4.3 EQ3. To what extent were findings and recommendations of the 2016 report addressed? 43 4.4 EQ4. To what extent are local governments (LGs) within the EGRP districts able to continue and sustain EGRP interventions? 44 Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 iv 4.5 EQ5. To what extent has the MOEST and CLAs built up the capacity at the central level to support the national scale-up of the NEGRP minimum packages at the federal and local levels? 46 5. Conclusions and Recommendations 47 5.1 EQ1. To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? 47 5.1.1. Teaching and Learning Materials 47 5.1.2. Teacher Training and Classroom Support 48 5.1.3. Community Mobilization 49 5.1.4 Student Assessment 50 5.1.5. Monitoring and Evaluation 51 5.1.6 Other 51 5.2 EQ2. To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? 52 5.3 EQ3. To what extent were findings and recommendations of the 2016 report addressed? 52 5.4 EQ4. To what extent are LGs within the EGRP districts able to continue and sustain EGRP interventions? 53 5.5. EQ5. To what extent have MOEST and CLAs built up the capacity at the central level to support the national scale-up of the NEGRP minimum packages at the federal and local levels? 53 5.5.1 Building Institutional Capacity at the Central Level 53 5.5.2 Scaling up NEGRP 54 Appendix 1: Evaluation Statement of Work 55 Appendix 2: Key Documents from 2017 Baseline Report 58 Appendix 2a: Building Institutional Capacity 58 Appendix 2b: End of Project Status: Institutional Capacity Built 62 Appendix 2c: Detailed Methodology 67 Central Level Stakeholders 71 Table 10: Informants at the Central Level 71 Regional, District, and Cluster-level Stakeholders 72 Table 11: Exploring Variability, Viability, Sustainability, and Scalability 72 Table 12: Informants at the Regional, District and Cluster Levels 72 School-level Stakeholders 73 Table 13: Informants at the School Level 73 Data Collection, Management, and Analysis: 74 Appendix 3: Data Collection Instruments 75 Appendix 4: MTR BRIEF 140 Appendix 5: Disclosure of Any Conflicts of Interest 144 Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 v Appendix 5: Sample of Excerpts, by Evaluation Question, Theme, and Stakeholder Level 145 Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 vi ACRONYMS AND ABBREVIATIONS AIN Association of International NGOs BCC Behavior Change and Communication CAS Continuous Assessment System CB-EGRA Classroom-based Early Grade Reading Assessment CDC Curriculum Development Center CDCS Country Development Cooperation Strategy CEHRD Centre for Education and Human Resource Development CFE Child-Friendly School CLA Central Line Agencies DCC District Coordination Committee DEO District Education Officer DoE Department of Education EFA Education for All EDCU Education Development Coordination Unit EGR Early Grade Reading EGRA Early Grade Reading Assessment EGRP Early Grade Reading Program ELT English Language Teaching ERO Education Review Office ETC Education Training Center FGD Focus Group Discussion G2G Government-to-Government GON Government of Nepal GPE Global Partnership for Education HT HT ICT Information and Communication Technology IR Intermediate Result LEU Local Education Unit LC Language Commission LG Local Government (Palika) M&E Monitoring and Evaluation MA Master of Arts MEL Monitoring, Evaluation, and Learning (USAID/Nepal Project) MOE Ministry of Education MOEST Ministry of Education, Science, and Technology MOI Medium of Instruction MP Minimum Package MT Mother Tongue MTR Midterm Review NA Not Applicable/Available NARN National Assessment of Reading and Numeracy NARMIN National Association of Rural Municipalities in Nepal NCED National Center for Educational Development NEGRP National Early Grade Reading Program NGO Non-governmental Organization NMP NEGRP Minimum Package Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 vii PAD Project Appraisal Document PEU Province Education Unit PMP Performance Management Plan PTA Parent Teacher Association RC Resource Center RCA Rapid Classroom Assessments RM Reading Motivator RP Resource Person SCM Social and Community Mobilization SIP School Improvement Plan SMC School Management Committee SRM Supplementary Reading Material SS School Supervisor SSDP School Sector Development Plan SSRP School Sector Reform Program TLM Teaching Learning Materials TPD Teacher Professional Development TPS Teacher Professional Support TSC Teachers Service Commission UNICEF United Nations Children’s Fund USAID United States Agency for International Development VSO Voluntary Service Overseas WFP World Food Programme Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 viii EVALUATION TEAM MEMBERS Colette Chabbott, PhD, Team Leader and Field Team Leader 1 Badri Dev Pande, PhD, EGRP Expert and Field Team Leader 2 Research Associates and Coders: Rashmi KC, MA Bipana Sharma, MA School-level Data Collectors: Nishu Aryal Maya Chhetri Pankaj Sharma Sumana Devkota CAMRIS Support Manorama Adhikari, Deputy Chief of Party Ganesh Sharma, Data Analyst/Statistician Swadesh Gurung, M&E Specialist Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 9 EXECUTIVE SUMMARY Evaluation Purpose This longitudinal evaluation focuses on the extent to which the Government of Nepal (GON) is gaining the capacity to sustain the Early Grade Reading Program (EGRP) over time and to scale the program to non￾program districts. The evaluation has assessed progress toward intermediate result (IR) 2 and related aspects of IRs 1 and 3 under EGRP, with a focus on improved GON capacity for early grade reading service delivery and a focus on the enabling conditions to sustain the Early Grade Reading Program in the original 16 districts (EGRP) and to scale up to 61 other districts (National Early Grade Reading Program, NEGRP) (see Figure ES-1). The 2017 EGRP Performance Evaluation Baseline Study1 of the Ministry of Education (MOE) institutional capacity2 laid out the basic metrics that are used to assess progress towards viability, scalability, and sustainability. Initially planned in the last year of the project as an end-line study, this follow-up evaluation was moved up one year so its findings could inform a) the Mid-Term Review of the School Sector Development Program (a sector-wide approach or SWAP) in May 2019 and b) the work plan for the final year of EGRP (2018-2019). The findings will help the Mission and the project implementer to understand the impact of federalization on institutional capacity building and to adapt the last year of the project to improve project performance. Project Background The GON’s School Sector Reform Program (SSRP) guided improvement in the accessibility and quality of primary education in Nepal 2009-2015. In 2014/15, Nepal’s MOE drafted a two-year extension of SSRP to align with the Education for All (EFA) timelines. A key component of the SSRP was a five-year, $71 million National Early Grade Reading Program (NEGRP) (2014/15–2019/20). The goal of the program is to improve reading outcomes in Nepali for children enrolled in grades 1–3 of all community (i.e., government-funded) schools across all 77 of Nepal’s districts. The GON has continued to prioritize NEGRP in the School Sector Development Plan (SSDP, 2016-2023) and the multi-donor Joint Financing Arrangement. The United States Agency for International Development (USAID) has a five-year, $53.8 million contract with RTI International to provide technical assistance to the GON’s NEGRP. The overarching goal of 1 CAMRIS (2017). Early Grades Reading Program Performance Evaluation Baseline Study, Kathmandu, Nepal: CAMRIS International, May 2017. 108. Retrieved from: https://pdf.usaid.gov/pdf_docs/PA00MRM4.pdf 2 Field work for the 2017 study was conducted in August and September 2016. Figure ES-1: EGRP – Intermediate IR 1: Improved early grade reading instruction IR 2: Improved national and district early grade reading service delivery IR 3: Increased family and community support for early grade reading Reading Skills Improved: Public primary school students in grade 1–3 in the 16 target districts with improved reading skills; GON Services Strengthened: The Contractor will have supported the GON through Phase I of the Nepal EGRP and completed the design and demonstration of a national model that the GON can then implement nationwide within its budget. Figure ES-2: EGRP Goals Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 10 EGRP is to improve reading skills for public primary students in grades 1–3 in 16 target districts. In support of this goal, EGRP partners are providing technical assistance to the GON that aims to develop an early grade reading (EGR) model that the GON can implement nationwide within its budget (Figure 2). EGRP is designed to strengthen the Ministry of Education, Science, and Technology’s (MOEST’s) (previously the MOE) capacity to advance and implement policies that aim to institutionalize and sustain pedagogical and curricular reforms for improved EGR instruction. As such, EGRP is first and foremost a capacity-building program. Evaluation Questions and Methodology This performance evaluation answers five questions posed by USAID using qualitative, longitudinal methods, guided by a methodology designed in collaboration with USAID, RTI, and non-governmental organization (NGO) implementing partners during the baseline study in August 2016. By February 2019, however, when this follow-on study began collecting data, changes in the project design and major changes in the organization and staffing of the education sector – mainly federalization – necessitated changes in interview subjects at the provincial level and below and in measuring institutional capacity built at the collaborating, learning, and adapting (CLA) level. The team gathered largely qualitative data in Kathmandu, and in six districts in two provinces, sampling EGRP districts and non-EGRP control districts. Within those districts, a purposive sample was taken of 24 schools from February 13 – April 9, 2019. The evaluation team conducted more than 250 individual interviews, focus group discussions (FGDs), and rapid classroom assessments (RCAs), at the school (24), palika/municipality (13), district (4 EGRP-assisted, 2 control), provincial, and CLA levels. Most of these interviews were recorded in Nepali by native Nepali speakers, subsequently translated into English and coded using Dedoose software. This produced more than 3,400 excerpts relevant to the evaluation questions. The team also supplemented this primary data with secondary data from documents and quantitative data from RTI, the GON, and the Nepal press. As the study is not representative of all 16 EGRP districts, the conclusions and recommendations below the CLA level are, therefore, indicative and provide a snapshot of EGRP. Findings, Conclusions, and Recommendations Question 1: To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? As of March 2019, EGRP had demonstrated several viable components of a model for increasing EGR skills in the 30-50 percent of schools in the six sample districts. Stakeholders in schools expressed satisfaction with Nepali teaching and learning materials and teacher training. EGRP components for providing necessary in-classroom teacher support and essential community mobilization had been adapted for the current (2018/19) school year and remained to be evaluated. Both teacher support and community mobilization components appeared to continue to need more adaptation to the new federal context. The student assessment component also needs strengthening at the school level. Based on the sample, no model for school improvement is likely to be effective if schools do not have sufficient, motivated, Nepali-speaking teachers; sufficient classrooms; and regular attendance by teachers and students. Mother tongue materials are necessary to support non-Nepali teachers and students but Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 11 have yet to become widely available through the EGRP program. Lastly, the orientation, training, and support processes for the monitoring and evaluation component of the EGRP minimum package have been disrupted and cannot be finalized until the Federal Education Act is passed and implementation for basic education is allocated across the CLA, provincial and local levels. The report includes 15 recommendations for strengthening the implementation of the EGRP minimum package model, organized around the five components of the NEGRP Minimum Package (NMP). A 16th recommendation suggests securing funding for ongoing technical assistance in the 20 EGRP districts for at least three additional years. Question 2: To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? As of March 2019, no EGRP IRs or sub-IRs have achieved “institutionalized” status (i.e., been completely integrated into standard operating procedures and recurring budgets and able to sustain themselves without special initiatives and project funding). Several sub-IRs have achieved “established” status per the detailed scales in Appendix 3 of the baseline study. However, due to the changed organization of the education sector, this no longer represents readiness to fully support the scale-up of an effective NMP at the federal and local levels. Most are closer to “initially established,” that is, some aspects of the sub￾IR have been implemented once or twice, but capacity is not firmly established and/or policies and guidelines necessary to support the sub-IR remain undeveloped. The status of the various sub-IRs is estimated, as shown in Exhibit I: Exhibit I: Progress on Strengthening GON Service Delivery Sub-IRs Progress Categorization Latent Emerging Initially Established Institutional￾ized IR 1: Improved reading instruction, central-level agencies, and number of districts CDC: Evidence-based EGR instructional materials designed, distributed, and in use 2016 2019 mother tongue 2019 Nepali CEHRD (formerly NCED), ETCs: in-service professional development for teachers in public schools on reading instruction and materials in use 2016 2019 Monitoring and coaching for teachers in EGR instruction (formerly RCs) 2016 2019 IR 2: GON capacity to implement EGRP EGR data collection and analysis systems 2016 2019 Policies, standards, and benchmarks that support improved EGR 2016 2019 Planning and management of financial, material, and human resources 2016 2019 (human resources) 2019 (financial & material) Utilization of data for course correction during implementation 2016 2019 Classroom-based & district-based EGRA processes 2016 2019 IR 3: Increased family and community support, number of districts* Community awareness of the importance of the language of reading instruction 2016 2019 Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 12 Family engagement to support reading 2016 2019 PTA/SMC contributes to quality reading instruction 2016 2019 Parent and community capacity to monitor reading progress 2016 2019 The main recommendations include updating the scales for measuring EGRP institutional capacity building (2017 Baseline Report, Annex 3 and 4), building appreciation across levels of the education implementers for the percentage of students achieving early reading benchmarks as the primary evidence of school quality, and providing technical assistance to the Community Development Center (CDC) to develop a plan to improve production and speed the development of mother-tongue materials and utilize relevant expertise. Question 3: To what degree have the findings and recommendations of the baseline performance evaluation been addressed? Exhibit II: Threats Identified in 2016 and Status in 2019 by Threat Category 2016: THREATS IDENTIFIED 2019: STATUS VIABILITY Enthusiasm for NEGRP (GON strategy) is strong, but the understanding of EGR (how to teach reading) is weak Improved, but still an issue Parents demand English medium, less Nepali, no Mother Tongue (MT) No progress School-level stakeholders do not identify learning as the most important indicator of school quality No progress Threats to time on task for EGR No progress Good Teaching Learning Material (TLM) use dependent on better incentives for Teacher Professional Support (TPS) provider No progress Delayed disbursement of Government-to-Government (G2G) funds undermining GON goodwill. Improved The Curriculum Development Center (CDC) needs strengthening in child psychology/development, children’s literacy, and mother-tongue based instruction No progress Undervaluing of EGR groundwork laid by NGOs Improved. The utilization of NGO work is still weak. Resource Persons (RPs) and teachers value the Continuous Assessment System (CAS) and Child-Friendly System (CFS) but do not recognize EGRP as a method to address both Minimal progress SUSTAINABILITY Additional funds needed from external donors Some progress No School Improvement Plans (SIPs) imply little local fund mobilization Much improved SCALABILITY Need coaches compatible with the existing system Experiments continue w/ HTs No small-scale research to test necessary variations in the model No progress Lack of appreciation for evidence-based innovation Need simpler, more compelling arguments for focus on reading Minimal progress Over-reliance on CB-EGRA to convince the public of the value of EGR No national champions Lacking clear relative advantage vis-a-vis the existing system Cost-effectiveness of EGRP vs. the existing model is not widely understood Lacking champions in or out of the GON Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 13 No nation-wide behavioral change communication (BCC)/social mobilization Reading evidence from other countries is not compelling No local research Minimal EGRP in Nepali press Vision/implementation slips in response to normal attrition of teachers and GON staff No improvement Note: Question asked was, “To what extent were findings and recommendations of the 2016 baseline report addressed?” Question 4: To what extent are the Local Governments (LGs) within the EGRP districts able to continue and sustain EGRP interventions? EGRP was designed to build institutional capacity in various agencies, including the MOEST at the CLA level and around District Education Officers (DEOs), Resource Centers (RCs), and Education Training Centers (ETCs) at the district level. Federalization eliminated DEOs and RCs, reduced the number of ETCs to seven (one per province), established Education Development Coordination Units (EDCUs) but limited their mandates, budgets, and staff, and, thus, dismantled many of the district and local level structures that EGRP had focused their capacity-building efforts on for the first two years of the program. For less than a year, EGRP has been providing orientation and training designed to help elected LG officials and local level education units (LEUs) to play a greater role in supporting and sustaining EGRP interventions. The main recommendation is that, while waiting for the Federal Education Act to finalize LG responsibilities in implementing EGRP/NEGRP, new EGRP program officers should draft an LG￾focused institutional capacity strengthening plan, including a pilot orientation, training, and other activities and incentives to participate in those activities. Question 5: To what extent have MOEST and CLAs built the capacity at the central level to support the national scale-up of the NEGRP minimum packages at the federal and local levels? This question was largely addressed under Question 2 above, and here produced two additional recommendations. First, prepare to pick up speed in the establishment and institutionalization of MOEST and CLA capacity after the passage of the Federal Education Act and the Education Law. Second, encourage updating of the NEGRP strategy to confirm for the operations level MOEST and CLA staff the highest level of commitment to NEGRP and the value of the extreme effort it will require to achieve it. Finally, a nationwide initiative demanding major sacrifices and heroic efforts from an over-extended education system and civil service suggests a major social mobilization campaign may be needed to convert stakeholders at all levels to a different metric for school quality. A campaign also is necessary to convince stakeholders that reading and writing in the early grades in the mother tongue (whether Nepali or another language) is the best foundation for all future learning, including the English language. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 14 1 EVALUATION PURPOSE This longitudinal evaluation focuses on the extent to which the Government of Nepal (GON) is gaining the capacity to sustain the Early Grade Reading Program (EGRP) and scale the program to non-program districts. The evaluation has assessed progress toward intermediate result (IR) 2 and related aspects of IRs 1 and 3 under EGRP, with a focus on improved GON capacity for early grade reading service delivery and on the enabling conditions to sustain the Early Grade Reading Program in the original 16 districts (EGRP) and scale up to 61 other districts (National Early Grade Reading Program, NEGRP) (see Figure 1). The 2017 EGRP Performance Evaluation Baseline Study3 of the Ministry of Education’s (MOE) institutional capacity laid out the basic metrics that are used to assess progress towards the viability, scalability, and sustainability of the EGRP. Initially planned in the last year of the project as an end-line study, this follow-up evaluation was moved up one year, in order for its findings to inform the Mid-Term Review of the School Sector Development Program (SSDP), a sector-wide approach, in May 2019 and the work plan for the final year of EGRP (2018-2019). The findings also are expected to help the Mission and the project implementer to understand the impact of federalization on institutional capacity building and to adapt the last year of the project to improve project performance. Consistent with USAID’s collaborating, learning, and adapting (CLA) approach, adaptation of the program will require close consultations with government and donor partners as well as with the local and international non￾governmental organizations (NGOs) that have played a unique role in piloting early grade reading (EGR) activities in different contexts within Nepal. 2 PROJECT BACKGROUND The GON’s School Sector Reform Program (SSRP, 2009-2015) guided improvement in the accessibility and quality of primary education in Nepal. In 2014/15, Nepal’s MOE drafted a two-year extension of SSRP to align with Education for All (EFA) timelines. A complementary component of the SSRP was a five￾year, $71 million NEGRP (2014/15–2019/20). The goal of the program is to improve the reading outcomes in Nepali for children enrolled in grades 1–3 of all community (i.e., government-funded) schools across all 77 of Nepal’s districts. The GON has continued to reflect the NEGRP as a school education priority in the School Sector Development Plan and the multi￾donor Joint Financing Arrangement. 3 CAMRIS (2017). Early Grades Reading Program Performance Evaluation Baseline Study. Kathmandu, Nepal: CAMRIS International May 2017. 108. Retrieved from: https://pdf.usaid.gov/pdf_docs/PA00MRM4.pdf. Figure 1: EGRP – Intermediate Results IR 1: Improved early grade reading instruction IR 2: Improved national and district early grade reading service delivery IR 3: Increased family and community support for early grade reading Figure 2: EGRP Goals • Reading skills improved: Public primary school students in grades 1–3 in the 16 target districts with improved reading skills. • GON services strengthened: The Contractor will have supported the GON through Phase I of the Nepal EGRP and completed the design and demonstration of a national model that the GON can implement nationwide within its budget. Source: EGRP Year 4 Annual Work Plan. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 15 The United States Agency for International Development (USAID) has a five-year, $53.8 million contract with RTI International to provide technical assistance to the GON’s NEGRP. In the first years of technical assistance provided under EGRP, five national NGO partners – Room to Read, SIL-LEAD, Plan International, Another Option, and New ERA – assisted with the project. In support of NEGRP, the goal of the EGRP is to improve reading skills for public primary students in grades 1–3 in 16 target districts. To achieve this, EGRP partners are providing technical assistance to the GON to develop an EGR minimum package model that the GON can implement nationwide within its budget (Figure 2). The EGRP is designed to strengthen the Ministry of Education, Science and Technology’s (MOEST’s) (previously the MOE) capacity to advance and implement policies to institutionalize and sustain pedagogical and curricular reforms for improved EGR instruction. As such, EGRP is first and foremost a capacity-building program. Working closely with central-level agencies to strengthen the coaching/mentoring model, instructional model, and teacher incentive model, EGRP aims to engage those agencies and MOEST officials to drive evidence-based policy advocacy. EGRP’s technical assistance to the GON is focused on: ● Providing embedded and other direct technical assistance services to the GON for the implementation and continuing evolution of the GON’s NEGRP plan. Assistance includes supporting the GON to harmonize and organize aspects of the NEGRP that are already underway. ● Maintaining consistency with NEGRP standards, making it feasible for the GON to replicate and implement the NEGRP on a national scale, which includes utilizing and helping to improve the policies, systems, and resources that are available to the GON. At the district level and below, technical assistance is to be provided in two waves, with a mix of high￾and low-intensity interventions, as shown in Table 1: Table 1: EGRP 16-District Implementation Plan Low-intensity Interventions (16 districts, School Years 1-4) • Early grade reading assessment (EGRA)/Education Management Efficiency Survey (EMES) • District planning and capacity building • Supplemental Materials • District Education Office (DEO) and Resource Person (RP) orientation/support • Planning for additional RP’s and Resource Teachers (RTs) • School-community sensitization and mobilization • Civil society organization (CSO) High-intensity Interventions (6 districts, School Years 1-2; 10 districts, School Years 3-4) • Provision of teaching and learning materials • Training of teacher training and teachers • Supplemental materials • Orientation of head teacher • Support and mentoring of teachers • Ongoing monitoring and assessments (lot quality assurance sampling [LQAS], etc.) • Policy piloting (timetables, incentives, performance standards) • Teacher induction system Source: USAID’s Early Grade Reading Program (EGRP) in Nepal Year 2 Annual Work Plan (April 2016–March 2017). (Revised, 8/16/2016) Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 16 At the time of this performance evaluation, EGRP had implemented central-level activities and supported the implementation of two academic years of EGRP in six districts (Cohort 1) and one academic year of EGRP in an additional 10 districts (Cohort 2). This longitudinal study is intended to assess these institutional capacity-building efforts. External Factors Affecting EGRP Implementation At least one factor external to the project had a major impact on EGRP implementation before the baseline study. Although the 2015 earthquake which occurred shortly after the project launch did not severely impact the selected EGRP districts, this tragic event, nevertheless, diverted much of the GON’s focus in the education sector towards emergency response. This significantly delayed the start-up and implementation of EGRP. Several other factors external to the project impacted EGRP implementation in the intervening two-and￾one-half years, as noted below. Federalization. Section 8 of the 2015 Constitution delegated 23 formerly federal responsibilities to the local level for basic and secondary education, constituting most management and monitoring responsibility and, implicitly, some funding. Planning and budgeting oversight were reassigned to seven provincial-level education planning units (See Figure 2). The MOEST supports local governments (LGs) in these tasks by assigning permanent MOEST staff to local level education units (LEUs) in each LG: A section officer (usually a former District Education Officer) and junior officer to the village and town level, an under secretary￾level officer (and perhaps another junior officer) to the town level, and a few additional MOEST staff to the sub￾metropolitan and metropolitan levels. All rural municipalities (aka Gaun Palikas) get the same configuration of staff, regardless of the number of schools or the geographic size of the rural municipality. (G.40.PC) District Education Offices (DEOs) were reduced in size and converted to Education Development Coordination Units (EDCUs) to coordinate the work of the LEUs. Resource Centers (RCs) and the Resource Persons (RPs) who staffed them were disestablished and demobilized. Presumably, some of their functions will be taken over by LEUs, but there are far fewer LEUs than there were RCs. In the Dang district, for example, 10 LEUs are supposed to replace 15 RCs (G.29.PD). Finally, the number of Education Training Centers (ETCs) was reduced from 34 to seven, one for each province. Federalization has impacted much of the work that EGRP had done during its first four years to build the institutional capacity of DEOs, RCs, and RPs, which have since been reduced in size. This was not immediately apparent because, during 2017-2019, many DEO heads stayed on in the EDCUs. Similarly, RPs were reassigned to work in interim LEUs; however, their duties there were largely administrative and their role in monitoring EGRP at the school level was minimized. During this same period, school monitoring by elected mayors and deputy mayors began, many with no background in education, and none of them trained by EGRP until late 2018/early 2019. LGs were assisted in this by school supervisors (SSs), who had been reassigned from DEOs to LEUs. Few of these SSs had EGRP training and, like RPs, SSs were often overwhelmed by administrative duties in these interim positions. Figure 2: Federal Government Structure Federal* Province* District Coordination Committee Local level (Palika)* Ward School Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 17 In March 2019, in accordance with the Employee Adjustment Act (2017), according to EDCU and LEU staff interviewed, most LEU staff and DEOs oriented by EGRP were reassigned to non-EGRP districts, and the RPs were demobilized. The number of civil servants reassigned to the local level, however, fell far short of the number of posts authorized at that level.4 Moreover, some MOEST staff, dissatisfied with their postings, were working on getting reassigned, reporting to the LEUs in name only. Therefore, by the beginning of EGRP’s last year, our informants suggested that most NEGRP focal points and technical personnel who had been trained by EGRP under the CLA level had been reassigned to non￾EGRP districts. In terms of this study, federalization had three impacts. First, the comparability of the baseline and follow-up studies are affected since school clusters have been disestablished; the cluster level of analysis used in the baseline study has been replaced with the local level in this study. Second, at the time the evaluation team was in the field, the interim staff at the district and LEU levels were being reassigned, and some RPs and SSs were unavailable for interviews. RPs and SSs, therefore, were interviewed at any level they were available. Additionally, DEOs in the baseline were replaced by the chiefs of the Education Development Coordination Units (EDCUs), often former DEOs. Finally, and perhaps most importantly, decisions regarding how implementation responsibility for basic education would be distributed across MOEST and local governments – and, therefore, where and what kind of institutional capacity for implementing EGRP should be built – were in flux. NGOs introduce EGR in control districts, and the GON delayed implementation in other non-EGRP districts: The original evaluation design planned to measure scalability by looking for signs of spontaneous uptake of one or more program elements in districts bordering EGRP districts. However, as described below, NGO-provided EGR activities and other efforts to improve school attendance and attractiveness in comparison districts complicate the interpretation of findings in these districts. Additionally, in 2019 the GON had plans to begin implementing NEGRP in four non-EGRP districts (and ten additional districts in 2020) before the end of EGRP, which could provide some indications of scalability. However, NEGRP in these districts was in the preliminary stages during the fieldwork phase for this study, and the team decided against observing them. The teacher service commission announced in April 2019 that 10,000+ primary teachers would become permanent and many temporary teachers would be dismissed.5 How this will reduce the number of teachers that have been previously trained by EGRP, how many need to be newly trained, and how many teaching positions will remain vacant remains to be determined. Like civil servants, many teachers resist appointments outside urban areas and towns and the temporary teachers may resist dismissal. 4 In March 2019, the government declared the civil service “adjustment” process was complete, but of the 66,908 posts created at the local level, only 31.043 were filled. In May 2019, the government announced it would recruit another 9,161 new civil servants for the local level, including 805 for the education service. https://thehimalayantimes.com/nepal/civil-servant-adjustment-process-concludes/ https://thehimalayantimes.com/kathmandu/local-governments-to-get-9161-civil-servants/ 5 Silwal, Prakash (April 10, 2019) “Over 14,000 Teachers Recommended for Permanent Appointment.” Nepal24hours. https://www.nepal24hours.com/over-14000-teachers-recommended-for-permanent-appointment/ Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 18 3 EVALUATION QUESTIONS AND METHODOLOGY This performance evaluation answers five questions posed by USAID (Figure 3) using qualitative, longitudinal methods. A baseline conducted in 2016 examined questions 1 and 2 only; question 3 addresses responses to the baseline findings, and questions 4 and 5 address external developments since the baseline. The baseline findings focused on several threats to viability, sustainability, and scalability. Recommendations are usually not included in baseline studies, but at the request of the USAID/Nepal Mission, the report included preliminary recommendations to address some of those threats. Neither the baseline nor this evaluation is nationally representative or statistically representative of the 16 EGRP districts. The study assumes that building capacity to sustain and scale EGRP is complex, and there are many pathways to results, with multiple actors and factors that can influence the change model. This performance evaluation attempts to identify those actors and factors that might influence GON capacity to implement and scale￾up NEGRP. It also attempts to identify the types of variation that may be required in EGRP, given the new federalist structure of the education system and the diverse contexts in which NEGRP is or will be implemented. 3.1 Methodology The methodology for both the baseline (2017) and this follow-on study (2019) drew on RTI’s Year 2 work plan and the statement of work in Contract Modification 1, which identified anticipated activities and results. In August 2016, consultants worked with USAID and RTI to develop illustrative measures of viability, scalability, and sustainability that align with sub-activities and results presented in the work plan relating to IR 2 (See 2017 Baseline, Appendix 2). As part of the validation process at that time, USAID, RTI, and the evaluation team identified a subset of those activities that could benefit from a more in￾depth qualitative approach that would add value to ongoing program monitoring and evaluation (M&E). In general, the theory is that EGRP must be viable before it is sustainable, and it should be sustainable before it is scaled up: ● EGRP performance in program districts speaks to viability. ● Performance in Cohort 1 districts in Phase 2 speaks to sustainability, as does commitments to EGR in budgets and planning documents at all levels of the system. Figure 3 Evaluation Questions 1. To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? 2. To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? 3. To what degree have the findings and recommendations of the baseline performance evaluation been addressed? 4. To what extent are local governments (LGs) within the EGRP districts able to continue and sustain EGRP interventions? 5. To what extent have MOEST and CLAs built the capacity at central level to support the national scale up of the NEGRP minimum packages at the federal and local levels? Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 19 ● Evidence of EGR interventions and positive attitudes towards EGR in non-EGRP districts speaks to scalability.6 The design planning workshop at the beginning of the baseline study also produced a series of detailed preliminary scales to measure the extent to which the GON demonstrates the capacity to implement various aspects of the EGRP with fidelity, intending to measure change at the mid- and end-line of the project (See the Baseline Study, annexes 3 and 4). It was expected these scales would be modified in the course of project implementation as contextual changes occurred, such as federalization. Additionally, the design workshop produced a broader-brush matrix summarizing those scales, arranged by sub-IRs, which was to be used to measure progress towards 1) GON EGR service delivery, and 2) capacity building of GON stakeholders (see the Baseline Study, Table 6). The items in the matrix’s “latent” column reflected the most basic level of understanding/knowledge of EGR, and management effectiveness; the “emerging” column reflected attitudes, training, and first-time implementation experience; the “established” column reflected the routinization of activities; and the “institutionalized” column reflected the ideal outcomes for each sub-IR as listed in the EGRP statement of technical approach. Note the “institutionalized” outcome is not a specific deliverable for the program. The expected outcomes of the program for each IR fall between “established” and “institutionalized.” To the extent possible, the follow-on study used the same theory of change, project descriptions, measures of viability, scalability, and sustainability, implementation capacity matrices, and sample, validated during the baseline design. The end-line was moved up approximately one year to February￾March 2019. Sampling At the field level: At baseline, the team purposively selected a sample of six districts: three districts in the East and three districts in the West. As shown in Table 1, each regional set of three districts was comprised of one Cohort 1 district, one Cohort 2 district, and one non-EGRP district. The non-EGRP districts did not receive EGRP interventions; however, they border EGRP districts and one participated as a control district for the Early Grade Reading Assessment Baseline and Midline (NORC 2015, 2018). Bordering districts were selected as part of the sample to examine two questions. First, how much change is occurring in non-EGRP districts? Second, does proximity to EGRP districts create spillover effects in terms of program knowledge, uptake, or demand? As noted above, the original study design included three data points for each cohort: baseline, midline, and end-line. The current design shown in Figure 6 provides just two data points: baseline and follow￾on. 6 As noted above, the team originally hoped to measure GON implementation of NEGRP in non-EGRP districts at mid- or end￾line, but as of the time of this study, GON had not initiated significant work in those districts. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 20 Figure 6: Exploring Variability, Viability, Scalability & Sustainability at the Field Level Performance Evaluation Type of District Baseline (2016) Follow-on (2019) Cohort 1 Banke, Saptari Variability of the primary education system in EGRP districts Sustainability of the high intervention model Cohort 2 Dang, Dhankuta (H) Viability of the adjusted high intervention model Non-EGRP* Salyan (H), Sunsari Variability of the primary education system in non-EGRP districts Variability of the primary education system Scalability with some contact with NGO EGR programs,* minimal EGRP contact NEGRP** N/A Scalability without technical assistance Note: All districts not marked (H) for Hills are in the Terai ecozone. At the evaluation design planning workshop on September 3, 2016, USAID, RTI, and CAMRIS developed a scale for measuring changes over time in institutional capacity for each of the three-project IRs. * Changed in 2019. Baseline differentiated between Cohorts 3 (Sunsari) and 4 (Salyan) due to different initial exposure to EGRP. However, NGO EGR programs started in both districts since 2016 standardized exposure to EGR concepts. ** Added for 2019 evaluation but indeed not sufficiently underway to evaluate. Central-level Stakeholders The international team leader and EGRP expert interviewed most senior stakeholders of the early grades of primary education at the central level during March 2019 (Figure 7). The team also interviewed the RTI acting chief of party and teams for each of the IRs. Figure 7: Informants at the Central Level Types Central Line Agencies (# of informants in 2016/2019)) Governmental Curriculum Development Center (CDC) (2/3), Centre for Education and Human Resource Development (CEHRD) (combined entity of former Department of Education (DOE), and National Centre for Educational Development (NCED) (2/2), Education Review Office (ERO) (1/1), MOEST (1/1), Teachers Service Commission (TSC) (1/1), Language Commission (LC) (1/0) Non-governmental Room to Read (1), Save the Children (1), World Education (1) [missing: Voluntary Service Overseas (VSO)] EGRP Implementers RTI Donor Partners United Nations Children’s Fund (UNICEF) (1), World Bank (1) [missing: WFP] * Repeated efforts to arrange appointments with the World Food Program (WFP) failed. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 21 Provincial, Regional, District, and Local-level Stakeholders In each of the six districts, the evaluation team interviewed the interim head of the EDCU, previously the DEO, the EGRP Focal Person (who in some cases was also the EDCU head), and the EGRP District Coordinator. The team also interviewed district-level NGOs working in early grade reading, journalists, teachers’ associations, and parent-teacher associations. As noted above, since the baseline, federalization eliminated an entire level of analysis and informants – the cluster level – and added two additional ones – the provincial and the local levels. In 2019, the sample schools were in two provinces, six districts, 13 palikas, and 17 wards. Except for a few ward chairmen who could not be located within the time constraints of field visits, key stakeholders were interviewed in all these targeted jurisdictions. Also, field visits took place during the period when interim staffing arrangements at the local level were being regularized. As a result, many RPs and SSs were being demobilized or were moving to new locations during this time, and the team interviewed these staff persons wherever they could be found, whether at the district or local level. All had been assigned more administrative duties during the interim federalization period and had less contact with schools than during the baseline survey. Figure 8: Informants at the Provincial, Regional, District and Local Levels Level Target Informants (Number of Informants in 2016/2019) Provincial Provincial Education Unit (0/2), Education Development Directorate (0/1) Regional Education Training Center Directors (4/2), EGRP Regional Coordinator (0/2) District (6) EDCU (6/6 DEOs), District Coordination Committee (0/1), EGRP Focal Person*(5/3), EGRP￾Coordinator (4/4), School Supervisors (8/5**), local NGOs (3/4), Teacher Associations (6/5), journalists (6), Parent-teacher- Association (2/4) Local*** (13 palikas/ 17 wards) Mayors (0/13), Resource Persons (12/7), Local Education Unit Head (0/9) school supervisor (0/1), Ward Chairmen ((0/17) * Overlapped with EDCUs. ** Some reassigned/moved on to LEUs. ***Replaces cluster-level from baseline. School-level Stakeholders During the baseline study, within each district, two RPs helped the study team identify the best school associated with their respective resource center/cluster and a second school with a different language profile; the same schools were revisited for the follow-up evaluation. Our sample, therefore, included four schools in each of the six sample districts and two schools in each of the two baseline clusters. However, in 2019, clusters were no longer active, and the 24 schools were instead associated with 13 palikas (rural, urban, sub-metropolitan). No schools were more than a two-hour walk from the nearest road; by the time of this evaluation, improvements in roads reduced the walking time to several schools. Two school visit teams each visited 12 schools; a male and female constituted one team, and two females constituted the second team. Three of the four members of the school visit teams had conducted the baseline school visits. Field supervisors observed the team in at least one school in each district. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 22 The school teams conducted interviews in Nepali and, as necessary, in local languages in each school for approximately one day. Within those schools, one Class 3 classroom was selected for a Rapid Classroom Assessment (RCA); where there was more than one Class 3 classroom in a school, the headteacher chose one. Thus, the selected classroom was likely to be of average or above-average quality. The headteacher supported the team by identifying and calling to the school for interviews of representatives of local NGOs, members of the school management committee (SMC) and parent￾teacher association (PTA), and local philanthropists who invested directly into schools. Figure 9: Informants at the School Level Level Type (Number of Informants in 2016/2019) School (28/24)7 Headteacher (28/24), SMC chair (28/22), PTA member (21/10), Guardian of G1-G3 child (22/23), local NGO (6/10), individual philanthropists (9/1), teacher focus group (22/22) Rapid Classroom Appraisal (28/22)8 Finally, the team also interviewed RTI EGRP staff. 3.3 Data Collection, Management, and Analysis The evaluation team revised the questionnaires from the baseline to consider the changed country context, largely necessitated by the elimination of the cluster level and its replacement by the local level. It also developed new questionnaires for the Province Education Unit (PEUs) and the Education Development Directorates (EDDs). As the field data collection took place close to the school final examination period, less than a week was available for revising, translating, and back translating instruments, allowing no time for piloting the new instruments. Two field teams collected data, one in the eastern region and the other in the western region of the country, each comprised of a field team supervisor and two field workers. One team covered the Banke, Dang, and Salyan districts while the other covered the Saptari, Dhankuta, and Sunsari districts from February 12 to March 8, 2019. The team was able to complete the data collection as planned, except for the rapid classroom assessments (RCAs) in two schools due to an unanticipated strike. Therefore, per Table 4, 22 RCAs were conducted. The team recorded central, province, district, and palika-level interviews in English then uploaded handwritten notes into Microsoft Word files. Handwritten Nepali interview notes for school-level interviews and RCAs were translated into English and transcribed into Word by native Nepali speaking research associates who are fluent in English. The evaluation team then uploaded all interview notes into Dedoose software for content analysis and coded excerpts for a dozen or so themes, as shown in Table 6. The team leader reviewed all excerpts and recoded about 10 percent. Additionally, the evaluation team complied selected data from headteacher interviews and RCAs into school summary sheets and summarized responses quantitatively to the question, “What is a good school?” 7The sample included four schools in each of six districts. In each district, the sample included two schools in each of the two clusters. There were 28 schools in the baseline sample, including 4 pilot schools in Bhaktapur. The Bhaktapur pilot schools were not revisited in the follow-up study. 8 At the end of the data gathering period, two schools were closed for two days for a nation-wide bandh (strike). During these closures, fieldworkers were able to track down and interview school staff, but students were unavailable. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 23 3.4 Methodology Limitations Although the team conducted more than 250 individual interviews, focus group discussions (FGDs), and RCAs, the findings at the district and the school level are merely illustrative of the range of possible conditions and responses to EGRP in the Terai and, to a lesser extent, in Hill eco-zones. These findings, therefore, should not be used to generalize across all 16 EGRP districts. Despite, the relatively large number of excerpts shown in Table 6, statistical analysis of these excerpts is inappropriate. Broad brush coding captured major themes and variations. Appendix 5 contains illustrative excerpts organized by theme and respondent level, and unique identifiers are assigned. Relevant excerpts are cited in the narrative below with those unique identifiers in brackets. Finally, the findings below are not exhaustive; rather, they are simply the most consistent across sampling informants and levels. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 24 4 FINDINGS The findings are organized to address the five evaluation questions in the order shown in Figure 5. By far, the largest part of these findings responds to Question 1, responding to USAID Mission emphasis on field results and implementation to inform programming and to ensure there was sufficient data collection at the school level, which is where the program is reaching the target beneficiary, students. All findings reported below reflect information provided by interviewees. Because informants at different levels of the system are in positions to observe and experience different things, wherever possible, findings are associated with a specific level of an informant. Figure 9 summarizes the number of excerpts addressing key themes and sub-themes by the level of the respondent. For example, in row A, in the district column, district respondents provided 136 comments about EGRP working well. A single interview might contain one or more excerpts for more than one theme/sub-theme, for example, including comments on what EGRP components were working well or not so well, perhaps skipping any mention of environmental factors helping or hindering viability, but identifying several environmental factors related to sustainability and scalability. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 25 Figure 9: Number of Respondent Excerpts Themes/Subthemes Level School Local District Province & CLA Total # of comments EXCERPTS RELEVANT TO EVALUATION QUESTIONS 1-2 To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? 3,179 Viability A. EGRP components: working well Cohort 1/Cohort 2 377 156/221 100 136 81 694 B. EGRP components: not working well Cohort 1/Cohort 2 150 70/80 65 51 91 357 C. Factors in the environment: helping Cohort 1/Cohort 2 316 190/126 50 63 45 474 D. Factors in the environment: hindering Cohort 1/Cohort 2 255 136/119 100 98 70 523 Sub-total 2,048 Sustainability F. Factors in the environment: helping 38 38 34 31 103 G. Factors in the environment: hindering 55 34 45 56 190 ● What is a good school? 119 119 Sub-total 412 Scalability H. Factors in the environment: helping Cohorts 1&2/3&4 290 42/248 57 65 14 426 I. Factors in the environment: hindering Cohorts 1&2/3&4 163 1/162 51 42 18 274 Sub-total 700 EXCERPTS RELEVANT TO EVALUATION QUESTION 4 To what extent are Local Governments (LGs) within the EGRP districts able to continue and sustain EGRP interventions? J. Local government Rural/urban 70 36/34 117 72/45 42 32 261 EXCERPTS RELEVANT TO EVALUATION QUESTION 5* To what extent have MOEST and CLAs built the capacity at the central level to support the national scale-up of the NEGRP minimum package at the federal and local levels? K. NEGRP Minimum Package and scaling up 18 18 * Excerpts relevant to this theme were not separately coded, but were drawn from excerpts coded under other themes, mainly Scalability. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 26 4.1Evaluation Question (EQ)1. To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? During the baseline in August 2016 and 15 months into the project, high-intensity interventions had been introduced into six EGRP districts (Cohort 1) and low-intensity interventions into 10 districts (Cohort 2). The team also found NGO partners and staff in place, except for reading motivators (RMs), as appropriate to Cohort 1 and 2 districts, and per the program plan (See Table 1). The baseline evaluation found this to be a significant achievement given that the previous (the first year of EGRP was punctuated by months of recovery after a major earthquake. In February 2019, almost four years into the project, ten districts were nine months into a 24-month implementation, and six districts completed 24 months of classroom implementation. In the Cohort 1 and Cohort 2 schools visited for this evaluation, the team found initial training completed, teaching and learning materials (TLMs) in place, and stakeholders—including children and parents—more satisfied with the program than not. However, just a year before the project end date, the team also found many components not yet working as planned. In the sections below, after an initial general summary, responses are described beginning with the school level, then the local government, district, and, finally, the provincial and CLA levels. 4.1.1 Viability of EGRP: What Components are Working Well? The largest number of stakeholders’ statements, from students to PEUs, confirm that many EGRP components are working well. In RCA, Class 3 students in 12 schools (Cohorts 1 and 2) say reading is easier this year than last year, and in only two schools (both Cohort 1) do they report reading is the same or harder9. TLMs and supplementary reading materials (SRMs) are much appreciated, and 14 of 16 sample schools visited were actively using the material. The teachers who are using the EGRP materials as planned and are teaching 90 minutes daily say it is improving students’ learning and making their work easier [A.5-6.S]. Teachers also note an increase in children’s attendance, in their ability to speak and understand Nepali, and in developing a reading habit [A.9-15.S]. All headteachers (HTs) and most EDCUs interviewed said EGRP is a good program and give it high marks for teacher and HT training and TLMs [A.2-3.S]; they attribute to EGRP better student attendance and, in a few cases, the return of some students from private to public schools [C.28.PD]. The SMC grants to the lowest-performing schools are strengthening those school management committees (SMCs), and more school improvement plans (SIPs) include budgets for EGRP. Various stakeholders report that children now prefer to stay in and read rather than to go outside at recess. In the middle of the second year for Cohort 1, the external EGRP mid-term review found the percent of parents who reported that they read to children or listen to them read at home, one of the goals of the community mobilization component, increased from 69 percent at the baseline to 90 percent. According to one FGD in a town school, the 10 days of EGRP training were more effective than 10 months of traditional teacher pre-service training delivered under the existing system; a GON staff person reported that one teacher told him, “This was the first training in my 25-year career that I am happy about” [A.7-8.S]. 9 In the “harder” case, the only Nepali speaking teacher had left this Cohort 1 school, and no one was teaching EGR in 2018-19. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 27 One EDCU, a former primary school teacher, explained that, historically, Nepal had few programs that targeted foundational learning10, and he says none have been more effective than EGRP. An EGRP Focal Person said, “EGRP makes teachers and parents aware that Classes 1-3 are really important.” Another described radio messages as touching, moving, and effective [A.21.PD]. At the provincial level, PEU staff expressed high levels of satisfaction with the program [A.23-25.PC]. Overall, estimates of the percentage of Cohort 1 and Cohort 2 districts implementing EGRP well at the time of the survey varied by respondent type. For Cohort 1, EGRP field staff estimated 60 percent of schools and GON district staff estimated 50 percent of palikas were implementing EGRP well. However, an RP said only 35 to 40 percent of teachers fully implement the TLMs when no one is watching. For Cohort 2, EGRP field staff estimated about 50 percent of schools, but, based on informal supervision, GON district-level staff said 30 percent of schools are using EGRP materials well [A.1.S, B.1.S, B.]. However, whether 30 or 60 percent of teachers, schools, or palikas are implementing EGRP may have as much to do with the state of the school system as with EGRP; this is discussed more fully below under Section D, Viability of EGRP: Factors in the Environment that Hinder Implementation. 4.1.2 Viability of EGRP: What Components are Not Working So Well? Evaluation surveys tried to balance questions about the positive and negative aspects of the program. Nonetheless, coding excerpts from surveys and interviews reveal that excerpts reporting EGRP components that are “not working well” at the school and district levels total less than half of those reporting “working well.” There also are fewer palika-level excerpts for “not working well” than for “working well.” In contrast, excerpts from provincial and central government officers tended to say more about “not working well” than “working well.” Within the EGRP/Cohort 1 and Cohort 2 model, several schools and EGRP field staff report that TLMs have not arrived on time, creating a gap between when teachers receive their EGRP training and when they start practicing what they have learned, leading to some loss of teachers’ instructional capacity [B.8.S, B.20-21.PD]. For example, many teachers started the year with the old Nepali book and were confused about where to begin teaching when the carefully sequenced EGRP workbooks finally arrived. Several teachers described trying to juggle two books and reported that EGRP has increased their workload [B.2-3.S]. Moreover, the EGRP book has about 192 lessons, one for each day of the official school year, but with many additional local holidays and many teacher absences, not to mention the late arrival of books, teachers struggle to decide which lessons to skip [B.16b.PD]. Indeed, near the end of the school year in late February/early March, in the 14 EGRP classrooms that were using Nepali books11 and participated in RCAs, children in seven classrooms said they had completed the Nepali book but in only three classrooms did they report being close to the end of the EGRP workbooks (summary statistics from RCAs). In addition, the official timetable issued at the central level has not increased the Nepali period from 45 to 90 minutes; teachers and schools have to figure out on their own how to fit all other subjects around a 90-minute Nepali lesson [B.5-6.S]. There also is some confusion around SRMs: some teachers said children could take the SRMs home, while others said they could not. Finally, the Flash Report has not been a reliable estimator of the number of books that schools will need for each class. Not everyone has received books and efforts by EDCUs to address shortfalls are not always effective [B.7-8.S, B.20-21.PD]. 10 Other programs included Continuous Assessment (CAS) and Child Friendly Schools (CFS). The latter, comprehensive model is comprised of five or six components, including an effective learning component. However, given funding limits, CFS often focuses on its water, sanitation, and health component. 11 A total of 16 schools participating in EGRP were surveyed and RCAs were conducted in 15 (1 classroom was unavailable due to a strike). In one EGRP school, the teacher did not speak Nepali and was not using either the Nepali textbook or the EGRP materials. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 28 In two of the 15 EGRP Class 3s the team visited, EGRP TLMs were not being used at all because the EGRP-trained Nepali teacher had retired or moved away, and replacements had not been trained [B.9- 10.S]. At least one informant reported that many inappropriate teachers had shown up at the ETCs for training (i.e., one about to retire and others that did not teach at the primary level or did not teach Nepali) [D.27.PC]. This is related to the problem of poor distribution of primary teachers, referred to in Section 2, but also is affected when criteria for participation in EGRP training are unclear or are ignored. How Cohort 1 and 2 HTs measure learning quality remains somewhat confused. EGRP and ERO staff insist the Classroom-based Early Grade Reading Assessment (CB-EGRA) has become an important part of school decision-making and parent awareness in EGRP schools. Few school level stakeholders, however, mention it as a way to monitor learning achievement, and school-level CB-EGRA was conducted in only 10 of the 16 sample schools in 2018. Only 4 of 14 HTs reported that CB-EGRA had been useful in their schools.12 RPs, many sidelined by administrative duties for the last two years and all now demobilized, were originally responsible for training schools to implement CB-EGRA. Although LEUs could be trained to take up this responsibility since LEUs have to support many more schools than the earlier Resource Centers did, their capacity to support CB-EGRA and other EGRP activities at the school level remains to be tested. Since 2016, teacher support responsibilities shifted from RMs to RPs (now demobilized) and then to HTs. Some HTs were not trained until February or March 2019; therefore, for most of the 2018-19 school year, they lacked the most basic concepts about EGRP instruction, such as the need for 90 minutes of instruction daily [B.6.S]. Moreover, many HTs are already over-employed, teaching full time in addition to their administrative duties [B.14.PC]. Based on Cohort 1’s experience, EGRP also changed the focus of community mobilization in Cohort 2 from establishing Tole Reading Groups to training more educated community members to be Peer Motivators to work with parents one-on-one or in small groups to motivate and train parents to support their children’s reading. The focus on SMC grants provided a good way to train SMCs in budgeting and managing resources, but according to one informant, the EGRP-hired Community Mobilizers essentially wrote most of the grant proposals and SMCs would need at least one or two more years of practice before they were ready to produce funding proposals on their own [B.15.PD]. Although new LGs have been in place since 2017, most in the sample district had only recently received EGRP orientation. Several mayors the team met with were enthusiastic after their brief orientation but were still unsure about the type of LG activities that could support EGRP and how to budget for those [I.25,28.PD]. How to attract mayors to training and motivate them to stay focused for more than an hour or two, particularly those from cities and large towns, was still being worked out [B.24.PD]. Delays in the release of the Government to Government (G2G) budget in at least one province delayed teacher training for about six months last year and led to heroic efforts by ETCs to meet training goals before the funds expired [B.29.PC]. In at least one year, delays in the disbursement of funds to ETCs came about because at least one PEU did not know it was responsible for passing G2G funds to ETC [B.28.PC]. This is just one example of the many gaps in directives and procedures in the new federalized system, discussed more completely in Section I, below [B.17.D,]. Training for teachers was provided in regional ETCs using a cascade training model and non-specialist trainers, many of them secondary school teachers, with no prior background in EGR. This led to some 12 Following discussions with RTI staff, the team reviewed the translation of CB-EGRA in our school questionnaires and concluded it could be clearer. A senior team leader subsequently followed up with 14 out of 16 EGRP HTs by telephone to ask them to talk more about CB-EGRA but this only produced the four cited examples of CB-EGRA usage. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 29 highly trained teachers from schools previously supported by NGO EGR programs being “trained” by trainers far less experienced than themselves [B.37.PC]. Also, the dates when funding for Teacher Professional Development (TPD) was finally available clashed with secondary level exams and created great pressure to shorten training so secondary school teachers seconded as trainers could return to their schools. Delays in the delivery of TLMs, described above, were sometimes attributed to delays in the production of TLMs at the CDC [B.21.D]. Revisions for the teachers’ guide for Cohort 2 were rushed through, with insufficient time for external review according to several informants. Some changes appeared to have been generated within the CDC, independent of feedback from the field and without sufficient help on content, scope, and sequencing from child literacy experts [B.33.PD]. For example, the revised teachers guide provided an exemplary lesson plan for each unit, rather than for each day, and some teachers said they missed the daily plans. The CDC said it was trying to reduce repetition. One informant reported that the CDC also is working to convert the student workbook into a separate, reusable textbook and a one-use practice book for students. For their part, as was the case in 2016, CDC staff reported that most of the curriculum experts EGRP contracted to provide materials were not Nepali experts, and the research findings and best practices they attempted to share were not relevant to Nepal. In addition, no technical assistance has been embedded in the CDC for at least two years. As for resident Nepali experts, in 2019, CDC was not working closely with the Language Commission and it rejected the most experienced Nepali mother tongue (MT) language experts available through EGRP because the EGRP partner organization to which they belonged (SIL) was Christian and evangelical. The CDC also chose not to authorize field-tested NGO materials in the MT in the interim while it produced its own materials or in languages for which the CDC does not plan to produce MT materials (e.g., Dagaura Tharu). Many of the CDC’s problems may be related to its small materials development staff, which has been further reduced recently by federalization (See D. Viability: Hindering factors, below, and I. Federalization). Finally, at least one senior respondent at the central level admitted that the goals set for EGRP, given federalization and the existing weaknesses in the primary school system, were probably overambitious [H.11.PC]. 4.1.3 Viability of EGRP: Helping Factors in the School Environment Unlike the variation in the number of excerpts indicating that schools were working well or not working well, the difference between the total number of excerpts addressing factors in the environment that helped EGRP’s viability and those that hindered is not a large one. Only at the school level in Cohort 1 did “helping”-related references exceed “hindering” references by a large margin. In addressing the question of what proportion of schools in the 16 EGRP districts began the program in a position to take advantage of the opportunities offered by EGRP [C.35.PD], in our sample of 16 Cohort 1 and 2 schools, no HTs ranked their schools as excellent, and only 6 HTs ranked their schools as good, with the rest ranking them satisfactory (7) or poor (3). The five relatively high performing schools, nonetheless, report problems with lack of in-service teacher training (5), too few teachers (3), student absences (3), inadequate infrastructure (3), lack of supervision (2), and low student enrollment (1). SMCs are active in one school and partially active in the other four, while PTAs are active in one, partially active in three, and inactive in one. All relatively high performing schools have SIPs, but one is not active, and only one school receives help from one or more private benefactors. All but one school are in rural areas that are unlikely to benefit from high-income parents. Nonetheless, these schools, almost one-third of the schools in our sample, were probably able to benefit from EGRP. This is like the estimate of schools in EGRP’s 16 districts that are fully implementing the program (see the end of Section A, above). Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 30 Some of the other factors that informants reported as helping to improve the quality of schools, in general, include a smaller catchment area (improved attendance for younger students); some commitment to continuous assessment and/or child-friendly school approaches; NGO programs in the school, early grade reading or other; and HTs willing/able to raise outside funds and teachers who were willing/able to address mixed mother-tongue instruction. Each of these is discussed briefly below. Most of the schools the team visited were part of the upper primary/middle (Grades 6-8) and, in some cases, secondary (Grades 9+) schools. Unlike standalone primary schools, the size of these schools meant their catchment areas were large, and distances to and from school could be long. Students who had to travel long distances to these schools, therefore, could not access toilets at home or go home for lunch and return in time for afternoon classes. In several districts visited with low food security and, by extension, high absentee rates (e.g., Salyan), the GON is experimenting with subsidies for school lunches. Though this program was only started in the middle of the year and local governments are largely subsidizing it until funds arrive from the central level, school officials say this has helped reduce absenteeism. Given that Nepali classes in lower primary tend to be taught in the mornings, however, reading instruction should be less directly affected than other subjects by partial day absenteeism issues. As noted earlier, EGRP is one of the few interventions that has focused on improving school learning during the foundational years. However, two such earlier, smaller initiatives—Continuous Assessment System (CAS) and Child-Friendly Schools (CFS)—along with NGO-supported EGR efforts helped to pave the way for EGRP in some schools. Individual, rapid, teacher administered EGRA fits very well in the CAS system that was supposed to replace the end of year exams in the early grades. The comprehensive CFS model supported by UNICEF in many parts of the world is comprised of five or six components, including an effective, child-centered learning component compatible with EGRP. Some of the school-level informants compared EGR favorably with the child-centered CFS instructional approach [C.31.S]. Finally, NGO-implemented EGR programs have been implemented previously on a small scale in several EGRP districts, for example, by World Education in Dang. These programs have been ongoing for up to four years and should make the introduction of EGR in these districts easier. The team found few efforts, however, to draw on the expertise in EGR developed at the school and district level by these NGO projects [e.g., by using stronger teachers from these programs as mentors or master teachers in ETC-run training or onsite teacher professional support (TPS)]. Overall in Cohorts 1 and 2, more schools are securing and managing funds from multiple sources, and more had prepared School Improvement Plans than in 2016. At the palika and district levels, particularly in rural palikas, mayors express an interest in using their funds to support education and have begun monitoring schools. In some schools where Nepali is not the mother tongue for some or most of the students, schools are not waiting for EGR materials in other languages; rather, they are using teachers with some knowledge of those languages to “bridge” learning for non-Nepali speaking children, working in two or more languages until their students learn enough Nepali to function in that language [C.4-6.S]. This has encouraged the development of EGRP TLMs in Nepali as a second language version, due to be piloted in 2019-20. 4.1.4 Viability of EGRP: Hindering Factors in the School/Education Environment Since 2017, the Nepali education system changes have been enormous as a result of federalization, culminating in the transfer of most EGRP-trained GON officers at the district level and below in March and April 2019. The impact of federalization on EGRP to date and implications for EGRP’s future are described more fully under Question 4 below. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 31 The team asked palika and district officials and NGOs operating independent EGR programs what proportion of schools were “dysfunctional,” or which had so many problems that no learning intervention was likely to make much of an impact. Estimates ranged from 35 to 50 percent [D.1.S, D.17.PD]. These estimates are crude, but they suggest that EGRP faces serious environmental hindrances in at least a third of schools. Informants said many of these “dysfunctional” schools have too few students but are resisting being merged, so their problems are unlikely to be resolved soon and are out of the manageable interest of USAID’s implementers.13 However, schools need not be perfectly equipped for EGRP to function; the five “good quality” Cohort 1 and 2 schools (see preceding section) fell short of many basic inputs yet were still functioning well. Lack of Sufficient Teacher Refresher Training, Student Absences, and an Insufficient Number of Teachers The most frequently reported problems for HTs in both Cohort 1 and 2 schools, independent of EGRP, are 1) a lack of sufficient refresher teacher training, 2) student absences and an insufficient number of teachers, and 3) student absences in Cohort 1 and infrastructure problems in Cohort 2. Also, in some schools, the PTA and SMC may be inactive. To the extent EGRP can help address any of these problems – and it, indeed, seems to be doing so – school-level stakeholder support for the program is likely to be higher. The program is strictly prohibited from funding infrastructure. However, as described in Section A, Cohort 1 and Cohort 2 stakeholders report EGRP is indeed helping to address student absenteeism, and almost all stakeholders appreciate teacher quality/regularity issues and EGRP-provided teacher training. Teacher Management and Recruitment The inappropriate teachers that schools send to EGRP training may be an artifact of the poor distribution of teachers in a tiered teacher system [D.27.PC]. The Nepal education system recognizes at least four types of teachers, with such distinct status and privileges that within some schools they use different staff rooms, and there may be little communication between different teacher types.14 Permanent teachers formally assigned to primary schools—most of which are embedded in crowded lower or high secondary schools—are often assigned by the HT to teach upper grades and temporary teachers assigned to teach lower grades. Permanent teachers, nonetheless, get priority when training opportunities (which include highly valued training allowances) appear. ETCs provide courses of varying length – such as EGRP’s initial training – to permanent teachers. The team heard from NGOs and ETC staff but could not confirm that non-permanent teachers may not receive more than five days of training in ETCs. Moreover, if some temporary teachers were trained under EGRP, the pool of EGRP-trained teachers will be reduced by the decision in May 2019 to dismiss thousands of temporary teachers who did not pass the 2018 TSC exam.15 EGRP alone cannot address the teachers ranked as a high-level problem by both Cohort 1 and 2 school￾level stakeholders. Some mayors and ward chairmen said they had hired teachers independently [C.13 and 25.S]. Some informants suggested hiring at the local level led to political favoritism and teachers with weak backgrounds [D.18.PD, G.38.PC]. Other mayors claimed government regulations regarding qualifications or pay did not allow them to hire teachers when they found them, and they asked for circulars/guidance from the central level on teacher hiring, qualifications, and pay [G.16.S, G.21.PD]. The 13 As noted above, as catchment areas become larger, distance to school increases for some children. Merging, therefore, may increase absenteeism and dropouts, particularly among small children who live far from the school. 14 Khanal, P. (2011). Teacher management in a decentralized school context in Nepal: Fueling tension and dissent? Compare: A Journal of Comparative Education, 41(6), 15. doi: http://dx.doi.org/10.1080/03057925.2011.592041. 15 Silwal, Prakash (April 10, 2019) “Over 14,000 Teachers Recommended for Permanent Appointment.” Nepal24hours. https://www.nepal24hours.com/over-14000-teachers-recommended-for-permanent-appointment/. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 32 recent announcement that almost 11,000 primary teachers passed the TSC and will become permanent suggests help is on the way, but some mayors claimed the constitution delegated the power to hire permanent teachers to LGs, and mayors may fight the MOEST’s efforts to hire and assign them. At the same time, like LEUs, many of the new permanent teachers will work hard to remain in urban areas and resist placement in the rural areas where they are needed most [D.14.PD]. Examination Although liberal or automatic promotion in primary school is government policy, teachers report that many parents continue to demand final exams and that schools are complying [D.8.S]. No informants reported that CB-EGRA had entirely replaced the end of year exams in the Classes 1-3. One set of teachers claimed children demand to study the old Nepali textbook because that is where questions for the final exams come from; they say both parents and children insist on finishing the conventional Nepali book so that students will be prepared for the end of year exams. This could be one reason why so few RCAs found classes that had completed the EGRP workbook. Meanwhile, performance on end-of year￾exams had gone up in at least one Cohort 2 district before the high-intensity phase and in at least one sampled non-EGRP, control district [F.22.PD].16 As there is at least limited evidence that conventional test scores are going up without EGRP, the need to invest in EGRP activities may not be as obvious to many stakeholders. Parental Demand for School Quality Not Driven by EGR Results Parents’ demands for private English medium schools and resulting decreases in enrollments in some public schools, particularly in towns and cities, have only increased since the 2016 baseline warned about this problem [D.10.S]. This issue has major implications for sustainability, as described in Section F below. As EGRP schools have improved student performance, this trend has begun to reverse itself though seemingly without a focused effort from EGRP. For example, a few EGRP and other NGO￾supported schools in Nepal have demonstrated that, within a few years, better performance by students from EGRP and other EGR programs in public reading competitions can raise public perceptions of public school quality and this has persuaded a few parents to return their children from private to public schools [C.45.PC, E.25.PC]. Second, the team heard of a few schools, both public or private, that are attempting to provide 90 minutes of Nepali EGR instruction in the early grades and use English as the medium of instruction for all other subjects [F.27.PD]. The feasibility of this approach, given the fixed length of the school day, remains to be determined. 16 The test score increase in one non-EGRP district was reported by one of the most credible qualified EDCU informants with a strong background in education, beginning as a primary school teacher and having served as both DEO and EDCU in a district since before August 2016 and had graphs on his or her walls tracking academic achievement of different levels of schooling across years. The test score improvements were modest but steady and the person their self was modest and readily talked about all the things in the district that needed improvement. Like many other findings of this qualitative study, this cannot be automatically generalized to other districts but raises a serious issue, which the USAID/Nepal Mission may want to follow up on. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 33 Parents remain largely unaware of how much time quality learning demands. For example, when asked at what age their children should be able to read in basic Nepali and English, some parents said five years old, others said eight, and a few suggested eleven years old. This variation in ages may be the result of parents’ definition of reading – perhaps just knowing letters of the alphabet and a few words qualifies – and what it takes to achieve it, which are very different from EGRP’s definitions. In addition, no stakeholders mentioned time on task or child engagement time as a problem, but separately many raised issues in the education system that hinder time on task and opportunity to learn, such as a school year of 192 days17 drastically reduced by frequent teacher and student absences, high student/teacher ratios, frequent strikes, ad hoc local holidays and multi-day family ceremonies, not to mention months without EGRP workbooks [D.23.PD]. Without an appreciation of the time on task concept and without acknowledging the real number of days classes meet – probably closer to 170 than 192 – it is difficult to persuade stakeholders to implement EGRP with sufficient fidelity to enable most children to achieve EGRP’s IR1. 4.1.5 Sustainability of EGRP: Factors in the Environment That Can Help Evidence for sustainability largely comes from the performance of Cohort 1 districts one year after the end of the high-intensity phase. This is the reason for the relatively small number of excerpts on this issue. Whether acknowledging the pre-existing weaknesses in the school system described in the previous section or the destructive impact of federalization on efforts to institutionalize EGRP in the GON, all stakeholders at all levels agree that two to three more years of technical assistance are needed in both Cohort 1 and 2 districts to demonstrate a robust model that will be sustainable in the pilot districts and scalable/adaptable to the rest of Nepal. In the two Cohort 1 districts, a few high-functioning schools with active leadership are sustaining many EGRP components, even Tole Reading Groups, which the EGRP decided not to continue in Cohort 2 [E5.-6.S]. In one rural palika in a Cohort 1 district, about half of the 62 TRGs active during the intensive 2016-2018 phase were still active in March 2019 and receiving some financial support from a ward [E.8.PD]. The most frequently mentioned components that need continued outside funding after the high-intensity phase are TLMs/SRMs and teacher refresher training sessions. One or two school and local government level stakeholders said they would try to manage TRGs and finance other community mobilization and reading festivals [E.8-11, 14.PD]. As noted above, few SMCs wrote their own grants for EGRP in the last two years, and more practice may be necessary for them to manage that in the future. [B.15.PD]. Finally, in terms of finding external funds, in the few schools with experience with private benefactors, the benefactors have primarily focused on infrastructure. Nonetheless, several school- and palika-level stakeholders thought it possible that, with the right acknowledgments (e.g., plaques on library walls, stamps on library books), benefactors might be persuaded to invest in books [E.19-20.PD]. More confidence about raising their own funds in the future was evident in Cohort 2 schools than in Cohort 1, where EGRP was no longer providing support, and the struggle to raise local resources had already begun. Province level stakeholders expressed confidence that continued enthusiasm for EGRP among the central-, provincial-, district-, and school-level stakeholders who have been trained by EGRP and remain assigned to work on EGRP bodes well for sustainability. GON staff at the central level expressed confidence that the National Assessment of Reading and Numeracy (NARN) and CB-EGRA would continue to be used after EGRP formally ends [E.27.PC]. GON staff at the central level say they understand that HTs have been overburdened with teaching responsibilities and that they are 17 NB: For comparison purposes, in lower primary the UK school year is 190 days of 5.5 hours, while U.S. states vary from 170 to 185 hours of 3-5 to 6 hours for lower primary. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 34 considering how to reorganize the HT role to focus more on leading and monitoring [E.31.PC]. Continued enthusiasm for EGR by donor partners is evident in that rolling out the NEGRP Minimum Package (see Section J below) is now the measure for disbursement of donor support to education. Finally, all stakeholders agree that to finish developing and testing the Nepali and MT materials, and a robust model that can work in the new federalized education system will require an EGRP Phase 2 [E.30.PC]. One education donor partner was vehement: “We support and continue EGRP; there is no alternative to improving school quality.” 4.1.6 Sustainability of EGRP: Factors in the Environment That Hinder Excerpts from data collected that reflect on factors that hinder sustainability greatly exceed those that reflect on facilitating sustainability at the school level, although they show less variation at the district and provincial/central levels. At the district level, EDCUs and others expressed some skepticism about whether LGs will pick up recurring costs for some EGRP components outside the minimum package that are not planned to be funded through conditional grants, such as SRMs, and how well these will compete with funding priorities in other sectors as well as within the education sector in LG budgeting. However, in both 2016 and 2019, to determine what stakeholders at the school level might conceivably prioritize in their SIPs and requests to DEOs and now LGs since EGRP has been implemented, the team asked an open-ended question: “What is a good school?” The answer from school-level stakeholders remained relatively consistent across time and cohorts: most stakeholders define school quality in terms of basic inputs, principally infrastructure [F.2]. For example, from the 110 responses in 2019, the team parsed about 620 items, with infrastructure constituting 35 percent of items mentioned18, teacher quality/regularity/number constituting about 27 percent, followed by parents’ participation, student quality/regularity/number constituting each about 10 percent. The most important finding, however, is that items relating to outcomes – learning, broadly defined, the goal of EGRP – represent 3 percent or less of all responses in each of the four cohorts. This lack of focus on learning has major implications for where schools, HTs, and, by extension, parents of school children and the officials they elect, invest their time and resources. In light of these excerpts, it is not surprising EGRP partner NGOs report difficulties in persuading stakeholders in Cohort 2 schools to use SMC grants for something other than infrastructure. [F.3.PD]. Additionally, while parent engagement, student attendance/discipline/engagement, and SMCs appeared in about 10 to 13 percent of the statements about good schools, HTs were only mentioned once. This is puzzling since the district￾level officials, and our own observations, suggest HTs are often the main agents for improving school quality, whether in terms of learning, parent engagement, infrastructure or raising funds. The 2016 baseline performance evaluation identified the rise of “English medium” private schools as a major environmental threat to EGRP. In 2019, one of the most discussed problems at the district level and below continued to be the siphoning off of stronger students and their relatively more engaged parents from public schools to private “English-medium” schools, particularly in towns and cities [B.11, 15.; C.8; D.10, 16, 20, 26c; F.1g, 35]. Interestingly, several informants suggested that the main attraction of these private schools is not the superior English language training they offer in the lower grades, rather it is the presence of inputs parents highly value, as identified in the preceding paragraph, principally better physical infrastructure and teacher regularity [D.11.S]. Consistent with that approach, in discussions with HTs and local government officials, all appeared more focused on English language instruction as an input, than on English language learning as an output. For example, one mayor ran for office promising to convert all public schools in his town to the English medium. After he was elected, 18 Most frequently mentioned infrastructure items include, in order of importance: drinking water (33), toilets (27), sufficient classrooms (26), and playground (24). Boundary wall, Furniture/carpeting, classroom and school ground maintenance/cleanliness, sports materials, school feeding programs/canteens, library and computers were also mentioned. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 35 he ordered schools to begin teaching in English in Class 1 and to convert Class 2 the following year, Class 3 the second year following, until all Classes became English medium. However, when he tested Class 1 teachers, he found that some did not speak enough English themselves to teach the Class 1 English textbook; in response, the mayor said he would fire teachers who failed and hire others [G.20.PD]. At least one other elected official, however, saw no problem in allowing parents who have the means to transfer their children to private, English medium schools. But as shown in Figure 10, the fall in enrollments in public schools creates a vicious cycle, reducing teacher quotas and per-student grants, leaving public schools with the most unprepared and unsupported students from marginalized communities, whose parents tend to be the least literate, least economically secure and, by extension, least engaged in their children’s education [F.2, 3.5, H.33.S]. All stakeholders understand this cycle well. For example, many parents insisted that teachers should be required to enroll their children in public schools and most teachers and officials while admitting that they enroll their children in private schools, did so with shame or defensiveness, clearly knowing this was reducing the quality of the public system in which they are employed. Figure 10: Cycle of School Quality Decline Few stakeholders at any level in our sample expressed confidence that schools could maintain an acceptable level of quality with funding from central levels alone. Therefore, a reduction in the number of parents who can contribute private resources or, most importantly, under the new federalism, who can influence the LG to invest public resources, is likely to result in a further fall in school quality. Without such support to schools, there will be no additional teachers, reading competitions, SRMs, and in-service teacher training necessary to sustain EGR even in the short run. 1. More aware parents send children to private schools 2. Fewer students --> Capitation grant & teacher quota reduced. Can't meet NGO selection criteria (>10 or 12 students in G1),-->Less infrastructure and less resources 3. Remaining public school students have lower socio-economic status: parents less able to provide private funds or lobby for public ones, more likely illiterate/unable to help with homework, food insecure 4, More students come to school hungry, not prepared for class, disengaged, needing more time for remedial work, attendance irregular 5. Poor quality public school: teachers unengaged/overwhemed; no resources to address less privileged students/multiple MTs; no libraries, few engaging TLM Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 36 Given the competition for scarce discretionary funds at the local level, EGR also will need support from the broader community, but the team saw no evidence of community mobilization beyond parents and school staff (e.g., with the public). Also, the period provided to NGOs working in Cohorts 1 and 2 – less than 24 months – was too short to establish even school-focused activities [F.7-9.PD]. Despite this, the EGRP partner NGOs the team interviewed had no plans or funds to continue or expand critical community/social mobilization activities after EGRP support ends [F.6.PD]. Finally, at the district and palika levels, sustainability is linked to the need for clearer directives and guidelines: “If the CLA sends us directives, if the CLA makes it our responsibility, if the CLA sends us the resources, we will continue the program. It will also be better if there is something sent to us from the center on paper and ownership is given to us” [F.10.D]. Maintaining a focus on funding and support for EGRP and NEGRP at the highest levels of the ministry also remains a challenge. Several informants suggested delays in the production of MT and Nepali as second language materials due to the small size of the CDC staff. At the request of the highest levels of MOEST, the small CDC staff were trying to produce an “integrated” primary curriculum in time to pilot in 2019-2020, despite warnings from the donor that this type of curriculum needed more than a decade for development [F.12-15.PC]. Other informants mentioned the attractiveness of Early Childhood Development to both donors and government, which could easily crowd out EGRP by absorbing scarce education resources in the LEUs as well as more established ones at the ETCs and CDC. Meanwhile, EGR is not scheduled to be integrated into the MOEST pre-service teacher professional development materials anytime soon. This implies that all teacher orientation and training for NEGRP will have to be provided through specialized, in-service training for the foreseeable future. 4.1.7 Scalability: Helping Excerpts and data from non-EGRP districts identify several factors that either help or hinder scaling up of NEGRP. The number of excerpts with the theme related to scalability and helping is large relative to other themes. However, that number is inflated by many very brief, categorical statements. Essentially, all stakeholders in every Cohort 1 and 2 schools, and in every non-EGRP school that had some experience with NGO EGR programs, stated that EGR programs should be scaled up in other districts. There are fewer gaps between the number of helping and hindering excerpts at the palika and district levels and too few references to attempt any generalizations from stakeholders at those levels. EGRP and NGO staff report inquiries about EGR and direct requests for help with EGR from the Siraha, Okhaldunga, Bhojpur, Rolpa, and Tehrathum districts [G.17.PD]; in some cases, these requests were from teachers visiting EGRP districts and in other instances from EGRP-trained GON officials that had been transferred to new districts. Experience with CFS concepts in some non-EGRP schools also has created a demand for better TLMs and instructional methods and a greater focus on student engagement that the EGRP model offers [E.17.PD, G.31.PD]. School stakeholders in some non-EGRP districts exposed to Save the Children, Plan, and Room to Read recognized that these programs were improving children’s learning, and they recognize that additional instructional time and TLMs were contributing factors [G.2-6.S]. At least one NGO suggested that success in CFS began with reorienting stakeholders’ around the “what is a good school” question [G.31.PC]. Some schools realize they do not have the teachers necessary to transition to the English medium and are looking for other ways to improve. Journalists are interested in improving education and in reporting on successes, not just failures [G.29.PD]. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 37 Many schools and palikas have received and managed funds from five or more governmental sources over the last two years and, as such, are better prepared to respond to NEGRP’s demands for ongoing support [G.12-15.S]. Finally, the federalization process that scattered all GON staff trained by EGRP to new districts and LEUs in April 2019 will have at least one positive effect. In the future, more districts will start NEGRP with one or more EDCU or LEU staff persons already trained or exposed to EGRP. At the central level, several senior GON staff who have been transferred in recent years remained in positions important to EGRP, and more senior staff will be exposed to EGRP in its final year. 4.1.8 Scalability: Hindering The factors that were raised as hindering scaling up in non-EGRP districts echo those that were raised as hindering sustainability in Cohorts 1 and 2. According to informants at the district, palika, and school levels, the proportion of dysfunctional schools is high in both of the non-EGRP sampled districts. In one district, the teacher association estimated that 25 percent of the schools are dysfunctional and a teacher association in the other district estimated that 30 percent of the teachers – specifically those with master’s degrees – were qualified but not taking teaching seriously [H.46.PD]. Although one mayor complained about teachers drawing “handsome” salaries [H.39.PD], one SMC reported that the school was compelled to distribute the salary of one teacher among five [H.34.S]. Again, as noted above, these comments most likely apply to different types of teachers. Uncertainty around teacher pay and conditions of employment disincentivizes teachers from taking up and fully implementing EGRP or any new intervention. Stakeholders at all levels and cohorts, despite expressing uniform enthusiasm for scaling up, acknowledged that the model would first have to be adapted to language and geography in other districts. Concerning language, however, after four years, the CDC is ready to test only one MT version of the TLM; how many more MTs it can address and on what schedule, given its limited staff and pressure to address other initiatives, such as the integrated curriculum, is unclear. Similarly, guidance on adapting NEGRP to geography will likely need to come from the central level. The CLAs, however, are still waiting for the final version of the new Education Law and afterward must develop directives as to how wards and palikas should manage education, including mandates and scope for adapting the EGR model. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 38 4.2 EQ2. To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? During planning meetings for the 2016 baseline study, the evaluation team worked with USAID and the EGRP staff and partner NGOs to develop a scale that could qualitatively benchmark progress on institutional capacity building (See Baseline Study, Annex 3). This instrument provided descriptions of latent, emerging, established, and institutionalized capacities for each sub-IR. Because of significant changes in the government organization as well as federalization (see the Background section above), many details of that scale are no longer pertinent. Nonetheless, the team attempted to estimate levels of strengthening, based roughly on the 2016 scale. The analysis below applies to IRs 1 and 2 between August 2016 and March 2019. An analysis of IR 3 also is included although this does not directly reflect on the institutional capacity building focus of this evaluation because no GON agencies currently deliver community mobilization services under EGRP. Figure 11 below was moved here from the Conclusions section at the request of USAID; however, it reflects some individual judgment, and the findings are not indisputable, per se. Findings to support the concluded levels of progress for each sub-IR are described in the narrative sections below the figure, by sub-IR. As of March 2019, no sub-IRs have achieved “institutionalized” status (i.e., become part of standard operating procedures and recurring budgets and able to sustain themselves without special initiatives and project funding). Several sub-IRs have achieved “established” status per the detailed scales in Annex 3 of the Baseline study. However, as described in the methodology section, due to the changed organization of the education sector, this no longer represents readiness to deliver EGR services fully. This table, therefore, includes a new category – “initially established – to indicate that some aspects of the sub-IR have been implemented once or twice but capacity is not firmly established and/or policies and guidelines necessary to support the sub-IR remain underdeveloped. Figure 11: Progress on Strengthening GON Service Delivery Sub-IRs Progress Categorization Latent Emerging Initially Established Institutional￾ized IR 1: Improved reading instruction, central-level agencies, and number of districts CDC: Evidence-based EGR instructional materials designed, distributed, in use 2016 2019 mother tongue 2019 Nepali CEHRD (formerly NCED), ETCs: in-service professional development for teachers in public schools on reading instruction & materials in use 2016 2019 Monitoring and coaching for teachers in EGR instruction (formerly RCs) 2016 2019 IR 2: GON capacity to implement EGRP EGR data collection and analysis systems 2016 2019 Policies, standards, and benchmarks that support improved EGR 2016 2019 Planning and management of financial, material, and human resources 2016 2019 (human resources) 2019 (financial & material) Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 39 Utilization of data for course correction during implementation 2016 2019 Classroom-based & district-based EGRA processes 2016 2019 IR 3: Increased family and community support, number of districts Community awareness of the importance of the language of reading instruction 2016 2019 Family engagement to support reading 2016 2019 PTA/SMC contributes to quality reading instruction 2016 2019 Parent and community capacity to monitor reading progress 2016 2019 IR 1.1 Evidence-based EGR instructional materials designed, distributed, and in use Conclusion: Initially established The most serious hindrances to EGRP discussed by provincial- and central-level informants are the limitations of the CDC [D.26.PC]. Some of these limitations are the result of reductions in staff caused by federalization (see Question 4 below), while others have been apparent since the 2016 baseline and remain unaddressed to this time. The latter include a lack of expertise in child development, early literacy, and mother-tongue-based literacy approaches. For example, CDC staff raised the issue of whether 90 minutes of reading instruction daily is necessary, and, in the absence of expert advice, they plan to allocate only 60 minutes to Nepali in the new integrated curriculum. Given the serious time-on￾task issues noted above – classes meeting far less than 192 days, not all teachers trained, TLM not in place on time, much time taken up with teachers translating from MT – 90 minutes for reading instruction is probably not optional. The CDC has produced one set of EGR instructional and TLMs for grades 1-3 in Nepali and revised them once. EGRP has been unable to retain a Nepali or international expert to embed in the CDC to provide the child development, early literacy, and mother tongue education expertise CDC lacks. Within the first two years of EGRP, the GON was busy with other priorities including development of integrated curriculum. Also, the lack of trust with the local experts because of their organizational associations and neutrality is the other prominent factor. This has contributed to the long-delayed delivery of TLM in mother tongue and Nepali as a second language; as of the end of the fourth year of the project, piloting had yet to begin. Also contributing to the delay in decisions within the MOEST to pilot an integrated primary curriculum in 2019, taking scarce curriculum developers away from earlier EGRP commitments. Staff shortages due to federalization also have contributed to delays in the Nepali TLM revisions and to rushed external reviews that provide insufficient time to incorporate feedback. IR1.2 In-service professional development for teachers in public schools on reading instruction and materials in use Conclusion: Emerging to initially established To date, ETCs have provided the 10-day training for teachers in the 16 EGRP districts for three years. Major delays in government-to-government fund release from Financial Comptroller General Office meant training was rushed and required heroic efforts of scarce trainers at ETCs to train nearly 1,000 teachers in less than two months. The cascade training, relying on “resource teachers” with no background in EGRP, was, nonetheless, deeply appreciated by some teachers [A.7-8, S]. The selection Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 40 criteria for teacher participants were unclear and/or enforced. Although originally planned for in the program, no teacher or ETC mentioned teachers have received TPD credit for EGRP training, and no informants identified a date for integrating EGR into pre-service teacher training. Routine training for new teachers in EGRP schools that have experienced the loss of EGRP-trained teachers is not planned for or implemented. IR1.3 Monitoring and coaching for teachers in early grade reading instruction. Conclusion: Emerging RMs (Year 1), were replaced by RPs (Year 2) and HTs (Year 3). Many HTs teach full time in addition to their administrative duties. It is not clear how they can make time for meaningful, twice-monthly coaching for each teacher in their school. In addition, since teacher coaching, in very broad terms, is already part of the HT responsibilities, no additional incentives are being offered for this additional work [B.14.PC]. Nonetheless, EGRP staff say HTs were trained late in 2018-19. Moreover, many HTs say they have received some training, and a few say they are observing EGR classes. There was a recent announcement of some 10,883 “new” permanent primary school teachers (some of whom were previously temporary teachers)19. If this results in more EGR teachers at the school level soon (as all other temporary teachers are being dismissed), more HTs may find they have time for this role. CEHRD reports that it also is discussing restructuring HT’s roles to focus more on leadership and monitoring but gave no details or a timetable [E.31.PC]. IR 2.1 Early grade data collection and analysis systems Conclusion: Initially established The CB-EGRA designed in 2016-17 and administered in 2017, 2018, and 2019 in a sample of EGRP schools provides the data necessary to meet the SSDP’s Year 2 Disbursement Linked Indicator 1.2. All EGRP schools were trained to do CB-EGRA in 2017 by DEOs and RPs. However, without RPs and with turnovers in EGRP-trained HTs and teachers, it is unclear whether most schools can conduct CB-EGRA unaided, and it is unlikely that teachers would use the results. IR 2.2 Policies, standards, and benchmarks that support improved EGR Conclusion: Initially established The NEGRP strategy (2014/15-2019/20) that aimed to provide improved EGR in Grades 1-3 to all 75 (now 77) districts by June 2020 largely remains unimplemented and, as of March 2019, had not been updated. Nonetheless, an EGR Technical Working Group, including NGOs, met regularly and moved forward on the NEGRP Minimum Package (NMP), which has been approved by the GON and accepted by stakeholders.20 The CB-EGRA reports for 2017 and 2018 provide the only benchmarks and enabled NEGRP to meet DLI 1. These achievements would be sufficient to meet the “established” metric from 2016 and, indeed, several stakeholders at the CLA level discussed the formalization of the NMP, particularly as a great advance. However, as described under Section 4.5, as of April 2019, a two-page list of NMP components21 was one of the few documents available for review; guidelines and standards for implementing the NEGRP are lacking for most components. Moreover, federalism has moved the goalposts in terms of what 19 https://www.nepal24hours.com/over-14000-teachers-recommended-for-permanent-appointment/. 20 The contents of the package itself are discussed under evaluation question 4. 21 Components of the NMP are summarized in a two-page table from Chapter 7 of the NEGRP Standardization and Harmonization Guidelines; the team could not locate a copy of the guidelines themselves. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 41 policies and standards need to be in place to move NEGRP forward. As of April 2019, the Federal Education Act was not finalized and, according to informants from the provincial to the local level, the mandates and relationships among the MOEST and LGs for implementing basic education remained unsettled. Once the law is enacted, a CLA informant warned that mandates and relationships would have to be restructured and formalized in many new government circulars and guidelines, which will take significant time to draft and approve at various levels of the CLA and LGs. IR 2.3 Planning and management of financial, material, and human resources Conclusion: Initially established (financial and material) Emerging (human resources) The National Planning Commission and Ministry of Finance agreed to the NEGRP 5-year plan. The Ministry of Finance is sending school support (excluding teacher salaries) directly to LGs to manage TLMs and other school costs. SSDP has integrated NEGRP, and GON began a (slow) rollout of the NEGRP Minimum Package into four new districts in 2018-19. As for IR 2.2, before federalism, this comes close to meeting the “established” metric. However, under federalism, most human resources from the district to the palika level will have different responsibilities, and, therefore, training developed for DEOs and RPs will need to be adapted for EDCUs, mayors, LEUs, and HTs. Training plans also need to be developed for new teachers and HTs who begin EGR off-cycle due to normal attrition or an increase in permanent teachers. Finally, as noted above, the quantity of TOT and teacher training at ETCs, has been and will continue to need to be heroic. Finally, there has been little attention to the impact of that instruction on the quality of classroom instruction to date. IR 2.4 Utilization of data for course correction during implementation Conclusion: Initially established Initial systems are established and equipped at the central level and are being established at the provincial level for the Education Management Information System. Systems originally located at DEO still need to be redistributed to EDCUs and LEUs. IR 2.5 Classroom-based and district-based EGRA processes Conclusion: Initially established ERO, the responsible CLA for EGRA processes, has been diligent in developing CB-EGRA and processing results; CB-EGRA Framework (2017) covered 2017 and 2018 data collection, and reports for both years are available. The framework will need to be revised to cover 2019 and the following years. CB-EGRA is being integrated into the national assessment of NARN, which will include an individual assessment sub-sample. However, guidelines to implement this plan were not available for review. These achievements rise to the level of “initially established.” However, the reassignments that took place as part of federalism require a review of how many trained staff remain in ERO and how many new staff may need to be trained. In addition, because EDCUs nation-wide were reassigned, ERO must now mobilize and train new staff in EDCUs to implement CB-EGRA in existing EGRP districts and new NEGRP districts. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 42 4.3 EQ3. To what extent were findings and recommendations of the 2016 report addressed? The baseline did not attempt to draw many conclusions and offered no recommendations. The baseline report, however, did identify key threats to EGRP. These are presented in Figure 12 below, which presents threats identified in the 2016 baseline and the current status by three categories: viability, sustainability, and scalability. The team has attempted to track the progress mitigating the threats identified in the baseline study, which is summarized in the figure below. Figure 12: Threats Identified in 2016 and Status in 2019 by Threat Category 2016: THREATS IDENTIFIED 2019: STATUS VIABILITY Enthusiasm for NEGRP (GON strategy) is strong, but the understanding of EGR (how to teach reading) is weak Improved, but still an issue Parents demand English medium, less Nepali, no MT No progress School-level stakeholders do not identify learning as the most important indicator of school quality No progress Threats to essential time on task for EGR No progress Good TLM use dependent on better incentives for TPS provider No progress Delayed disbursement of G2G funds undermining the GON’s goodwill Improved The CDC needs strengthening in child psychology/development, children’s literacy, and MT-based instruction No progress Undervaluing of EGR groundwork laid by NGOs Improved. The utilization of NGO work is still weak. RPs and teachers value CAS and CFS but do not recognize EGRP as a method to address both Minimal progress SUSTAINABILITY Additional funds needed from external donors Some progress No SIPs imply little local fund mobilization Much improved SCALABILITY Need coaches compatible with the existing system. Experiments continue w/ HTs No small-scale research to test necessary variations in model No progress Lack of appreciation for evidence-based innovation Need simpler, more compelling arguments for focus on reading Minimal progress Too much reliance on CB-EGRA No national champions for EGR identified. No nation-wide BCC/social mobilization Lacking clear relative advantage vis-a-vis the existing system Cost-effectiveness of EGRP vs. the existing model not being widely understood Lacking champions in or out of the GON Reading evidence from other countries not compelling No local research Minimal EGRP in Nepali press Vision/implementation slips in response to normal attrition of teachers and GON staff No improvement Note: Question asked was, “To what extent were findings and recommendations of the 2016 baseline report addressed?” Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 43 4.4 EQ4. To what extent are local governments (LGs) within the EGRP districts able to continue and sustain EGRP interventions? EGRP was designed to build institutional capacity in various agencies, including the MOEST at the CLA level and around DEOs, RCs, and ETCs at the district level. Federalization eliminated RCs, greatly reduced the number of ETCs, reduced the resources and mandates of DEOs, and, thus, dismantled many district and local level structures around which EGRP had built capacity in its first two years. The project and future efforts by education stakeholders now face the task of building institutional capacity from scratch beginning with mayors, deputy mayors, and their LEUs, the latter staffed by civil servants from the education cadre newly relocated by the MOEST from other districts and CLAs. For this task, in May 2019, EGRP hired and oriented 17 new district program officers for the 10 Cohort 2 districts. The new program officers are mandated to work directly with palikas and district education coordinators to support LG planning, help formulate provincial policy, develop capacity at the district level, support HTs in providing TPS, and develop classroom monitoring tools for HTs, among other responsibilities. The SSDP Programme Implementation Manual for 2018/19 included capacity development for SMCs, PTAs, and LEUs in EGRP districts to develop school monitoring skills and find ways to reward regular teachers and students. Mayors, some SMCs, and LEUs reported that they indeed received orientation in the 2018-19 school year, though some as late as February of 2019. However, not all mayors chose to attend, and those who attended did not necessarily focus or remain for the entire orientation [I.24.PD]. Most of those who attended orientation expressed enthusiasm about the program, but even those who attended could not articulate basic EGR concepts. As a result, some had invested the modest discretionary funds available at the local level to support EGR in activities that did not necessarily do so [I.25, 28.PD, I.37.PC]. Stakeholders at the district level and above frequently expressed the hope that mayors and their staff would be able to monitor schools more closely than DEOs. However, some mayors reported that if the permanent teachers are hired and paid by the MOEST, those teachers will not listen to elected officials [C.36.PD, D.14.PD]. In addition, teachers’ associations reported that teachers do not necessarily appreciate having their classrooms observed (and, presumably, their teaching critiqued) by elected officials with little formal background in education [I.14.S, D.18.PD]. Significant amounts of money are flowing from many sources through the LGs to the schools. Some municipalities have promulgated “local education acts” to fill in the gaps, including one that included appointing ward chairs as the head of the SMCs in their ward, a move that could irreparably politicize those SMCs [I.13.S, I.30.PD]. However, many mayors are waiting for the release of the new Federal Education Law to clarify relationships between the central and local levels around basic education and trying to limit MOEST action in the meantime. For example, the MOEST is moving forward with long￾awaited appointments of more than 10,880 permanent teachers who passed the 2018 TSC exam and dismissal of thousands of temporary teachers who failed it.22 However, associations of LG mayors are challenging the constitutionality of that position, and temporary teachers are protesting their dismissals. Whether this will delay the arrival of desperately needed teachers, many of them needing EGRP training, remains to be seen. A National Association of Rural Mayors brought hundreds of mayors to Kathmandu 22 The appointment of these teachers will probably not radically reduce the student/teacher ratio in many primary schools. This is because a) a significant proportion of those almost 11,000 “new” permanent teachers were already teaching as temporary teachers, b) primary schools will be losing thousands of temporary teachers who failed to pass the TSC exam, c) getting teachers to report regularly to some of the remote schools that need them most is very difficult, and d) there are 28-30,000 primary schools in Nepal. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 44 in early April 2019 to defend their rights against the CLAs [I.5.PD]. LGs also have a new mandate for producing up to significant amounts of curriculum content on local topics for basic education, but how they will do this without any curriculum or textbook writers or printers in many local areas is to be determined [I.35, 42.PC]. Nepali – the subject most pertinent to EGR – will remain part of the required, nationally determined curriculum. Hence, the time necessary to work out the appropriate distribution of responsibility for basic schooling between the elected LG officials and the CLAs is likely to delay working out LG support for EGRP for a year or two, at least. Elected LGs may be in a somewhat better position to support EGRP IR 3 activities than MOEST officials have been in the past. Some already have sponsored reading competitions. One sponsored a workshop to raise community-level stakeholders' understanding of child psychology and how parents can take a more active role in their children’s education. LGs have discretionary funds sufficient to support community-based organizations, such as mothers’ clubs, rural savings cooperatives, and youth groups, to continue some of the community mobilization activities currently being carried out by local NGOs. At least one mayor proudly declared, “We won’t need the [EGRP partner] NGOs anymore” [I.6.PD]. Finally, at least three issues described in the Background section make it unlikely all LEUs will be able to assume all EGRP responsibilities at the palika and school levels promptly: many posts are unfilled, there is resistance to reassignment on the part of LEU staff, and there are far fewer LEUs than the earlier RCs. Some time may be needed to sort out what minimally staffed LEUs can realistically manage EGRP activities, what training they will need, and how that training can be delivered as the LEU expands. Training for LEU, EDCU, and PEU staff will, by definition, be pilots that will need to be improved before they are rolled out to new NEGRP districts. Because of the limitations of the LEUs and the widely appreciated work of many RPs, all stakeholders at the school and palika levels and many above those levels insist that RPs, or some new version of them, be reinstated [I.26.PD]. Indeed, four of the six RPs that the team was able to interview were among the most knowledgeable and most appreciated mentors/monitors at the classroom level. At least one mayor said he would try to retain the highly effective interim RP after he was reassigned, but the RP was less certain the mayor would find and disburse his scarce discretionary funds for this purpose when the time came. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 45 4.5 EQ5. To what extent has the MOEST and CLAs built up the capacity at the central level to support the national scale-up of the NEGRP minimum packages at the federal and local levels? Many of the findings (and conclusions) for this section have already been covered in Section 4.2. The SSDP PIM for 2018/19 indicates that grant funds were allocated to scale up NEGRP in four more districts. The PIM, however, only shows what components have been authorized to receive funding; training protocols, guidelines, directives, and standard operating procedures are all better indicators of implementation readiness and, to a lesser extent, capacity. Training protocols and implementation guidelines have been developed as part of EGRP and by NGO EGR projects for all NMP components in generic terms. However, as shown in column 2 of Figure 13, the evaluation team failed in tracking down many relevant GON guidelines and training protocols. The lack of these documents was cited as evidence by several informants at the CLA level that MOEST and the CLAs, regardless of actual capacity, were not yet taking the lead on NEGRP. The few informants at the CLA level in a position to evaluate the GON’s readiness to support the national scale-up said they thought not. One education donor insisted that NEGRP should expand quickly, even though the model or the MOEST is not “ready”; people learn by doing, he said, not by preparing a model for them then expecting them to implement it. He spoke of expanding NEGRP to nine new districts in 2020-21 and nine more in 2021-22 [J.13.PC]. Tacitly acknowledging the GON was not ready, one senior GON official recommended scaling up NEGRP more slowly, considering available resources and capacity [J.3.PC]. Indeed, in its FY 2018-19 plan, the MOEST allocated a very limited budget to start NEGRP in four additional districts, and by April 2019, few informants cited progress in those districts [J.7.PC]. But 10 additional districts showed up in the SSDP PIM for 2019/20. If this analysis is correct, why might the CLAs and MOEST be unready to support scaling up? Several factors discussed in earlier sections may have a bearing here. Federalization and massive movement of civil servants from the capitol to the provinces and municipalities took up much time and energy in all CLAs in 2018/19. After the dust dies down, the MOEST, like all CLAs, has less staff to do the same work, and those remaining may not have been those oriented to/working with EGRP. Second, until the Federal Education Act apportions responsibility for implementing basic education between the MOEST, provinces, and palikas, it may be premature to invest time in writing detailed guideline and directives; LGs given primary responsibility for a task may not be willing to implement guidelines into which they have not provided substantive input. Third, the NMP summary continues to refer to what appears to be defunct Resource Centers – no resources or staff – and LEUs interchangeably, despite there being far fewer of the latter (e.g., 30 percent less in Dang). Planning for RCs should be different than planning for LEUs. Similarly, there is less than one ETC for every four or five in existence when EGRP was launched in 2015. Whether scaling up to new districts is fast or slow, all informants at all levels agreed that the MOEST and CLAs are not ready to move forward with NEGRP without further technical assistance and uniformly recommended extending donor support in the 16 NEGRP districts for at least another two, if not five, years to establish a viable but hopefully flexible EGR model. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 46 5. CONCLUSIONS AND RECOMMENDATIONS With a multi-level analysis of this type, the scope of conclusions and recommendations can vary dramatically. These are intended to cover only the priority issues that arose in the course of analysis. 5.1 EQ1. To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? As of March 2019, EGRP had demonstrated several viable components of a model for increasing EGR skills in the 30-50 percent of schools in the six sample districts. Stakeholders in schools expressed satisfaction with Nepali teaching and learning materials and teacher training. EGRP components for providing necessary in-classroom teacher support and essential community mobilization had been adapted for the current (2018/19) school year and remained to be evaluated. Both teacher support and community mobilization components appeared to continue to need more adaptation to the new federal context. The student assessment component also needs strengthening at the school level. Based on the sample, no model for school improvement is likely to be effective if schools do not have sufficient, motivated, Nepali-speaking teachers; sufficient classrooms; and regular attendance by teachers and students. Mother tongue materials are necessary to support non-Nepali teachers and students but have yet to be developed in more than one non-Nepali language and become widely available through the EGRP program. Lastly, the orientation, training, and support processes for the monitoring and evaluation component of the EGRP minimum package have been disrupted and cannot be finalized until the Federal Education Act is passed and implementation responsibility for basic education is allocated across the CLA, provincial, and local levels. 5.1.1. Teaching and Learning Materials Observation: At the classroom level, the TLMs provided are much appreciated; however, confusion remains about how to coordinate the use of the conventional textbook versus the EGRP workbook. The workbook can essentially replace the textbook and there is no time to complete both. Demand to complete the Nepali textbook remains strong in schools where the end of year exams – supposedly replaced by CAS – that draw exclusively on review questions found in the textbooks remain routine. Recommendation 1: For the 2018-19 school year, encourage the GON to issue a circular to schools that the EGR workbook is the primary text, and the old Nepali textbook should be used as a supplement. In the case of the integrated curriculum, ensure that teachers have clear guidance on when to transition to IC and what TLMs to use. Observation: Many school-level stakeholders describe SRMs as highly motivational, but they are not included in the NMP. Many informants mentioned this as something within the budgets of LGs. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 47 Recommendation 2: Reading corners also are included in the conditional grants line but may need to be more explicit in the PIMs to guide schools about how to use that budget line. Explore how NMP can incorporate reference to encouraging LGs/SMCs to fund SRMs through LG resources or ASIP. Observation: The difficulties of completing the EGRP curriculum are compounded by several factors relating to their production at the CDC. First, the workbook provides one lesson per official school day (192) rather than the number of days most classes meet (closer to 170). Recommendation 3: Reduce the number of Nepali/reading lessons for no more than 170 days, with 22 days of enrichment material. Observation: As of year 3 of the project, EGRP workbooks were still arriving late in many sample schools, compounding the lag between teacher training and when teachers start using EGRP in the classroom and creating confusion about which daily lessons to skip or how to combine them. Part of this delay was ascribed to delays in the CDC completing revisions, even though some informants reported a rushed and incomplete review process. Work on the integrated curriculum and other competing priorities appear to contribute to the rush and delays. Recommendation 4: Identify ways to support the CDC to ensure EGRP material development and adaption, especially in the context of integrated curriculum and L2 learners, can be prioritized. Observation: Given many factors that reduce the number of contact hours with materials in hand and teachers in place – high teacher and student absentee rates, many holidays, weeks without EGRP workbooks at the beginning of the year – 90-minute daily instruction periods likely remain necessary. Recommendation 5: Retain the 90 minutes of Nepali daily as the standard or in the case of the pilot integrated curriculum where the time has been reduced to 60 minutes of Nepali daily, study the effects of that reduced time, add more time if results suggest more time is necessary, and provide more supportive structured teachers’ guides that are easy to use, such as scripted lesson plans. Observation: The EGRA mid-term report shows persistent inequities in learning outcomes for L1 and L2 learners. Many MT materials generated by RTI NGO partners and other NGOs are not being used. In many classrooms, children speak more than one MT and limited Nepali as L2. Recommendation 6: NEGRP should promote wider use and availability of the existing EGR materials developed in MTs in classrooms where children speak no or little Nepali by making the materials open licensed and available on digital platforms for printing and reproduction. 5.1.2. Teacher Training and Classroom Support Observation: TPD is among the EGRP outputs most appreciated at the local level, although the ETCs revealed much could be done to strengthen it. Recommendation 7: CEHRD should incorporate three key units on EGRP into pre-service TPD as soon as possible. GON should review the effectiveness of HT as providers of TPS and look for other providers for TPS as necessary. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 48 Observation: Many teachers are managing classrooms with students that speak two or more non￾Nepali mother tongues. Recommendation 8: Provide support to CEHRD to ensure in-service TPD includes a basic introduction to teaching Nepali as a second language and well as for classroom management for small, large, and multi-lingual classrooms. Observation: According to our informants (see textboxes throughout Appendix 5), the EGRP high interventions are viable in at least 35 percent and perhaps as many as 50 percent of primary schools in EGRP districts. With adequate TPS (not the case currently), some informants suggest the model could succeed in as many as 70 percent of schools in some districts. TPS at the classroom level, however, has proved one of the outputs most difficult for EGRP to provide effectively. As currently planned in EGRP and the NMP, HTs are supposed to provide TPS but many have full teaching schedules to make up for chronic teacher shortages and do not have the time to provide TPS. Recommendation 9: CEHRD should prioritize TPS and restore a model like RPs with a job description that focuses on TPS, with a minimum of administrative responsibilities. Classroom-level coaches could use smartphones to make videos of good practice to implement better teacher and HT training and develop ways to share good instruction in a variety of Nepali classrooms with teachers. As more effective EGR teachers develop, explore ways to incorporate them into coaching roles. Observation: The importance of time on task is not a strong message in the EGRP program to date; certainly, teachers in FGDs and HTs did not place much emphasis on how they are trying to maximize it. Recommendation 10: In-service EGR-related TPD and TPS on increasing children’s time on task on reading through reducing absenteeism, starting class on time, using time well, and encouraging reading outside the classroom, should be emphasized in TPD. 5.1.3. Community Mobilization Observation: Community mobilization is critical to the success of EGRP by increasing time on task through improving teacher and student attendance, increasing pressure on HTs and teachers for better classroom-level by focusing more attention on CB-EGRA and EGRA results, and encouraging a culture of reading at home. The Cohort 1 community mobilization model was judged unworkable and the Cohort 2 model remains to be evaluated. The proposed NMP approach – orientation to prepare EGR￾linked SIP and one review meeting with stakeholders annually – cannot be implemented by LGs without substantially more strengthening, and it will not accomplish many of the critical functions described at the beginning of this paragraph. Recommendation 11: NEGRP should explore developing community mobilization partnerships beyond parents and teachers, with community-based organizations and youth in higher levels of schools where EGR is being implemented. EGRP also should focus more on encouraging elected officials to support community mobilization. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 49 Observation: TLMs and teacher trainings are the most appreciated inputs at the school and local level, but sufficiently trained teachers are beyond the ability of schools and palikas to replace each year. Replacing SRMs and funding community mobilization (e.g., reading competitions) are within the means of palikas/schools to sustain, but LGs and wards do not necessarily prioritize or understand budgeting for those. Recommendation 12: Prioritize strategy for providing TLMs and refresher teacher training in the long run through the GON systems. 5.1.4 Student Assessment Observation: The extent to which learning outcomes are not prioritized in stakeholders’ perception of school quality remains a major threat to EGR’s viability, sustainability, and scalability in public schools Although lower-level primary sections in some government schools are high performing, CB-EGRA and the performance evaluation baseline and mid-line studies suggest that most are not. EGRP is the first effort to improve instruction at these lower levels. Therefore, it should not be surprising that any improvement the project brings about is greeted with enthusiasm by stakeholders at the school level (teachers, HTs, parents, and SMCs). Modest improvements and incremental change that do not meet program and or national goals could be viewed as satisfactory to many stakeholders. Under these conditions, it may be difficult to motivate stakeholders (and others at the district or province levels to implement the program more intensely. Increasing understanding of and implementation of EGRA and/or CB-EGRA and CAS at the school level is an important strategy for moving the focus away from the end-of-year exams that are based on the traditional textbook. The NMP does not suggest implementation agents for this component, and the SSDP PIM 2019/20 assigns responsibility to municipalities, HTs, and SMCs that require significant training. Recommendation 13: EGRP implementers should identify agents to raise awareness of the importance of CB-EGRA, EGRA, and CAS at the school level and help schools to implement it instead of end-of-year examinations based on the traditional Nepali textbook. Several key rationales justifying the focus on EGR are still not widely understood or accepted at all stakeholder levels. These include: ● School quality must improve for socio-economic development to proceed on a more equitable, stable basis. There is no more equitable alternative to school quality improvement than EGR; school improvement efforts at higher levels of the system reach only a small proportion of children. ● School quality should be defined in terms of student learning, and nothing strengthens student learning for more children better than establishing strong foundational skills in the early grade of primary education. ● Strong skills in reading one’s mother tongue are the best foundation for learning other languages, particularly English. ● Strong reading skills are the result of good instruction and lots of practice time. ● Communities should support activities to increase children’s time spent reading outside of school. Social mobilization, in addition to community mobilization, is necessary to sustain EGR. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 50 Recommendation 14: EGRP implementers should develop and test messages that capture much of these rationales with various levels of stakeholders and the broader public. 5.1.5. Monitoring and Evaluation Observation: EDCUs are much reduced in their size and capacity compared with DEOs; The RP system has been dismantled, and Resource Centers, without staff or resources, provide meeting spaces only; there are far fewer LEUs than there were Resource Centers and many posts remain unfilled in rural areas, and LEU staff have largely administrative rather than pedagogical responsibilities. Few school￾and LG-level informants mentioned participating in the monitoring and evaluation activities mentioned in the SSDP PIM 2018-19. Recommendation 15: Identify the GON implementing agents for NMP monitoring and evaluation component. Observation: According to the findings above, highlighted in the boxes in Appendix 5, 30-60 percent of primary schools are too weak and dysfunctional to benefit from EGRP or any other program focused on academic achievement. Those schools distort any evaluation of the potential impact of EGR on the quality of education in Nepal. Recommendation 16: Experiment with applying different support for schools depending on their capacities and those of the local governments/LEUs. NEGRP implementers could use a school survey to categorize schools into categories, such as stronger, satisfactory, and unsatisfactory. EGRP should experiment with different levels/types of support for schools at different levels of quality. For example, where a school is unsatisfactory because of a shortage of teachers, extra funds may be allocated to hire a private teacher fluent in Nepali and, if necessary, local languages. EGRP implementers should disaggregate reports on EGRP impact by underlying school capacity. Observation: Where EGRP is working well, it has demonstrated some potential to address some of the school- and palika-level stakeholders’ most serious concerns about their schools: teacher motivation, student absenteeism, and the tendency for more aware parents to enroll their children in private English medium schools. Recommendation 17: To increase willingness to fund EGR with public and private funds, develop engaging social messaging that addresses the impact of EGR on these issues and disseminate beyond the schools. 5.1.6 Other Observation: All stakeholders agree EGRP will need to provide technical assistance for at least two to three more years to demonstrate a sustainable/scalable model for NEGRP: one year to make up for the earthquake and at least one year to adjust to federalism. Recommendation 18: EGRP should continue to be funded through the JFA, and additional technical support will be needed outside the minimum package in those districts. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 51 5.2 EQ2. To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? Observation: The main conclusions are summarized in Section 4.2 above. The Building Institutional Capacity scale that was designed at baseline to track the progress of the GON EGR service delivery at the central and local levels (see Baseline report, Annex 3 and 4), has been superseded by federalization and many changes in EGRP since 2016. Recommendation 19: Future institutional capacity building projects should include Building Institutional Capacity scales that should be updated every year as part of the Annual Work Plan review process. Observation: In-service TPD, although much appreciated, could be stronger. Few trainers have any first-hand experience with EGR. Recommendation 20: Early in the year, when ETCs are less busy, modest travel funds should be provided to allow ETC trainers to visit schools that have strong EGR teachers and make videos of teachers in action to use in training. Observation: A few informants at the central level insisted on describing the program the way it was laid out in the technical proposal not how it was working on the ground and pointed to guidelines and policy documents as evidence that the project has achieved its goals without reference to evidence of changes in school quality. Recommendation 21: Continue to build appreciation at all levels for the proportion of students achieving early reading benchmarks as the best evidence of school quality. Observation: The CDC needs more strengthening and perhaps staff to produce robust materials for the integrated curriculum, L2 support materials, and a range of MT necessary to scale up NEGRP. Recommendation 22: Provide technical assistance to CDC to develop a plan to ensure that: 1. Minimal national and international technical standards are maintained on child psychology and development, children’s literacy, time on task, and MT-based instruction. 2. Ongoing work on TLMs involves timely external review. That plan should include closer coordination with the Language Commission and better utilization of existing non-Nepali MT TLM produced by NGOs. 5.3 EQ3. To what extent were findings and recommendations of the 2016 report addressed? Observation: The baseline did not attempt to draw many conclusions and offered no recommendations. However, as shown in Figure 12, most threats identified in the baseline report do not appear to have been followed up on. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 52 Recommendation 24: The USAID Education Technical Team should sit with the implementing partners every six months to track progress on threats and recommendations identified in 2016 and 2019 performance evaluations. 5.4 EQ4. To what extent are LGs within the EGRP districts able to continue and sustain EGRP interventions? Observation: LGs are not ready to continue and sustain EGRP interventions. Moreover, federalization will likely produce several more disruptions to efforts to build institutional capacity for EGRP at the local level before the system normalizes. For example, some debates and delays are predictable around the release of the Federal Education Act; the new National Education Policy; and the circulars, directives, and guidelines that must be produced to operationalize those two. Additionally, how and when new permanent teachers certified by TSC will be integrated into schools and when they will be trained for EGRP is uncertain. Recommendation 25: EGRP should draft a focused LG-based institutional capacity strengthening plan with outputs of tested training modules for sustaining EGR. EGRP and CEHRD should identify the large number of institutional capacity building activities already focused on LGs. Drawing on insights from the grassroots generated by new EGRP program officers, the program should draft a focused LG-based institutional capacity strengthening plan, including incentives to participate in EGRP training. The outcome of the plan should be a series of tested training modules for sustaining EGR through LGs that can be rolled out through NEGRP. 5.5. EQ5. To what extent have the MOEST and CLAs built up the capacity at the central level to support the national scale-up of the NEGRP minimum packages at the federal and local levels? 5.5.1 Building Institutional Capacity at the Central Level Observations: In several areas, the MOEST and CLAs have come close to achieving “established” levels of institutional capacity, but a changed federal and civil service environment means that capacity to scale up the NMP has not been achieved. As noted above, the MOEST is not yet moving expeditiously to establish the standard operating procedures, guidelines, training protocols, and formal government directives necessary to implement the NMP on a large scale. Section 4.5 identified three potential reasons why MOEST and CLAs may not yet be taking the lead in moving forward with formalizing the guidelines and standard operating procedures to implement NMP promptly. In addition, the 2015 NEGRP Strategy that originally envisioned scaling up to all districts by 2020 has not been revised, which could create doubts about commitment at the highest levels and reduce motivation to undertaking the tedious guideline and directive revision process, lest it be unnecessary. Recommendation 26: EGRP should prepare to quickly implement new institutional capacity building activities for the MOEST and CLAs after the passage of the Federal Education Act and the Education Law, consistent with new responsibilities laid out in that act and law. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 53 Recommendation 27: USAID and RTI should work with the MOEST to encourage NEGRP strategy updating to reassure the MOEST and CLAs of the highest-level commitment and that the hard work of reformulating guidelines and standard operating procedures will be valued by the GON, not just the donors. 5.5.2 Scaling up NEGRP Observation: The 2016 baseline analyzed scalability in terms of four drivers: vision, evidence-based ideas, leadership (or champions), and incentives and accountability.23 For the most part, these drivers have not been addressed in the NEGRP scale-up currently being discussed in terms of four districts in 2019-20, nine new districts in 2020-21, and nine more the following year. A nationwide initiative demanding major sacrifices and heroic efforts from an over-extended system requires a major social mobilization campaign. Currently, there remains: ● No great enthusiasm to revise the 2015 NEGRP strategy, despite a request from the MOEST for technical support to revise the NEGRP core document and a GON task team to lead this process. ● No trumpeting of achievements of EGRP. ● No champions in the GON or out explaining EGRP to the public and generating openness to a new approach. ● No additional incentives planned to motivate a system that is under-resourced on many levels, ● Limited help planned for the CDC, although TLMs in many new languages remain to be developed quickly. ● No widely shared understanding of how EGR will help address the top-of-mind problems of HTs, SMCs, and parents: motivating teachers, providing a more solid foundation for learning at higher levels and for English, reducing student absenteeism, and increasing student engagement. Recommendation 28: USAID and EGRP should encourage the GON to undertake a major social mobilization campaign. The social mobilization campaign should be designed to 1) convert stakeholders at all levels to a different metric for school quality (i.e., learning), 2) convince them that reading and writing in the early grade in MT (Nepali or another tongue) is the best foundation for all further learning, particularly the English language. The goal of this campaign should be to convince the electorate and politicians to invest enough time and resources for at least five to ten years to see the results of NEGRP. 23 Cooley, L., & Johannes, L. (2014). Taking innovations to scale: methods, applications, and lessons Washington, DC: Results for Development Institute. September. 18. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 54 APPENDIX 1: EVALUATION STATEMENT OF WORK Statement of work for 2019 Performance Evaluation of Early Grade Reading Program (EGRP) USAID Nepal seeks CAMRIS International/MEL Project’s services in conducting 2019 performance evaluation of EGRP as outlined in the following statement of work: Task: Design and conduct an end-line for Performance evaluation of EGRP project. The performance evaluation will focus on the sustainability and replicability of EGRP interventions that are being implemented by RTI in collaboration with GON Ministry of Education, Science and Technology. The key performance evaluation questions include: ● To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? ● To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? ● To what degree have the findings and recommendations of the baseline performance evaluation been addressed? ● To what extent are Local Governments (LGs) within the EGRP districts able to continue and sustain EGRP interventions? ● To what extent have MOEST and CLAs built up the capacity at the central level to support national scale-up of the NEGRP minimum packages at the federal and local levels? The MEL project’s evaluation team will conduct an end-of-project study suitable for the performance evaluation of EGRP. In particular, the evaluation team will conduct a longitudinal study using qualitative data collection and analysis of institutional and human resources capacity. The study will cover the same six districts namely Banke, Saptari, Dang, Dhankuta, Sunsari, and Salyan as was done in the baseline. Concerned stakeholders also will be interviewed in the Kathmandu, Lalitpur, and Bhaktapur districts. Expertise/Personnel Requirement: USAID requires a team leader with the following qualifications24: ● Experience designing and leading baseline and end-line studies, evaluations in the education sector in developing countries. ● Experience in analyzing institutional capacities and sustainability of project interventions. ● Experience in working in Nepal desirable. ● Experience or familiarity with the EGRP performance evaluation baseline 24 Taking into account the expertise available within the MEL team, MEL can propose additional personnel needed for this study. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 55 The Team Leader should be assisted by a small team of experts in conducting the end-line study. The experts’ team must have experience in conducting similar studies in Nepal’s education sector in the past. Deliverables: 1. Work plan for conducting the EGRP performance evaluation end-line survey. The evaluation team leader will provide and present a written work plan for the conduct of the EGRP performance evaluation end-line survey, allocating roles and responsibilities among team members, and including survey instruments to be utilized for evaluation fieldwork. The plan also will include a detailed methodology for the end-line, including discussions about different categories of respondents that will be consulted during the study. 2. Qualitative data collection and analysis tools. The team leader will revise and adapt instruments for qualitative data collection and analysis for inclusion in the evaluation work plan and then train evaluation team members in their use during field testing/adjustment of the instruments. 3. Presentation of preliminary findings and conclusions. The evaluation team will analyze data gathered by the evaluation team itself for the performance evaluation end-line study and present their initial findings and conclusions to the DO3 Education Development Office to be ready to present for the GoN during the SSDP midterm review (MTR). The initial findings will be based on interviews with few stakeholders or might include findings from one-two districts. Those findings will be shared with an independent consulting firm responsible for supporting the SSDP MTR process by the end of March. 4. Draft EGRP evaluation end line report. The evaluation team will deliver a draft report of not more than 25 single-spaced pages in length, excluding appendices, but including an executive summary of not more than three (3) pages in length, that takes into account the comments and suggestions received during the presentation of preliminary findings and conclusions from analysis of performance evaluation qualitative data. 5. Final EGRP evaluation end-line study report. The evaluation team will deliver a final report of not more than 25 single-spaced pages in length, excluding appendices, including an executive summary of not more than three (3) pages in length, responding to comments from USAID on the draft report. 6. Tentative timelines. The timeline for this end-line will be determined in consultation with the EGRP Implementing partner, and USAID Education team and will be included by MEL in its work plan (deliverable #1). Project information: All project documents and updated project M&E plans need to be provided to MEL. For additional information that is needed for the design of this end-line study, MEL can consult USAID or the EGRP Implementer. The Mission will share the EGRP quarterly and annual reports to MEL activity. Other comments to consider for end-line: ● Update stakeholder list to include newly formed Local Government structures ● The end-line evaluation will need to address the threats identified for viability, sustainability, and scalability in the summary of the baseline on pages 7-9 and 18-24. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 56 ● End of project status institutional capacity built - revised to reflect current MOEST and LG structures under the federal structure. Perhaps the evaluation, should map out to the best of its ability in the current transitional phase where responsibilities existed and with whom they are now or non-assigned. This will be good to show where risks are to the sustainability of NEGRP implementation. For instance, if ERO was responsible but now no one is clearly responsible for an activity that is a risk for capacity building. This will also be key moving forward to understand where/at whom our capacity building TA should target. ● The district-level instruments will need to be revised to reflect current stakeholders. It is important to understand whether the district is conducting CB-EGRAs and whether those results have been shared with schools and parents. It will also be important to get a sense if people are new in their positions how many of them know or do not know about EGRP and to what extent. ● Can we get the school records for teacher absenteeism? ● In communities where Mother tongue interventions are to be piloted are local governments and communities supportive? ● Ensure that RTI’s comments on baseline reports are considered. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 57 APPENDIX 2: KEY DOCUMENTS FROM 2017 BASELINE REPORT APPENDIX 2A: BUILDING INSTITUTIONAL CAPACITY Table 8: Building Institutional Capacity EGRP Sub-IRs Latent Emerging Established Institutionalized EGRP IR1: Improved reading instruction, CL,A and number of districts CDC: Evidence-based early grade reading instructional materials designed, distributed, in use • Recognition of the need for change in curriculum and materials to support EGR • Familiarity with core EGR elements and five components scope sequence need for decodable and etc. • Limited buy-in; getting approval of materials may be difficult. • Recognition that students should have books or teachers should have teachers guides • Ability to develop materials reflecting knowledge of core EGR elements with assistance • Teachers begin to try implementing EGR practices. Most teachers show recognition of the importance of EGR. Majority show mastery of minimum/few elements of practice (i.e., competently teach activities for one-two components effectively) • Ability to develop EGR materials independently, assess, review and obtain approval (a/o assistance) • GON identifies the funding source (GON, DP, INGO’s) for TLM availability and distribution. • Teachers of school demonstrate the competency of elements of instrumental practice. • The CDC retains EGR experts or tries to involve EGR expertise in the CDC • Renewal of materials is part of the CDC’s work, not questioned or challenged but seen as a necessity • The CDC actively seeks global EGR evidence and integrates into revised materials • Students have TLMs and guide by 2 weeks of school starting (Assuming GON has financial commitment/budget for TLMs) • Most teachers demonstrate mastery of most EGR institutional techniques. NCED, ETCs: In-service professional development for teachers in public schools on reading instruction & materials in use • Recognition of the need for change in the way that teachers are teaching • Basic understanding of EGR core teacher practices/principles • First time development of initial in-service incorporating EGR practices with support • Development and implement 1st time 10-day TPD for EGR including lessons learned from the first-time initial in service with support in target districts • Regular/repeated use of 10-day TPD of EGR • Applied to all INGOs • Teachers get TPD credits for salary • TPD revision process is conducted independently • TPD implemented nationwide • Certification in EGR of TPD credits • Training made available for all primary school teachers Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 58 RCs, Monitoring & coaching for teachers in early grade reading instruction • Recognition of the necessity of ongoing support for teachers in EGR • Initial TOR for coaches/mentors established criteria laid out • Initial training for RMS • RMS making regular visits to schools in EGRP districts • Review and refinement of tools, training for RMS with support • RM/RPs are regularly supporting teachers, at least 2 times a month • Budget is planned, sources are varied (DP’s, INGO’s) • RM/RPs identified in all NEGRP districts, program districts (GON, INGOs) • Regularly review/revise training for RM/RPs on their own • Established position (RM/RP) assigned to provide ongoing support to teachers • Dedicated budget (GON source only) • Specific training program for RM/RPs to provide ongoing support to teachers • GON updates coach or motivates capacity based on global/evidence-based practices. EGRP Sub-IRs Latent Emerging Established Institutionalized • IR2: GON capacity to implement EGRP Early grade reading data collection and analysis systems • NASA-core language (reading) components need to be added • CB-EGRA knowledge • Central level: MOE/DOE/ERO sets up basic infrastructure to capture student reading data • District level: School communicates/clusters meet to learn about measuring reading +introduce EGRA/CB-EGRA tools • NASA includes core language component piloting • CB-EGRA- adopted tool￾piloting • EGRA 2014/2016 result + policy dialogue • EGRA data used to inform + set benchmarks • Reaching national consensus on how to integrate NEGRP into EMIS • CB-EGRA regular part of assessment system- formative, summative • Teachers doing summative • Report cards produced • Parents are shared/informed District level: • Teachers conversant, reflective and tracking student reading progress • Schools regularly (2 times per year), implement CB-EGRA and share the results with parents • Data goes to DEO offices for comparative analysis and planning support to the school. • EGRA Policies, standards, and benchmarks that support improved early grade reading • Central Level: MOE/DOE/CDC, etc. learn and discuss reading benchmarks • The CDC develops draft benchmarks • NEGRP steering committee formed, active, ratifying new policy decisions (materials) • Tentative work benchmarks completed by the CDC • International best practice to drive workshop August 15-16 • Time on task socialized • A thematic task force formed to standardize components of the reading program • Standard and benchmarks are ready • Stakeholders agreed to adopt the standard package • NEGRP meets GPE goals (DLI 1) • Policies, standards, benchmarks regularly tracked, reviewed and revised as necessary • PSBs adopted by INGOs /NGOs • All schools implementing a standard reading program Planning and management of financial, material, and human resources • Sector assessment • 2012 framework • NEGRP program document • USAID drives reading ahead on the ed. agenda • GON buy-in for reading budget line item created • TA contract to RTI for 2015- 2020 • RMs (Reading motivators) hired + trained • SSDP includes NEGRP • NEGRP trains teachers on reading • SSDP reflects components of EGRP • EGRP roll out plan is agreed • Development partners agreed to fund EGRP expansion (rollout plan) • NPC and MoF agreed to 7-year plan • 7-year (and onward) NEGRP. Funded and being implemented in all (75) districts of Nepal • Reflected in annual national development plans • EGRP activities are adequately budgeted for 5/7 years. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 59 • G2G 2014-2019 • Coaching (RMs) on regular salary seen as useful, used Utilization of data for course correction during implementatio n • 2014 Nationally representative EGRA sets initial benchmarks for reading • Design based on a suite of diagnostic tools to inform a strong design, tailored for Nepal • NEGRP task teams integrate all best practices in EGR (INGOs) + plan for NEGRP implementation based on lessons learned (will influence the future direction for all core tenants of reading) learned. • Systems at district region and central level are established with appropriate hardware and software • Users are identified and trained to use • Standard reporting formats are created with demos • Data flows established: Schools-districts-national level • Process in place to analyze and use data to implement fine-tuned EGR programs DoE: Classroom￾based & district-based early grade reading assessment processes • 2014 Nationally Representative EGRA by RTI • INGO assessments (SAVE) • CB-EGRA orientation + concept introduced • ERRA basics introduced • Adaptation workshop • New Era enumerators trained (Great cohort of reading assessors for Nepal) • Parents briefed on SB-EGRA results • GPE goals/objectives integrated into NEGRP • EGRA integrated into ERD assessment system, student data collected, analyzed and disseminated publicly District level: • DEOs know to use EGRA data, GON supports EGRA, NEGRP pays for EGRA at a later stage • CB-EGRA embedded/merged with the CAS (Continuous Assessment System) EGRP Sub-IRs Latent Emerging Established Institutionalized • IR3: Increased family and community support, number of districts Community awareness of the importance of the language of reading instruction • Communities are asking about the language their children are being taught in school • Communities are asking about teachers’ ability to teach in local language • Some community members feel that their children would learn better if instruction was in their own language • Communities are demanding or using reading materials in their own language • Communities are more organized • Communities are active in preparing local reading material in their own language (local language) • Communities are more active and realized the importance of the language of reading instruction • Communities are happy and actively contribute to after school activities • Communities are visiting schools to learn about the program • Local-level/community-based organizations/groups will support resources to some of the schools to buy materials in the local Nepali language. • Local organizations/ reading groups such as mother groups, VDCs will engage themselves on the importance of the language of reading Family engagement to support reading • Family members encourage children to read at home • Family members sometimes visit schools and ask teachers • Parents encourage/motivate their children to read at home • Family member/communities are making a favorable environment for their children by giving more time to them • Family/parents will provide time for children for reading and help their children to read Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 60 about their children’s performance at school • Some family members support children with their homework • Family/parents are regularly conducting/participating CB-EGRA with of school/teachers PTA/SMC contributes to quality reading instruction • Some parents attend SMC/PTA meetings and listen to discussions • Parents ask about teacher’s abilities to provide quality instruction to their children • Some parents ask for the instruction materials from the teachers • Parents sometimes ask their children about how they are learning at school • SMC will discuss children’s reading achievement at SMC meetings. • PTA/SMC are organizing a regularly scheduled parent￾teacher conference • PTA/SMC also are closely working with different community groups to organize community forums • PTA/SMC understand and are doing their responsibilities with more accountable towards children/parent for improving EGR • PTA/SMC will regularly monitor reading instructions • ECTR will be one of the priorities under SIP • Share reading progress/report with parents and community (accountability) Parent & community capacity to monitor reading progress • Parents ask their children to read to them and give them some feedback • Few parents check with teachers about their children’s progress at school during PTMs • Parents/communities/SMC show an interest in understanding how their children are doing, occasionally check their children’s exercise books, etc. • Parents will be interested to know their children’s reading achievement/progress at the time of PTC • Parents/communities increase their capacity to understand and contribute to CB-EGRA • Children’s reading programs have been shared and regularly discussed with teachers, SMCs, and parents • Parents are helping school/teacher to improve the reading habit of children at home • Parents are regularly meeting/visiting school and sharing knowing children feedback with school Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 61 APPENDIX 2B: END OF PROJECT STATUS: INSTITUTIONAL CAPACITY BUILT Table 1: Institutional Capacity Built IR 1: Improved early grades reading instruction Sub IR Sub-activities Demonstrating Viability/Sustainability/Scalability, with Decreasing Support from RTI Over Time CDC: Evidence￾based early grade reading instructional materials designed, distributed, in use 1.1.1 Collaborate with the MOE and CLAs to develop/revise scope and sequence, TLM, and other student material for G1-G3 reading instruction in Nepali for students who speak Nepali as a first language. The CDC maintains up to date EGR materials and development plan and provides new staff with an orientation to EGR materials development. 1.1.2 Collaborate with the MOE and CLAs to develop/revise scope and sequence, TLM, and other student material for G1-G3 reading instruction in Nepali for students who speak Nepali as a second language (NSL), including appropriate strategies for transition from mother tongue (MT). The CDC uses evidence-based learning outcomes and scope & sequence to develop all teaching and learning materials across Nepali, NSL, and MT curricula. 1.1.3 Collaborate with the MOE and CLAs to develop/revise scope and sequence, TLM, and other student material for G1-G3 reading instruction in Tharu, Awadhi, and Maithili and provide technical assistance for other languages. 1.1.4 Collaborate with MOE and CLAs to produce accompanying teachers’ guides for G1-G3 for each set of materials. 1.1.5 Materials printed and distributed. The CDC prints and distributes evidence-based teaching and learning materials in the languages prioritized by the government 1.1.6 Incorporate provisions for expenses related to the recurring costs of providing EGR materials into Village Development Committee (VDC) and district-level annual budget plans VDCs and DEOs incorporate the cost of providing EGR materials into recurring budgets. 1.2 NCED, ETCs: In-service professional development for teachers in public schools on EGR instruction and materials in use 1.2.1 Collect baseline data on teacher knowledge, skills, and attitudes (KSA). Teacher training is adapted to a real-time assessment of teacher practices and understanding. 1.2.2 Employ behavior change communication (BCC) strategies targeting teacher behavior change. Effective BCC strategies targeting teacher behavior fully integrated into teacher training/support as NEGRP spreads to new districts. 1.2.3 Provide training of master trainers to all target districts, VDCs, and clusters on the use of the new materials packages for G1-G3 reading instruction. Effective master TOT training ready to roll out as EGR spreads to new districts. 1.2.4. Provide technical assistance to MOE to conduct TOT for all target districts, VDC, and cluster RPs on the use of materials packages for G1-G3 reading instruction. Effective TOT training ready to roll out as EGR spreads to new districts. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 62 1.2.5 Provide technical assistance to MOE to train all G1-G3 teachers on the use of the materials. packages for G1-G3 reading in Nepali. Effective training for teachers in EGR in Nepali as a first language, NSL, and MT, including transitioning to Nepali, ready scale out to more districts. 1.2.6 Train master trainers and teacher trainers on transitioning to Nepali as the language of instruction by Grade 4. 1.2.7 Provide technical assistance to MOE to conduct TOT for transitioning to Nepali-only for reading instruction by the start of G4. 1.2.8 Provide technical assistance to MOE to train all G1-G3 teachers on the use of the materials packages for all NSL students. 1.2.9 Provide technical assistance to the MOE to conduct orientation workshops for headteachers on EGR. Effective school leadership curriculum to support EGR ready to scale out to new districts. Headteachers play an active role in supporting and supervising EGR programs. 1.3 RCs and DEOs: monitoring for teachers in early grade reading instruction 1.3.1 Support cluster and district-level partners to implement criterion-reference classroom monitoring of EGR instruction. District and RC-based monitors use common criterion-reference for monitoring EGR instruction. 1.3.2 Develop and test non-monetary systems for incentivizing teachers. Teachers associate professional advancement with good EGR instruction. 1.3.3 Provide technical assistance to plan for teacher mentoring visits to classrooms implementing EGR. HTs, district supervisors, RPs, and RTs make regular, effective visits to EGR classes. 1.3.4 Incorporate monitoring/transport provisions into DEO/VDC budgets. School supervisors and RPs/RMs have the means to conduct regular visits to EGR classrooms 1.4 Classroom￾based and district-based EGR assessment processes improved in target districts 1.4.1 Create item banks for the generation of EGRAs that are equated for the same language and comparable across languages. Districts construct and implement their own NEGRP progress monitoring, consistent with national standards. 1.4.2 Support districts to generate LQAS based on EGRA or ASER-like assessments. 1.4.3 Improve the capacity of district monitors and inspectors to administer EGRA in all languages. 1.4.4 Collaborate with district monitors/inspectors to collect data using LQAS. 1.4.5 Provide TA to districts to plan for LQAS. 1.4.6 Assist districts to analyze and report their LQAS results. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 63 IR 2: Improved National and District EGR Service Delivery Sub IR Sub-activities Demonstrating Viability/Sustainability/Scalability, with Decreasing Support from RTI Over Time 2.1: Early grade reading data collection and analysis systems improved for NEGRP 2.1.1: Provide technical assistance to improve government data collection capacity (enhancing EMIS) Routine annual school census forms incorporate teacher assignment, pupil/teacher language, GIS items into (EMIS source). 2.1.2: Update data collection systems – the language of communication by students MOE uses enhanced EMIS to track learner language diversity and teacher assignments by grade, language, 2.1.3: Update data collection systems – and qualification to inform teacher assignments and materials production/distribution. the language of communication by teachers 2.1.4: Provide technical assistance to school mapping from the linguistic point of view MOE and DOE use enhanced EMIS to develop school maps with linguistic data to guide timely annual assignment of teachers and language materials. 2.1.5: Improve NASA for nationwide EGR data collection CB-EGRA incorporated into NASA and administered system-wide. Central EGR database and reporting tools continue to be used after the program 2.1.6: Incorporate EGRA results into EGR barometer-type tool. ERO effectively cleans, processes, and analyzes EGRA and CB-EGRA NASA data systems using up-to-date learning standards for EGR. ERO makes results of up to date EGRA available to all levels of the education system, including online access for VDCs Summary EGRA results stimulate discussions about targeted approaches to improving EGR at all levels of the primary education system. 2.1.7, 2.1.9, 2.1.11: Conduct sample￾based language, teacher mobility, and community attitude and belief surveys in a sample of schools in the EGRP districts. ERO conducts teacher mobility studies and language surveys in all districts ERO contracts for language-related, teacher observation, time-on-task, or other formal studies as needed to continue improving EGR performance. DEO uses ERO studies and surveys to improve EGR instruction and support in non-program districts. 2.1.8, 2.1.10: Develop language maps for use by policymakers to mobilize resources ERO conducts and regularly updates teacher mobility and language surveys in all districts. Policymakers use language maps to inform strategy, allocate resources, and deliver necessary support for EGR. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 64 2.1.11 Investigate parent and community attitudes and belief regarding mother tongue See 2.2.4. DEO regularly assesses parent and community attitudes and beliefs regarding mother-tongue instruction in the early grades. Targeted, evidence-based behavior change campaigns encourage confidence in the mother tongue. 2.2: Policies, standards, and benchmarks that support improved early grade reading instruction institutionalized for EGRP 2.2.1: Develop standards framework for student performance 2.2.2: Develop standards framework for teacher performance Evidence-based standards framework for student performance in Nepali and major local languages are widely disseminated and used for planning in DOE and DEOs. MOE uses evidence-based teacher standards for Nepali and mother tongues to identify and strengthen teachers who do not meet standards. 2.2.3: Develop and implement time-on￾task policy All DEOs adjust curriculum timetables to enable 90 minutes daily of EGR instruction in G1-G3. Training materials for HTs and DHT’s incorporate school-level strategies for providing 90 minutes of daily EGR instruction in G1-G3. 2.2.4: Finalize medium of instruction (MOI) for each target school TBD. For example, the medium of instruction for each target school determined in consultation with communities and MOI status widely distributed through the education system. 2.2.5: Finalize and apply policies for mitigating teacher mobility TBD based on consultations with the team lead, USAID/Nepal, and RTI. 2.3: Planning and management of financial, material, and human resources devoted to early grade reading improved 2.3.1: Develop and keep up to date a five-year plan NEGRP strategy translated into feasible action plans that are informed by the annual school census and other data described above, reviewed semi-annually by stakeholders, and revised as needed. All DEOS, RCs, teachers, and HTs have been trained on and use NEGRP guidelines/manuals. NEGRP performance monitoring plan flows down to NEGRP manual and district EGR implementation and monitoring plans 2.3.2: Develop textbook and materials procurement and distribution plan The CDC has an evidence-based publication plan and specifications to optimize production value and affordability. National and district-level materials procurement, distribution, and tracking system in place for G1-G3 Nepali and mother tongue instructional materials in line with language mapping. A textbook tracking system developed and implemented in all districts. 2.3.4-2.3.5: Determine the match between teacher assignments and the MOI for a given school and develop/implement a plan for rectifying mismatch between teacher assignment and MOI 2.3.6: Implement a plan to reduce teacher mobility To be determined based on consultations with the team lead, USAID/Nepal, and RTI. Teacher unions support policy to reassign teachers based on school MOI, including teacher incentives. 2.3.7: Policy workshop regarding the MOI adjustment process To be determined based on consultations with the team lead, USAID/Nepal, and RTI. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 65 Increased Family and Community Support for Early Grade Reading - IR 3 Sub IR Sub-activities Demonstrating Viability/Sustainability/Scalability, with Decreasing Support from RTI Over Time 3.1 Community awareness of the importance of the language of instruction (LOI) increased in target districts 3.1.1 Design and implement social and behavior change communications (SBCC) campaigns SIP, VEP, and DEP incorporate EGR advocacy in their plans. Radio Nepal and FM radios broadcast EGR messages and programs about the importance of LOI as part of their regular programs. District specific grassroots campaign strategies incorporate tools for extracurricular and home-based learning opportunities. 3.1.2 Discuss classroom-based assessment results and language of instruction with communities Village Education Committees/SMC advocates for local resources through VDCs and districts drawing on the schools’ reading improvement plans. In line with social/community mobilization strategy/guidelines, local NGOs facilitate VECs, SMCs, and PTAs. 3.1.3 Promote after-school EGR-related activities in MT VECs/SMC receive matching grants for their contributions to activities that support reading, access, and after school materials development. 3.1.4 Involve communities to develop after school reading materials VDCs, VECs, and NGOs provide technical guidance to support local writing workshops and the production of materials and publications. 3.2 Family engagement to support reading increased in target districts 3.2.1 Produce home-school materials for EGR that parents and families can use Parents and family members use locally developed and produced after school reading materials in selected language in their homes to support reading acquisition. Local communities use guidelines produced under EGRP to develop after school EGR reading materials. 2.4: National standards for early grade reading improvement adopted and geographically expanded 2.4.1: Leverage sufficient resources to support EGR at the national level A standard set of program-based budgeting tools for EGRP planning and budgeting purposes is being used at all levels of the system. Regular advocacy on the part of CSOs for adequate budgeting for EGR. The MOE and DoE request and receive resources in line with the NEGRP budget for two years in a row. 2.4.2: Standardize the government contributions to EGR from NGOs and INGOs MOE engages NGOs and INGOs to contribute towards meeting EGR outcomes and NEGRP goals 2.4.3: Seek private partner support regarding the steady supply of EGR materials Significant public/private partnership support for EGR. Nepal Early Grade Reading Program Performance Evaluation 2019 Contract Number: AID-367-C-15-00001 66 3.2.2 Develop an approach to work with illiterate parents and caregivers Illiterate parents are aware of their children’s language development and engage in locally developed, culturally relevant, home-based activities to advance their children’s reading achievement. 3.2.3 Propose innovative methods to engage both parents and children to use cell phone applications for home-based EGR practice Parents engage in SMS –Audio in support of developing their children’s reading habits. 3.3 PTA/SMC ability to contribute to quality reading instruction increased in target districts 3.3.1 Build on current best practices for community mobilization PTA’s/SMCs and RCs draw on research-based technical guidance and best practice in designing community mobilization initiatives to promote EGR at the village level. 3.3.2 Identify and mobilize local NGOs to provide support to PTAs/SMCs Local NGOs implement large-scale district-wide reading improvement programs. 3.3.3 Support PTA/SMCs and RCs to hold community events in support of EGR PTA/SMCs plan and implement community events in support of EGR using good financial and organizational management. 3.4 Parents’ and the community’s capacity to monitor reading progress increased 3.4.1 Parents support teachers to use measurement tools EGR end-line assessment shows the quantifiable change in parental attitudes toward reading instruction in selected languages of instruction and in parent and community support to reading instruction. Teachers hold semi-annual parent-teacher conferences where they discuss the results of continuing the assessment system. 3.4.2 Organize community forums to share reading data Stakeholders are aware of children’s reading progress, goals, and purpose of NEGRP, CB-EGRA results will be shared at various community forums, baseline data on learner performance are in place, and priority action planning is facilitated drawing from school-developed activities and evidence-based initiatives. 67 APPENDIX 2C: DETAILED METHODOLOGY The performance evaluation25 will answer two questions posed by USAID (see Exhibit 4) using qualitative, longitudinal methods, including baseline (described in this report), midline, and end-line studies. None of these studies will be nationally representative or statistically representative of the 16 EGRP districts. The study assumes that building capacity to sustain and scale EGRP is complex, and there are many pathways to results, with multiple actors and factors that can influence the change model. This baseline, therefore, attempts to identify the actors and factors that might influence change. Moreover, Nepal is a diverse country with three ecological zones, 75 districts, and 100+ languages. This suggests the EGRP model may have to be adapted in many contexts to have the desired outcome for all children. The baseline, therefore, also attempts to identify some of the most salient factors and the types of variation they may require in EGRP. The research design is qualitative and, therefore, cannot establish causality; rather, it can help explore the mechanisms behind cause and effect relationships. The primary research described here attempts to understand the mechanisms by which EGRP is succeeding or falling short in “demonstrating a viable and sustainable model for increasing EGR skills that could be scaled to other districts” and “strengthening GON EGR service delivery capacity?” The first section below explores the meaning of “demonstrating,” “viability,” “sustainability,” and “scalability” under the heading of the theory of change.26 The following sections describe sampling; data collection, management and analysis; and preparing data for future analysis. TERMINOLOGY EGRP is to “demonstrate” rather than “produce” a “viable, sustainable model that could be scaled up to other districts.” This evaluation assumes that the audience to whom this demonstration is directed includes, at a minimum, the staff at all relevant levels of the GON. Additionally, given that few education innovations worldwide have been brought to a sustainable standard in less than 10 years, we infer the audience also is the education donor community that could potentially support GON efforts to increase the flow of funds to EGR activities after EGRP has ended. Note that most donor funds for education in Nepal now flow through GPE, with the power for allocating them among competing for activities largely 25 This design incorporates the results of an evaluation design planning workshop held on August 3, 2016, at the MEL-CAMRIS office in Kathmandu that included all senior stakeholders from USAID and RTI International and the EGRP-PE Baseline Study team. See Annex 2 for a description of this workshop and the institutional capacity/viability scales it produced. 26Note that a theory of change was not explicit in the EGRP statement of work but it is now strongly recommended in USAID ADS 201, partial revision, 3/23/2017. https://www.usaid.gov/sites/default/files/documents/1870/201.pdf Exhibit 4: Key Evaluation Questions 1. To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? 2. To what extent did EGRP strengthen GON EGR service delivery? 68 in the hands of the GON. The GPE has given an initial vote of confidence to EGR by integrating it into a disbursement linked indicator (DLI). To “demonstrate” an innovation suggests more convincing of and engagement by future adopters of that innovation than simply “showing” them what innovation can do. As such, we assume that the degree to which GON staff at all relevant levels are engaged in implementing or, at a minimum, participating in project activities will influence the degree to which they find the EGRP’s demonstration convincing. Also, to the degree the donor community is an important audience, other donors should be included in reviewing reports, visiting sites, engaging in project-related discussions Viability Before scaling up or sustaining an innovative model, it is necessary to determine whether it is viable. In this case: Viability: The degree to which the EGRP model can be implemented in the target districts over the next four years within existing GON structures by GON staff at an agreed-upon level of effectiveness at an acceptable cost. A design planning workshop produced a series of preliminary scales to measure the extent to which the GON demonstrates the capacity to implement various aspects of the EGRP with fidelity. The workshop produced a matrix summarizing those scales, arranged by sub-IRs (see Annex 3). The items in the “latent” column reflect the most basic level of understanding/knowledge of EGR, and management effectiveness; “emerging” reflects attitudes, training, and first-time implementation experience; “established” reflects the routinization of activities; and the “institutionalized” column reflects the ideal outcomes for each sub-IR as listed in the EGRP statement of technical approach. Note the “institutionalized” outcome is not a specific deliverable for the program; rather, the expected outcomes of the program lie somewhere between “established” and “institutionalized.” “Utilization of data for course correction during implementation” is not a standalone sub-IR, but it is central to the program’s theory of change. IF the GON staff gathers the data and works with RTI to analyze it in ways that have implications for project implementation, THEN it will be more willing to adjust implementation and improve project performance during the life of the project. This is critical BECAUSE viability in one context and later scaling out to new contexts often demands reinventing/adapting the model as the context—language, geography, population density, livelihoods, and cultural factors, for example—changes from geographic area to the next. During the early stages of scaling up in a new context, context-specific data should be regularly collected and used to reinvent/adapt the original model as necessary. Sustainability The term “sustainability” was originally applied to agriculture and environmental projects, not to education or even institutional capacity-building efforts. For this evaluation: 69 Sustainability: The ability of the EGRP to generate a longer-term focus on the importance of EGR in Nepali primary schools; to strengthen the GON’s willingness and ability to continue providing the resources and leadership necessary to support an evidence-based approach to EGR; and to enable Nepali schools to promote high levels of EGR achievement for all children.27 This evaluation is undertaken prospectively, ex-ante, while there is time to conceptualize and test key sustainability factors better. Capacity and ownership are strong proxy indicators for sustainability. USAID Forward associate sustainable development with “high impact” public-private partnerships.28 Both local NGOs and local private sector philanthropists already appear to be active on a small scale in many Nepali schools. Several EGRP components aim to engage the private sector in program-related activities and to engage the GON in working with the private sector, mainly local NGOs but also community-based, small-scale private philanthropists. The evaluation will, therefore, track progress in these two modalities, including the ways that the GON and Nepali NGOs do or do not reinforce each other’s efforts to sustain some or all of the program’s intended achievements. The following measures of sustainability are proposed. With decreasing support from RTI: • GON has the political will and intention to sustain EGRP in the districts where it has been introduced. • Policy framework is updated and implemented to accommodate the project. • Governance structure is improved (including coordination, participation, transparency, and accountability). • Organizational structure is improved (mechanisms, systems, and procedures). • Capacity of GON officials to carry out and institutionalize EGRA is improved. • Capacity of GON and NGOs to work collaboratively around EGR is improved. • The GON allocates funds to sustain EGR activities, whether from GON recurring budget or its development budget.29 Scalability The GON’s National Early Grades Reading Program assumes that EGRP will scale up quickly across all Nepali districts. The international development field has recently produced a score of major papers and reports exploring the requirement for scaling up innovations quickly, including some focused specifically on basic education. For this evaluation: Scalability: The ability of EGRP to rapidly expand and adapt to 59 other districts in order to reach a significantly greater number of children. Several features of an education innovation, such as EGRP, are likely to speed scaling out beyond the area where it is initially invented and piloted.30 Several components of EGRP address these features: 27 Adapted from Shroter, D. (2010). Sustainability Evaluation Checklist Kalamazoo, Michigan: The Evaluation Center. Western Michigan University. August. P. iv. Retrieved from: http://usaidprojectstarter.org/sites/default/files/resources/pdfs/SEC￾revised.pdf. 28 “In order to achieve long-term sustainable development, we have to support the institutions, private sector partners and civil society organizations that serve as engines of growth and progress for their own nations.” USAID Forwardembraces new models for public-private partnerships and increased investment directly to partner governments and local organizations.” Retrieved from: https://www.usaid.gov/usaidforward, 6/15/2016. 29 One donor suggested one intermediate indicator of GON commitment to EGR: the GON allocates GPE resources to EGR activities. 30 Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press. 70 • Relative advantage vis-a-vis the existing model. Much thought has been invested by the implementing organization and by USAID, vis-a-vis its external evaluator, NORC at the University of Chicago, in establishing control schools to which intervention schools can be compared. It also assumes that relative advantage will change over time as EGRP partners use feedback from implementation to adapt their model to improve effectiveness. • Compatibility with the existing system. EGRP is assisting the government at all levels to develop EGR-strengthening components that are compatible with many parts of the existing primary education system. • Trial-ability. The two-phase EGRP design allows relative advantage to be measured in just five years, a much shorter time frame than is usually necessary for educational innovations. • Cost relative to the existing model. The government has committed to supporting many other components of the primary education system that remain, nonetheless, underfunded. Once the incentives/conditions for using large amounts of funds for EGRP have expired, strong ownership will be important to maintain support for the innovation. • Evidence of efficacy. Credible evidence that EGRP improves learning outcomes and helps save money in the short- to medium-run, system-wide, is expected to increase government ownership. One important feature with a significant impact on scaling up, however, constitutes a problem for most education innovations. Complexity makes all education innovations difficult to scale out relative to other sectors, such as agriculture and public health, where results can be measured in less than a year and sometimes in just minutes. In addition, features of the system in which the innovation is embedded also play a role in scalability. A recent Brookings paper includes a Scalability Assessment Tool developed by MSI International that captures many of these issues. The evaluation will adapt this tool and structure questions around four “drivers” for scaling up that are suggested in the same paper:31 • Vision. The MOE collaborates with RTI and other experts to develop a vision at the outset to scale the program. • Evidence-based ideas. The MOE draws on evidence to inform ideas to scale the project after it is complete. • Leadership. Champions exist in the MOE to scale the project. • Incentives and accountability. MOE staff have the conviction and motivation to scale the program. Mechanisms for accountability and transparency are in place to ensure that the program is positioned to scale. The method EGRP has chosen for scaling up is quantitative replication, for which there are several methods outlined for public-private partnerships in education in recent years.32 Given the need for flexibility as described above, in many instances, NGOs pioneer scaling up innovations in new contexts and governments and adopt those innovations once they have been proven successful. The evaluation will track public-private partnerships in the sample districts. 31 Cooley, L., & Johannes, L. (2014). Taking innovations to scale: methods, applications, and lessons Washington, DC: Results for Development Institute. September. 18. 32 Jowett, A., & Dyer, C. (2012). Scaling-up successfully: pathways to replication for educational NGOs. International Journal of Educational Development, 32(6), 10. 71 To frame the evaluation, the MEL project team drew on RTI’s Year 2 work plan and the statement of work in Contract Modification 1, which identifies anticipated activities and results. The team developed illustrative measures of viability/scalability/sustainability that align with sub-activities and results presented in the work plan relating to IR 2. As noted above, these measures have been validated with USAID/Nepal and RTI as part of the design activities for this study. As part of that validation process, CAMRIS International, Inc., USAID, and RTI identified a subset of these activities that could benefit from a more in-depth qualitative approach that would add value to ongoing program M&E. In general: • EGRP performance in program districts speaks to viability. • Performance in Cohort 1 districts in Phase 2 speaks to sustainability, as does commitments to EGR in budgets and planning documents at all levels of the system. • Evidence of EGR interventions and positive attitudes towards EGR in non-EGRP districts speaks to scalability. Sampling and Bias The analysis is based on 256 in-depth interviews at the central, district, cluster, and school levels. While this number is relatively large for qualitative research, overall, it is nonetheless too small to represent in a statistically significant way the four treatment districts, much less the 16 in EGRP or the 77 nationwide. While it is consistent with good research practice, the description below identifies where the design or implementation of the research may have introduced bias and, where possible, the mitigation efforts undertaken. Central Level Stakeholders At the central level, as shown in Table 1, we interviewed all key counterparts in the main Central Level Agencies as well as several key counterparts who had recently been reassigned to agencies beyond the scope of EGRP. We also interviewed most of the INGOs previously or currently involved in EGR activities and several foreign donors active in the primary education sector. We used interviews with each type of organization – GON, INGO, foreign donor – to triangulate or verify information provided by other types of organizations. In this way, although our sample size was small, it is relatively comprehensive of stakeholder organizations and we have some confidence that our findings indeed reflect the situation at the central level. Table 2: Informants at the Central Level Types Central-level Agencies Governmental CBS, CDC (3), DOE, ERO, MOE, NCED, TSC Non-governmental PLAN, Room to Read, Save the Children, World Education, World Vision [missing: VSO] Donor partners UNICEF, World Bank [missing: UNESCO, WFP] 72 Regional, District, and Cluster-level Stakeholders At the district, cluster, and school levels, our sample was purposive, not statistically representative. Among the 16 EGRP districts, we identified two groups of three districts, including one Cohort 1 and one Cohort 2 district and a third (Cohort 3) district that bordered on the first two but was not itself participating in EGRP. We specifically excluded the mountain districts due to time constraints; those selected included both Terai and hill districts. We hoped these three types of districts would provide some idea of the range of responses to EGRP. In the field, however, we discovered one of the Cohort 3 districts was serving as a control for the EGRP impact evaluation and, therefore, had some exposure to EGRP concepts. We, therefore, designated the district with no formal exposure to EGRP as Cohort 4. Our sample, thus, allows comparisons between districts with four different levels of treatment, without being able to assign a level of probability to our results. Table 3: Exploring Variability, Viability, Sustainability, and Scalability Performance Evaluation Stages Type of District Baseline Midline End-line Cohort 1 Banke, Saptari Variability of the primary education system in EGRP districts Viability of a high intervention model Sustainability of a high intervention model Cohort 2 Dang, Dhankuta (H) Viability of a low intervention model Viability of a high intervention model Cohort 3 Sunsari Variability of the primary education system in non￾EGRP districts Scalability with minimal EGRP contact Scalability with minimal EGRP contact Cohort 4 Salyan (H) Scalability with no direct EGRP contact Scalability with no direct EGRP contact Note: All districts not marked (H) for Hills are in the Terai ecozone. At an evaluation design planning workshop on September 3, 2016, USAID, RTI, and CAMRIS developed a scale for measuring changes over time in institutional capacity for each of the project’s three IRs. In each of the six districts, we interviewed the one or two school supervisors associated with the target Resource Centers. For example, in each of the three Cohort 1 districts, the EGRP Focal Person was a school supervisor and might even supervise one of the target Resource Centers, in which case we interviewed only one additional school supervisor. In another district, one school supervisor was responsible for both target Resource Centers. Table 4: Informants at the Regional, District and Cluster Levels Level Target Informants at the Regional, District, & Cluster level (# interviewed) Regional Education Training Center Directors (4) District (6) DEO (6), EGRP Focal Person (5*), EGRP-Coordinator (4), School Supervisors (8, matched to resource centers), local NGOs (13), Teacher Associations (6), journalists (2) Cluster (2) Resource persons (12), Reading motivators (4) Note: In the district included in the impact evaluation’s control group, one School Supervisor identified himself as the EGRP Focal Person. 73 School-level Stakeholders Within each district, we selected two clusters of schools, and in each cluster, the resource person (RP) helped staff identify the best school associated with their respective Resource Center and a second school with a different language profile. No schools were more than two hours' walk from the nearest road. Within those schools, one Grade 3 classroom was selected for a Rapid Classroom Appraisal; where there was more than one Grade 3 classroom in a school, the HT (HT) chose one. Thus, we expect the selected classroom to be of average or above quality. The team asked the HT to help call local NGOs, School Management Committee (SMC), and Parent-Teacher Association (PTA) members, parents, and benefactors to the school for interviews. Table 5: Informants at the School Level Level Type (# interviewed) School (28) HT (28), SMC chair (28), PTA member (21), Guardian of G1-G3 child (22), local NGO (6), private benefactor (9), teacher focus group (27), Rapid Classroom Appraisal (28) This approach introduced at least two potential sources of bias. First, the RP might be unwilling to send outsiders to a truly poorly performing school. Second, the team’s time limitations precluded visits to the most remote schools. In the rest of the world, remote schools tend to be relatively poor performing. Both these biases mean that findings at the school level, therefore, may not adequately reflect the lower end of school quality. The timing of the fieldwork also introduced bias in an unknown direction. Many official national and local holidays shut down schools during this period. Also, many teachers and students took unofficial holidays on the days adjacent to the official ones. Therefore, in the schools that were visited on unofficial holidays (i.e., days adjacent to official holidays), headteachers said a) attendance was low and 2) students living close to the school were more likely to be attending. This may have affected a few items on the Rapid Classroom Appraisals. However, in what direction remains undetermined. At the school level, interviewers requested the headteacher (HT) to call: • One or more guardians. • One or more local NGOs/philanthropists that had provided support to the school. This method of selection introduced two potential sources of bias. First, it was unlikely the HT would call individuals critical of the school. Second, the guardians available within walking distance of the school, able to come for an hour or more in the morning or afternoon, might not be representative of guardians who were occupied outside the home on a weekday. 74 Data Collection, Management, and Analysis: Four field team members—two men and two women—were recruited based on prior experience carrying out in-depth interviews, their ability to work with children, and fluency in one or more of the local languages likely to be encountered in the selected districts. They had no prior experience specific to primary education or early reading. The senior national consultant agreed to serve as the field team leader for one female and one male interviewer working in one group of districts. The senior national consultant recruited a second field team leader with a master’s degree in education who led the second female and male interviewers in the second group of districts. The team leader and the senior national consultant drafted interview instruments and protocols for each type of informant at each level of the education system. The senior national consultant and the MEL evaluation specialist then trained the field team members on these protocols and instruments over two days. During this time, several documents to ensure the data was gathered with the informed consent of the interviewee were prepared in English and translated into Nepali. These included an introduction letter from the MOEST, a one-page explanation of the purpose of the study, and an informed consent form. The international team leader observed one day of the two-day pilot and spent four days in the field in one group of districts. The team tested the draft instruments and informed consent protocols during a pilot in Bhaktapur, August 21-22. Following the pilot, the Nepali team spent one day revising the form and interview protocols, then each field team proceeded to their group of three districts. The final documents and informed consent protocols are in Annex 4. The field teams were in the districts feom August 26 – September 9. The field team leaders focused on interviews at the district, regional ETC, and cluster levels; other field team members conducted most of the interviews at the school level, joined by the field team leader after interviews at other levels were complete. Teams arrived at the school before the start of classes; the schedule allowed one day per school and time to return if key informants were not available on the day of the initial visit. Handwritten notes from the central level English medium interviews (mainly CLA) were transcribed into Microsoft (MS) Word. Interviews at the district, cluster, and school-level were conducted and handwritten in Nepali. These were later simultaneously transcribed and translated into English by professional translators and entered into MS Word. Each interview was then assigned a unique identifier and analyzed for themes using Dedoose software. The international team leader conducted much of the content analysis. For the school-level interviews, she was assisted by a Nepali research assistant with experience in primary education in Nepal. As an informal test of inter-rater reliability, both the team leader and RA coded the headteacher and DEO interviews individually and found little variation. At the time of coding, descriptors for cohort, district, cluster, and school, as appropriate, were added to the dataset to allow for cross-sectional analysis. Several themes derived from the evaluation proposal and others emerged from the content analysis: community/parent involvement, EGR understanding, funding, gender, helping, hindering, learning achievement, the medium of instruction (MOI), NGO, school improvement activities, suggestions, surprise, time on task. 75 APPENDIX 3: DATA COLLECTION INSTRUMENTS Central Level Agencies: Interview Guidelines Government agency NB: Please check the EGRP Performance Evaluation Baseline (2017), Annex 4, IRs 1 & 2, For the intended End of Project Status (EOPs, Col 3) and adapt this form to specific agencies’ EOPs. Date: ________________________ Name of informant: _______________________________ Position: ________________________ Organization: ______________________________ Years in this position: Years of experience with the government: Position prior to this one: Experience at the school level? Thank you for taking the time to talk to us. I am ________________ [pass out business cards] and this is my colleague _____________. We are both experts in education, and I have taught in universities and served as a consultant to the Government of Nepal and international organizations, such as ________________. We are currently working on a performance evaluation for USAID’s support to the National Early Grades Reading Program. Our task today is to talk to you about your work and how it relates to EGRP. 1. Please describe for us the responsibilities of your department/agency with respect to EGRP. 2. What are your specific responsibilities with respect to EGRP? What sort of orientation or training did you receive to prepare you for those responsibilities? Was it sufficient? How would you improve that orientation or training? 3. Overall, do you think the EGRP activities in which your department are engaged are going very well, well, or not so well? 76 4. Thinking now about the specific, individual EGRP activities in which your department is engaged. Have you gotten any feedback on success or difficulties with these activities? Which ones do you think: a. Are going very well? b. Are satisfactory but not outstanding? c. Are going not so well? 5. Regarding the specific EGRP activities in which your department is engaged, what are some of the factors that have hindered the implementation of these activities as they were originally planned? How has your department addressed these hindering factors? What other resources or written directives/authorities does your department need to address these hindrances effectively? Wait for their spontaneous responses, then try these prompts: • Data and studies to inform decision making. • Capturing successes and lessons learned from early implementation. • More staff/changes in departments. • More funding: to which departments? • Policy changes. • Standards and benchmarks. 6. Is your department involved in producing new materials, training, policies, procedures, standards, or benchmarks in support of NEGRP? Which ones have you completed already? Did you get any help (consultants or special funding) to produce these? If so, from whom? What other policies, procedures, standards, and benchmarks do you expect to produce? Do you have any outside consultants for these? If so, from whom? 7. What kind of data does your department collect to carry out its NEGRP work? How do you collect that data? In the past year, have you been instructed by your superiors to collect or use more data? If so, have you found this additional data useful? Can you describe any instances where the data collected caused your department to change or adapt its activities? 77 8. Do you think your department is ready to carry out the functions of NEGRP successfully without technical assistance? Why or why not? If not, how many more years and what type of technical assistance will this department need before it can do so? 9. Thinking now about EGRP as the pilot for NEGRP: a. What components of EGRP do you think have been the most successful? Which government agencies will carry out these components under NEGRP? In what ways do you think these agencies are ready to implement those successfully in four new districts? In what areas might they still need technical assistance? b. What components of EGRP do you think have been less successful? Are those components essential to the success of NEGRP? If so, how do you think the government might revise these components? c. How long will it take the government to scale up successfully to all 77 districts? What, if any assistance will they need to do that? 78 10. How has federalization affected your department? With which decentralized government organizations is your department directly in contact? Examples: • Provincial education units. • DCCs. • EDCUs. Palikas. Do you think these decentralized units are ready to carry out NEGRP without further technical assistance? If not, how many more years and what type of technical assistance will this department need before it can do so? 11. During your time in government or education, what other initiatives has the government undertaken to improve the quality of learning at the primary level? What role, if any, did you play in those efforts? What can you tell us about the factors that helped those initiatives move ahead? What hindered the implementation of those initiatives? What were the results of those efforts in terms of learning? How do the results of EGRP compare to the results of those other initiatives? 12. We have asked many questions. Is there some important aspect of EGRP that we have missed, and you would like to tell us about? Do you have any questions you would like to ask us? 13. OBSERVATIONS: 79 14. EGRP hopes to achieve several goals involving many different departments. I’m going to list them one by one and ask you to tell me if these are very likely, probably likely, or unlikely to be achieved by the end of the project next year. a. CDC: Evidence-based early grade reading instructional materials designed, distributed, and in use. b. CEHRD, ETCs: In-service professional development for teachers in public schools on reading instruction and materials in use. c. DOE ERO: Classroom-based and district-based early grade reading assessment processes. d. HTs, monitoring, and coaching for teachers in early grade reading instruction. e. Policies, standards, and benchmarks that support improved early grade reading. f. Planning and management of financial, material, and human resources. g. Utilization of data for course correction during implementation. h. Community awareness of the importance of the language of reading instruction. i. Family engagement to support reading. j. PTA/SMC contributes to quality reading instruction. k. Parent & community capacity to monitor reading progress. Can you talk a bit about the reasons you think _______ are unlikely? 80 Central Level Agencies: Interview Guidelines INGO (Plan, RtR, SC, WE, WV) NB: Please check the EGRP Performance Evaluation Baseline (2017), Annex 4, IR 3 for the intended End of Project Status (EOPs, Col 3) and adapt this form to specific agencies’ EOPs. Date: ________________________ Name of informant: _______________________________ Position: ________________________ Organization: ______________________________ Years in this position: Years of experience with the government: Position prior to this one: Experience at the school level? 1. How and when did this organization first become involved in early grade reading activities? 2. Describe for me some of the EGR activities in which this organization has been involved before EGRP that have gone well. What was the scope of those activities (# of jurisdictions/coverage/# beneficiaries/years)? 3. Describe for me some of the EGR activities in which this organization has been involved before EGRP that have gone not so well. What was the scope of those activities (# of jurisdictions/coverage/# beneficiaries/years)? How have those experiences affected your current work? 81 4. What is the role of this organization in EGRP? 5. How has your organization managed coordination with EGRP in districts where you had pre-existing EGR activities? What do you think of the current Minimum Standards for EGR? How do you think they should be improved? In the future, in districts where you have ongoing independent EGR activities, what do you plan to do when NEGRP begins work in those districts? 6. Regarding your involvement in EGRP, what is going well? What is going less well? 7. Do you think EGRP has demonstrated a viable model for the GON to scale up nation-wide? What components look most promising? Which components are likely to be more of a challenge? What components are missing? 8. What is a reasonable time frame to establish effective and sustainable EGR in a palika? In a district? Should donors provide more support to EGRP districts and, if so, in what form and for how long? 9. How do you measure the success of your organization’s overall efforts in EGR? Can you describe some success stories? Can you give me some examples of lessons you have learned that perhaps EGRP or NEGRP has yet to learn? 82 10. How has federalization affected your work in EGR? How is it affecting EGRP and how might it affect the scaling up of NEGRP? 11. Concerning the areas where your organization is working intensely, what proportion of schools have such serious problems that no intervention at the level of intensity that government can provide is likely to make a significant improvement? 12. During your time in government or education, what other initiatives has the government undertaken to improve the quality of learning at the primary level? What role, if any, did you play in those efforts? What can you tell us about the factors helped those initiatives move ahead? What hindered the implementation of those initiatives? What were the results of those efforts in terms of learning? How do the results of EGRP compare to the results of those other initiatives? 83 13. What important aspects of your work have I not asked about? 14. Are there any questions you would like to ask me? 84 Central Level Agencies: Interview Guidelines Donor Partners (UNICEF, WB, WFP) Date: ________________________ Name of informant: _______________________________ Position: ________________________ Organization: ______________________________ Years in this position: Years of experience with the government: Position prior to this one: Experience at the school level? 1. Why is your organization interested in early grade reading? 2. Can you describe the involvement of your organization in EGR activities in Nepal? How satisfied are you with these activities? What involvement in EGR activities in Nepal is your organization planning in the future? 3. What can you tell me about EGRP? 4. Do you think it has demonstrated a viable model for the GON to scale up nation-wide? What EGRP components do you see as being most important? Are those components ones that the GON can implement successfully on a larger scale? If not, what further support/strengthening does the GON need to implement those components? 5. How effective do you think EGRP has been in improving the quality of learning in government-run primary schools? How do you measure success? Can you give me some examples? 85 6. Are you satisfied with the current ability of GON agencies to monitor the progress of NEGRP? How would you like to see GON monitoring strengthened? 7. What do you think of the GON’s current Minimum Standards for EGR? How do you think they should be updated? In districts where NGOs already have EGR programs that exceed the minimum standards, how should they coordinate with new NEGRP efforts? 8. What is a reasonable time frame to establish effective and sustainable EGR in a district? Should donors provide more support to the 16 districts where EGRP has already provided two years of intensive input? If so, in what form and for how long? 9. How do you measure the success of your organization’s overall efforts in EGR? Can you describe some success stories? Can you give me some examples of lessons you have learned that perhaps EGRP or NEGRP has yet to learn? 10. How is federalization affecting the scaling up of NEGRP? 11. What proportion of schools in Nepal have such serious problems that no intervention at the level of intensity that the government can provide is likely to make a significant improvement? 12. During your time working in education in Nepal, what other initiatives has the government undertaken to improve the quality of learning at the primary level? What role, if any, did you play in those efforts? What can you tell us about the factors helped those initiatives move ahead? What hindered the implementation of those initiatives? What were the results of those efforts in terms of learning? How do the results of EGRP compare to the results of those other initiatives? 13. What have I not asked about? 14. Are there any questions you would like to ask me? 86 Central Level Agencies: Interview Guidelines Government agency NB: Please check the EGRP Performance Evaluation Baseline (2017), Annex 4, IRs 1 & 2 for the intended End of Project Status (EOPs, Col 3) and adapt this form to specific agencies’ EOPs. Date: ________________________ Name of informant: _______________________________ Position: ________________________ Organization: ______________________________ Years in this position: Years of experience with the government: Position prior to this one: Experience at the school level? Thank you for taking the time to talk to us. I am ________________ [pass out business cards] and this is my colleague _____________. We are both experts in education, and I have taught in universities and served as a consultant to the Government of Nepal and international organizations, such as ________________. We are currently working on a performance evaluation for USAID’s support to the National Early Grades Reading Program. Our task today is to talk to you about your work and how it relates to EGRP. 14. Please describe for us the responsibilities of your department/agency with respect to EGRP. 15. What are your specific responsibilities with respect to EGRP? What sort of orientation or training did you receive to prepare you for those responsibilities? Was it enough? How would you improve that orientation or training? 16. Overall, do you think the EGRP activities in which your department is engaged are going very well, well or not so well? 87 17. Thinking now about the specific, individual EGRP activities in which your department is engaged. Have you gotten any feedback on success or difficulties with these activities? Which ones do you think: d. Are going very well? e. Are satisfactory but not outstanding? f. Are going not so well? 18. Regarding the specific EGRP activities in which your department is engaged, what are some of the factors that have hindered the implementation of these activities as they were originally planned? How has your department addressed these hindering factors? What other resources or written directives/authorities does your department need to address these hindrances effectively? Wait for their spontaneous responses, then try these prompts: • Data and studies to inform decision making • Capturing successes and lessons learned from early implementation • More staff/changes in departments • More funding: to which departments? • Policy changes • Standards and benchmarks 19. Is your department involved in producing new materials, training, policies, procedures, standards, or benchmarks in support of NEGRP? Which ones have you completed already? Did you get any help (consultants or special funding) to produce these? If so, from whom? What other policies, procedures, standards, and benchmarks do you expect to produce? Do you have any outside consultants for these? If so, from whom? 20. What kind of data does your department collect to carry out its NEGRP work? How do you collect that data? In the past year, have you been instructed by your superiors to collect or use more data? If so, have you found this additional data useful? Can you describe any instances where the data collected caused your department to change or adapt its activities? 88 21. Do you think your department is ready to carry out the functions of NEGRP successfully without technical assistance? Why or why not? If not, how many more years and what type of technical assistance will this department need before it can do so? 22. Thinking now about EGRP as the pilot for NEGRP: a. What components of EGRP do you think have been the most successful? Which government agencies will carry out these components under NEGRP? In what ways do you think these agencies are ready to implement those successfully in four new districts? In what areas might they still need technical assistance? b. What components of EGRP do you think have been less successful? Are those components essential to the success of NEGRP? If so, how do you think the government might revise these components? c. How long will it take the government to scale up successfully to all 77 districts? What, if any assistance will they need to do that? 89 23. How has federalization affected your department? With which decentralized government organizations is your department directly in contact? Examples: • Provincial education units • DCCs • EDCUs • Palikas Do you think these decentralized units are ready to carry out NEGRP without further technical assistance? If not, how many more years and what type of technical assistance will this department need before it can do so? 24. During your time in government or education, what other initiatives has the government undertaken to improve the quality of learning at the primary level? What role, if any, did you play in those efforts? What can you tell us about the factors helped those initiatives move ahead? What hindered the implementation of those initiatives? What were the results of those efforts in terms of learning? How do the results of EGRP compare to the results of those other initiatives? 25. We have asked many questions. Is there some important aspect of EGRP that we have missed, and you would like to tell us about? Do you have any questions you would like to ask us? OBSERVATIONS: 90 EGRP hopes to achieve several goals involving many different departments. I’m going to list them one by one and ask you to tell me if these are very likely, probably likely, or unlikely to be achieved by the end of the project next year. a. CDC: Evidence-based early grade reading instructional materials designed, distributed, in use b. CEHRD, ETCs: In-service professional development for teachers in public schools on reading instruction & materials in use c. DOE ERO: Classroom-based & district-based early grade reading assessment processes d. HTs, Monitoring & coaching for teachers in early grade reading instruction e. Policies, standards, and benchmarks that support improved early grade reading f. Planning and management of financial, material, and human resources g. Utilization of data for course correction during implementation h. Community awareness of the importance of language of reading instruction i. Family engagement to support reading j. PTA/SMC contributes to quality reading instruction k. Parent & community capacity to monitor reading progress Can you talk a bit about the reasons you think _______ are unlikely? 91 Central Level Agencies: Interview Guidelines INGO (Plan, RtR, SC, WE, WV) NB: Please check the EGRP Performance Evaluation Baseline (2017), Annex 4, IR 3 for the intended End of Project Status (EOPs, Col 3) and adapt this form to specific agencies’ EOPs. Date: ________________________ Name of informant: _______________________________ Position: ________________________ Organization: ______________________________ Years in this position: Years of experience with the government: Position prior to this one: Experience at the school level? 1. How and when did this organization first become involved in early grade reading activities? 2. Describe for me some of the EGR activities in which this organization has been involved before EGRP that have gone well. What was the scope of those activities (# of jurisdictions/coverage/# beneficiaries/years)? 3. Describe for me some of the EGR activities in which this organization has been involved before EGRP that have gone not so well. What was the scope of those activities (# of jurisdictions/coverage/# beneficiaries/years)? How have those experiences affected your current work? 92 4. What is the role of this organization in EGRP? 5. How has your organization managed coordination with EGRP in districts where you had pre￾existing EGR activities? What do you think of the current Minimum Standards for EGR? How do you think they should be improved? In the future, in districts where you have ongoing independent EGR activities, what do you plan to do when NEGRP begins work in those districts? 6. Regarding your involvement in EGRP, what is going well? What is going less well? 7. Do you think EGRP has demonstrated a viable model for the GON to scale up nation-wide? What components look most promising? Which components are likely to be more of a challenge? What components are missing? 8. What is a reasonable time frame to establish effective and sustainable EGR in a palika? In a district? Should donors provide more support to EGRP districts and, if so, in what form and for how long? 9. How do you measure the success of your organization’s overall efforts in EGR? Can you describe some success stories? Can you give me some examples of lessons you have learned that perhaps EGRP or NEGRP has yet to learn? 93 10. How has federalization affected your work in EGR? How is it affecting EGRP, and how might it affect the scaling up of NEGRP? 11. With respect to the areas where your organization is working intensely, what proportion of schools have such serious problems that no intervention at the level of intensity that government can provide is likely to make a significant improvement? 12. During your time in government or education, what other initiatives has the government undertaken to improve the quality of learning at the primary level? What role, if any, did you play in those efforts? What can you tell us about the factors helped those initiatives move ahead? What hindered the implementation of those initiatives? What were the results of those efforts in terms of learning? How do the results of EGRP compare to the results of those other initiatives? 94 13. What important aspects of your work have I not asked about? 14. Are there any questions you would like to ask me? 95 Central Level Agencies: Interview Guidelines Donor Partners (UNICEF, WB, WFP) Date: ________________________ Name of informant: _______________________________ Position: ________________________ Organization: ______________________________ Years in this position: Years of experience with the government: Position prior to this one: Experience at the school level? 1. Why is your organization interested in early grade reading? 2. Can you describe the involvement of your organization in EGR activities in Nepal? How satisfied are you with these activities? What involvement in EGR activities in Nepal is your organization planning in the future? 3. What can you tell me about EGRP? 4. Do you think it has demonstrated a viable model for the GON to scale up nationwide? What EGRP components do you see as being most important? Are those components ones that the GON can implement successfully on a larger scale? If not, what further support/strengthening does the GON need to implement those components? 5. How effective do you think EGRP has been in improving the quality of learning in government-run primary schools? How do you measure success? Can you give me some examples? 6. Are you satisfied with the current ability of GON agencies to monitor the progress of NEGRP? How would you like to see GON monitoring strengthened? 7. What do you think of the GON’s current Minimum Standards for EGR? How do you think they should be updated? In districts where NGOs already have EGR programs that exceed the minimum standards, how should they coordinate with new NEGRP efforts? 96 8. What is a reasonable timeframe to establish effective and sustainable EGR in a district? Should donors provide more support to the 16 districts where EGRP has already provided two years of intensive input? If so, in what form and for how long? 9. How do you measure the success of your organization’s overall efforts in EGR? Can you describe some success stories? Can you give me some examples of lessons you have learned that perhaps EGRP or NEGRP has yet to learn? 10. How is federalization affecting the scaling up of NEGRP? 11. What proportion of schools in Nepal have such serious problems that no intervention at the level of intensity that the government can provide is likely to make a significant improvement? 12. During your time working in education in Nepal, what other initiatives has the government undertaken to improve the quality of learning at the primary level? What role, if any, did you play in those efforts? What can you tell us about the factors helped those initiatives move ahead? What hindered the implementation of those initiatives? What were the results of those efforts in terms of learning? How do the results of EGRP compare to the results of those other initiatives? 13. What have I not asked you about? 14. Are there any questions you would like to ask me? 97 EGRP PERFORMANCE EVALUATION 2019 PROVINCE/DISTRICT/LEU PACKAGE For NON-EGRP Districts GENERAL INTRODUCTION: Thank you for taking the time to meet our team. Since 2015, USAID has been helping the Government of Nepal to test a model of early grades reading instruction in 16 districts, including two districts adjacent to this one. In 2016, we visited this district to observe conditions in the schools and the education agencies at the beginning of the project. We have returned this week to see what may have changed in education over the past two years. In other countries, we have seen good practices spread from schools participating in the pilot to other, non-participating schools nearby, without any outside help. We want to see if that happens here. We have permission from MOEST to visit schools and to talk with all government agencies involved with EGRP. 98 Early Grade Reading Program: Performance Evaluation 2019 Chief, Education Development Coordinating Unit (EDCU) Name: Date of Interview: Years of Experience with the Government: Years of Experience as a DEO/School Supervisor: Years of Experience this District: Issues: Motivation, policies, funding 1. What is the current status of basic education in your district? 2. In your opinion, what proportion of primary schools in your district qualify as “good” or excellent schools? 3. What activities has your office undertaken to improve the quality of the schools in the last 12 months? What are the most important improvements that you would like to make in the primary schools in your district? 4. With the move to federalization, are there improvements that you are hoping to make in schools in the future, or has the initiative passed entirely to the local level? Have all the Resource Persons been assigned to the LEUs, or at least some of them? 5. Two years ago, we visited several schools, and we would like to talk to the School Supervisors responsible for them now. Can your office provide us with the names and contact information for those supervisors? 99 6. What type of support are NGOs in your district providing to primary schools? What do you think about the NGOs’ activities? Are they helping to improve the quality of primary schools? If there are some NGOs/CSOs that have been particularly helpful in basic education in this district, we would like to interview them? Can you recommend any? Provide contact information? 7. We are also interested in talking to other organizations that may be playing a role in basic education in this district and would appreciate any you can recommend. These include: a. Journalists who write about basic education. b. Teacher associations. c. Guardian associations. 8. In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in? 9. What do you know about NEGRP? How do you know this? Do you think it might help the primary schools in your district? Thank you for your time. 100 Early Grade Reading Program: Performance Evaluation –2019 Interview Schedule for NEGRP Focal Point District: Name of the EGRP Focal Person: Date of Interview: Years of Experience with the Government: Months of Experience as a Focal Person: 1. What experience have you had with NEGRP to date? 2. What orientation/training prepared you to be the NEGRP Focal Point/Coordinator in this district? a. What did the training cover? b. Was it adequate for your participation in NEGRP? c. What else would you have liked to have learned? 3. (Viability) Is there much interest in NEGRP in this district? What activities are associated with NEGRP in your district? Are non-government organizations or groups implementing activities to support early grade reading in this district? Can you tell me a bit about those organizations and their activities? 4. (Viability) What, if anything, have you heard about the EGRP pilots being conducted in two adjacent districts? Where have you heard about them? What are people saying about them? Which activities seem to be doing the best? Which are harder to implement? Do you think EGRP would be a good model for this district? 5. (Viability) What, if any, is the relationship between mother-tongue instruction and NEGRP? 101 6. The goal of EGRP is to improve student learning. Can you suggest better, more cost-effective ways to improve student learning? 7. (Sustainability/Scalability) Is there anything more you would like to add about the implementation of NEGRP in your district or EGRP in the adjacent districts? a. When do you think NEGRP might launch in this district? Do you think it will be a very important activity when it comes to this district? b. How long do you think it will take to scale up NEGRP nation-wide? Why? Thank you for your time. 102 Early Grade Reading Program Performance Evaluation – Baseline Study Interview Schedule for School Supervisor 1________________ District: Name of the SS: Date of Interview: Years of Experience with the Government: Years of Experience as SS: Years of Experience as the SS in this District: 1. How many clusters and primary schools do you supervise? Do you have the resources you need to do this work well? What other resources would you like to have? 2. What is the current status of basic education in the schools you supervise? 3. In your opinion, what proportion of primary schools that you supervise qualify as “good” or excellent schools? 4. What activities has the district undertaken to improve the quality of the schools in the last 12 months? 103 5. What are the most important improvements that you would like to see in the schools you supervise? 6. How does federalization impact your work? Will it make your job harder or easier? Do you think LEUs will be able to make significant improvements in the schools you supervise? 7. In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in? 8. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools you supervise? How so? Thank you for your time. 104 Early Grade Reading Program Performance Evaluation – Baseline Study Interview Schedule for School Supervisor 2________________ District: Name of the SS: Date of Interview: Years of Experience with the Government: Years of Experience as SS: Years of Experience as the SS in this District: 1. How many clusters and primary schools do you supervise? Do you have the resources you need to do this work well? What other resources would you like to have? 2. What is the current status of basic education in the schools you supervise? 3. In your opinion, what proportion of primary schools that you supervise qualify as “good” or excellent schools? 105 4. What activities has the district undertaken to improve the quality of the schools in the last 12 months? 5. What are the most important improvements that you would like to see in the schools you supervise? 6. How does federalization impact your work? Will it make your job harder or easier? Do you think LEUs will be able to make significant improvements in the schools you supervise? 7. In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in? 8. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools you supervise? How so? Thank you for your time. 106 Early Grade Reading Program: Performance Evaluation 2019 Mayor or Chair of Rural Palika 1___________ Name: Date of Interview: Occupation: Years of Experience as Mayor/Chairman of the Rural Palika: Experience with Education: Happy to be back in your town. I was here in 2016 before you were elected. Congratulations. How are you enjoying your job? What parts of the job do you like the best? What parts are the hardest? Thank you for taking the time to talk to us today. 1. Please give a brief description of the status of primary education in your palika. Are your schools doing better or worse than schools in neighboring villages? How can you tell? What aspects of schooling in your village are strong? What aspects of schooling need to be improved? 2. Has there been much change in the management of basic education in the last two years in your palika? Has a Local Education Unit replaced the Village Education Committee? How are the LEUs working? How is the LEU working out? Is it facing any difficulties? 3. How much money did the palika receive this year from the Min of Finance to support basic education? Is it enough? If not, do you expect to get more in the future? 4. How are you monitoring the work of the LEUs? Are they working well or not so well? Please explain. 107 5. What sort of technical help are the LEUs receiving from Resource Persons (previously assigned to education Resource Centers)? From School Supervisors? 6. What other support does the LEU need to do its job effectively? 7. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools in this palika? How so? 8. Is there anything else you would like us to know about managing basic education at the local level? Thank you. 108 Early Grade Reading Program: Performance Evaluation 2019 Mayor or Chair of Rural Palika 2___________ Name: Date of Interview: Occupation: Years of Experience as Mayor/Chairman of the Rural Palika: Experience with Education: Happy to be back in your town. I was here in 2016 before you were elected. Congratulations. How are you enjoying your job? What parts of the job do you like the best? What parts are the hardest? Thank you for taking the time to talk to us today. 1. Please give a brief description of the status of primary education in your palika. Are your schools doing better or worse than schools in neighboring villages? How can you tell? What aspects of schooling in your village are strong? What aspects of schooling need to be improved? 2. Has there been much change in the management of basic education in the last two years in your palika? Has a Local Education Unit replaced the Village Education Committee? How are the LEUs working? How is the LEU working out? Is it facing any difficulties? 3. How much money did the palika receive this year from the Min of Finance to support basic education? Is it enough? If not, do you expect to get more in the future? 109 4. How are you monitoring the work of the LEUs? Are they working well or not so well? Please explain. 5. What sort of technical help are the LEUs receiving from Resource Persons (previously assigned to education Resource Centers)? From school supervisors? 6. What other support does the LEU need to do its job effectively? 7. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools in this palika? How so? 8. Is there anything else you would like us to know about managing basic education at the local level? Thank you. 110 Early Grade Reading Program: Performance Evaluation 2019 Head of LEU for Palika 1___________ Name: Date of Interview: Occupation: Years of Experience as Head of LEU: Prior Experience with Education: Thank you for taking the time to talk to us today. Issue: Education at the palika level 1. How is the new federalization going at your level? Has your job changed very much? What parts have changed the most? 2. What activities is this LEU responsible for? 3. Has the palika received funds this year from the Min of Finance to support basic education? Is it enough? If not, do you expect to get more in the future? 111 4. Please give a brief description of the status of primary education in your palika. Are your schools doing better or worse than schools in neighboring villages? How can you tell? What aspects of schooling in your village are strong? What aspects of schooling need to be improved? 5. What is going well in this palika in terms of basic education? What is more difficult? 6. What staff do you have in this LEU? Any education Resource Persons (previously assigned to education Resource Centers)? Any School Supervisors? Do you have enough staff? If not, what other staff do you need to do your job? 7. What other support does the LEU need to do its job effectively? 112 8. In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in? 9. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools in this palika? How so? Thank you for your time. 113 Early Grade Reading Program: Performance Evaluation 2019 Head of LEU for Palika 2___________ Name: Date of Interview: Occupation: Years of Experience as Head of LEU: Prior Experience with Education: Thank you for taking the time to talk to us today. Issue: Education at the palika level 1. How is the new federalization going at your level? Has your job changed very much? What parts have changed the most? 2. What activities is this LEU responsible for? 3. Has the palika received funds this year from the Min of Finance to support basic education? Is it enough? If not, do you expect to get more in the future? 114 4. Please give a brief description of the status of the primary schools that you support. Are your schools doing better or worse than other schools in the district? How can you tell? What aspects of schooling in your palika are strong? What aspects of schooling need to be improved? 5. What is going well in this palika in terms of basic education? What is more difficult? 6. What staff do you have in this LEU? Any education Resource Persons (previously assigned to education Resource Centers)? Any School Supervisors? Do you have enough staff? If not, what other staff do you need to do your job? 115 7. What other support does the LEU need to do its job effectively? 8. In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in? 9. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools in this palika? How so? Thank you for your time. 116 Early Grade Reading Program: Performance Evaluation 2019 Level of Analysis: Education Training Center Informants: Regional education director, ETC director, ETC staff who have been trained in EGR District/Region: Date of Interview: Name of the Informant: Position: Years in this position: Prior experience with education: 1. What proportion of your work does EGRP represent? 2. What orientation or training relevant to EGRP have you had? What did it cover? Was it sufficient? What other orientation or training do you expect to attend? 3. In what EGRP activities has this ETC been engaged? Have you received adequate funding and personnel to carry out these activities? If not, what additional funding and personnel do you need to carry out these activities? 117 4. In your opinion, which of these activities have been most successful? Why? Which activities have been more difficult to implement? Why? Which aspects of EGRP do teachers struggle with the most? 5. What percentage of primary school teachers do not speak the MT of their students? How do you prepare teachers to teach in multilingual classrooms? What resources do you have to address this issue? 6. Have you received all the standards, policies, benchmarks that you need to support EGRP effectively? a. What enables/hinders the development/implementation of such standards/policies/benchmarks today? b. Do you think EDCU should be more involved in planning EGRP? c. What would you change about EGRP if it were up to you? 118 7. Is this ETC working with any NGOs, CSOs, or private groups on issues relating to EGR? If so, please describe those. 8. Do you think that EGRP can really make a difference in student achievement or learning? Have you seen any evidence yet about the impact of EGRP on student learning? If not, when do you expect you will be able to see some impact from EGRP? How will that impact be measured? 9. In your opinion, what components of EGRP are likely to have the biggest influence on improving student learning in the short run? In the long run? 119 10. Do some groups of students benefit more from EGR than others? If so, which ones benefit more, and which ones benefit less? 11. Can the EGR activities at this ETC be sustained when EGR project finishes? a. What additional support will be needed to sustain EGR? 12. Is there anything more you would like to add with regard to teacher training aspects of EGRP a. Is EGRP a good model for Nepal? b. Would the EGRP in Dhankuta and Saptari work in Sunsari? How it might need to be modified. c. Can you suggest other, more cost-effective ways to improve student learning? Thank you for your time. 120 Early Grade Reading Program: Performance Evaluation 2019 Interview Schedule for Resource Person (RP)1______________ District: Name of the RP: Date of Interview: Name of the Resource Center: Number of Schools in the Cluster: Years of Experience in Education Sector: Years of Experience as an RP: Years of Experience as an RP in this RC: 1. How were you prepared for the job (training, orientation, etc.)? How many schools do you support? 2. What do you like best about your job? What is the hardest? 3. How has federalism affected your job? Has it made it harder or easier? 121 4. Please give a brief description of the status of the primary schools that you support. Are your schools doing better or worse than other schools in the district? How can you tell? What aspects of schooling in your palika are strong? What aspects of schooling need to be improved? 5. In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in? 6. What type of support are NGOs in your cluster providing to primary schools? What do you think about the support those NGOs provide? 7. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools in this palika? How so? Thank you for your time. 122 Early Grade Reading Program: Performance Evaluation 2019 Interview Schedule for Resource Person (RP) 2______________ District: Name of the RP: Date of Interview: Name of the Resource Center: Number of Schools in the Cluster: Years of Experience in Education Sector: Years of Experience as an RP: Years of Experience as an RP in this RC: 1. How were you prepared for the job (training, orientation, etc.)? How many schools do you support? 2. What do you like best about your job? What is the hardest? 3. How has federalism affected your job? Has it made it harder or easier? 123 4. Please give a brief description of the status of the primary schools that you support. Are your schools doing better or worse than other schools in the district? How can you tell? What aspects of schooling in your palika are strong? What aspects of schooling need to be improved? 5. In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in? 6. What type of support are NGOs in your cluster providing to primary schools? What do you think about the support those NGOs provide? 124 7. Have you heard about NEGRP? If so, how did you hear about it? Does it sound like something that would be useful in the schools in this palika? How so? Thank you for your time. 125 Early Grade Reading Program: Performance Evaluation 2019 Interview Schedule for NGOs that support basic education 1_________ Respondent’s Organization: Address of the Organization: Name of the Respondent: Year Organization/Local Branch Established: Focus of organization: 1. What is the status of basic education in this district/palika? 2. Tell me about the types of support you provide to basic education in this district/palika. 3. How many schools do you serve? How many more do you expect to serve in the future, or do you plan to keep your program the same size? 4. What is the main goal of your organization? 126 5. Have you heard about EGRP in the two neighboring districts? How did you hear about it? What did you hear? Can you tell me the goal of EGRP? Does it sound like it might be useful for primary schools in this palika/district? Why or why not? 127 Early Grade Reading Program: Performance Evaluation 2019 Interview Schedule for NGOs that support basic education 1_________ Respondent’s Organization: Address of the Organization: Name of the Respondent: Year Organization/Local Branch Established: Focus of organization: 1. What is the status of basic education in this district/palika? 2. Tell me about the types of support you provide to basic education in this district/palika. 3. How many schools do you serve? How many more do you expect to serve in the future, or do you plan to keep your program the same size? 128 4. What is the main goal of your organization? 5. Have you heard about EGRP in the two neighboring districts? How did you hear about it? What did you hear? Can you tell me the goal of EGRP? Does it sound like it might be useful for primary schools in this palika/district? Why or why not? 129 Early Grade Reading Program: Performance Evaluation 2019 Interview Schedule for Teacher Association Respondent’s Organization: Address of the Association: Name of the Respondent: Years of association with the Organization: Position in the Association: Years of Teaching Experience: 1. As an experienced teacher and administrator, what factors do you think would bring about the biggest improvements in education in this district? 2. What has the government done to improve education in this district in the last two years? 3. What has been the role of your association with this regard? 130 4. Have you heard of EGR? What can you tell me about it? 5. Do you think that EGRP has demonstrated a good model that can scale up in other districts in Nepal? 6. Have you heard of NEGRP? What can you tell me about it? 7. What is your view about these programs? 131 8. What are the chances for the successful implementation of this program? 9. What role might the Teachers’ Association play in these programs? 10. What might raise the chances of success for these programs? 11. What will hinder their success? 12. EGRP aims to improve student learning in Grades1-3 so students will be prepared to achieve more learning in Grades 4 and up. Can you think of any other program(s) that might have a bigger effect on student learning at a lower cost? 132 Early Grade Reading Program: Performance Evaluation 2019 Interview for Guardian Association Respondent’s Organization: Address of the Association: Name of the Respondent: Years of association with the Organization: Position in the Association: Ages of children now in school in the district: 1. What factors does your association think would bring about the biggest improvements in education in this district? 2. What has the government done to improve education in this district in the last two years? 3. What has been the role of your association in these improvement efforts? 133 4. Have you heard of EGR? What can you tell me about it? 5. Have you heard of NEGRP? What can you tell me about it? 6. What are your views about these programs? 7. What are the chances for the successful implementation of this program? 134 8. What role might the Guardians’ Association play in these programs? 9. What might raise the chances of success for these programs? 10. What will hinder their success? 11. EGRP aims to improve student learning in Grades1-3 so students will be prepared to achieve more learning in Grades 4 and up. Can you think of any other program(s) that might have a bigger effect on student learning at a lower cost? 135 Early Grade Reading Program: Performance Evaluation 2019 Interview Protocol for Journalist Respondent’s Organization: Address of the Association: Name of the Respondent: Years of association with the Organization: Position in the Association: Experience with education 1. What has the government done to improve education in this district in the last two years? 2. What has been the role of your news organization in reporting on it? 3. Have you heard of EGR? What can you tell me about it? Why the emphasis on early grades? Why the emphasis on reading? 136 4. Have you heard of NEGRP? What can you tell me about it? 5. What are your views about these programs? 6. What do you think are the chances that this program will increase student learning and become integrated into the schools permanently? 7. What do you know about the link between Nepal’s language issues and EGR? 137 8. What role do you think social mobilization and communication might play in these programs? Have you heard radio announcements and programs? Did they sound effective to you? What other communication—newspapers, posters, meetings—have you seen/heard? 9. What are other people saying about EGRP? Is there an interest in EGR in nearby districts that have not participated in EGRP? 10. EGRP aims to improve student learning in Grades1-3 so students will be prepared to achieve more learning in Grades 4 and up. Can you think of any other program(s) that might have a bigger effect on student learning at a lower cost? 11. Is there anything else you would like to tell us about your perspective on EGR? 138 Early Grade Reading Program: Performance Evaluation 2019 Chairman of Ward for School 1___________ Name: Date of Interview: Occupation: Years of Experience as Ward Chairman: Experience with Education: Issue: Changes in education at the ward level 1. Tell me about the status of schools in your community. In what ways have they gotten better in the last few years? In what ways have they gotten worse? 2. If there have been any special efforts to improve the quality of schools in the last two years, could you describe them for me? 3. Have you heard anything about the government’s National Early Grade Reading Project? If yes, what have you heard? 4. A pilot for NEGRP—the Early Grade Reading Project-- is running in 20 districts, including two districts adjacent to this one. Have you heard anything about this EGRP pilot? If yes, what have you heard? 5. Of all the things the government could do to improve education, why do you think it wants to focus on helping children to learn to read in Grades 1-3? 6. Are people in your community worried about the quality of schools? If yes, what do they think might help to improve the quality of schools? 139 7. Are some groups of children benefitting more from public schools than others? Who are they? Which children are benefitting less? Do you think something should be done to make sure more children are benefitting from public schooling? What do you think that should be? 8. Is there anything else you’d like to say about the quality of schools in your community? 9. Are you familiar with the subject of mother-tongue-based learning? Is this something that people in your community might be interested in? Thank you for your time. 140 APPENDIX 4: MTR BRIEF March 24, 2019 BRIEFING NOTE TO: USAID and Donor Partners FROM: EGRP 2019 Performance Evaluation Team SUBJECT: Preliminary Findings Relevant to SSDP Mid-Term Review The EGRP 2019 performance evaluation was designed to address several key questions: • To what extent did EGRP demonstrate a viable and sustainable model for increasing EGR skills that could be scaled to other districts? • To what extent did EGRP strengthen GON EGR service delivery at the central and local levels? • To what extent have MOEST and CLAs built up the capacity at the central level to support the national scale-up of the NEGRP minimum packages at the federal and local levels? The team33 conducted interviews in four EGRP and two control districts during February 16-March 8 with primary education stakeholders at the community (government) school (163)34, palika (30), district (43)35 , and regional/provincial (13) levels. Since March 12, senior team members have been conducting interviews in the GON Central Line Agencies (CLA), INGOs active in early grade reading (EGR) and donor partners, while other team members are cleaning and uploading more than 250 interviews for analysis. The following observations from the field are therefore preliminary and impressionistic, put together as an input with the intention of supporting the SSDP MTR process by the end of March. The status of EGRP: good news EGRP is now in its third year of implementation in 16 districts and all stakeholders—from parents to provincial education office staff--agree that EGRP is a good program and no better alternative for improving learning at the primary school level currently exists. • The Teaching and Learning Materials (TLM) and Supplementary Reading Materials are much appreciated and, where they are being used, are having a significant impact on student learning, as shown in the EGRA mid-line conducted by EGRP. • Parent engagement, student attendance, and teacher motivation have all increased in schools where EGRP TLMs are being used by trained teachers. • Reading competitions are demonstrating the value of EGR to the broader community. We heard anecdotes about students from community schools with two or more years of EGR (both EGRP and other NGO-led EGR projects) beating boarding school students in Nepali 33One senior foreign expert in early grades reading and organizational analysis assisted by one senior Nepali expert in Nepali education reform, both of whom led the 2016 EGRP baseline performance evaluation. They were assisted by two mid-level Nepali experts in qualitative research and schooling and four field surveyors. 34 Schools were selected by DEOs and RPs in 2016 based on these criteria: one excellent, one with a large enrollment of disadvantaged students, and two others at their discretion. 35 Banke, Dang, Dhankuta, Salyan (control), Saptari, Sunsari (control). Sunsari and Salyan are hilly; the other districts are mainly Terai. No mountain districts were included. 141 reading competitions, resulting in some children transferring back to community schools or in participating boarding schools adopting 90 minutes per day of Nepali. All this is remarkable given the impact of a real earthquake and the administrative earthquake that is federalization (see below). The status of the school system: weak Nonetheless, the large increase in oral reading fluency demanded by the SSDP targets between 2017/2018 (27.2 wpm) and 2018/19 (42 wpm) is unlikely to be met. According to our informants, 30- 70% of schools in Nepal have one or more problems that prevent them from retaining the benefits of any targeted learning intervention, such as EGRP. The image of the school as a bucket with holes in it comes to mind, the holes including: • Poor distribution of primary level teachers: Many primary schools are embedded in over￾crowded secondary schools that reassign primary teachers to work in upper grades. Permanent teachers work hard to remain in urban areas. Where more teachers are needed, palikas are unsure of how to hire them. • Too little time-on-task: Local holidays, tardy teachers, teachers engaged in political work, children who do not return to school after lunch: all result in far less than 192 school days and 90-minutes daily of Nepali instruction. • Little classroom level supervision/mentoring: The GON has discontinued reading motivators and resource persons (RPs), both of whom were originally expected to play a major role in EGRP mentoring at the classroom level, helping teachers to apply their training and use the TLM to their best effect. Many headteachers, recently trained to do classroom-level mentoring for EGRP, do not have the time and perhaps not enough EGR subject knowledge, to carry out effective EGR supervision. • Language: Even in our limited sample, we found schools with teachers and students who spoke no Nepali and cannot begin to use TLMs effectively until they are available in their mother tongue or feature Nepali as a second language. • Great variability in school, community, and local government capacities: Some schools are in desperate straits and will need more assistance than others. One size fit all does not work for schools or for local governments, some of which lack any expertise in education. • Dropping enrollments: Parents are transferring their children from community schools into “English medium” boarding (private) schools where teachers’ attendance is regular, classrooms are freshly painted, children wear neat uniforms, transportation to school is available and social status is higher. This leaves community schools with lower capitation funds and teacher allocations as well as a larger proportion of the disadvantaged students, who tend to have less school readiness. It also leaves many children in boarding schools learning little, if any, Nepali and poor English. Preliminary feedback from the central level suggests that resources and overall service delivery for EGRP are still weak. Most of our informants agreed that the model needs two to three years more assistance to become sustainable. The model is not scaling up spontaneously anywhere. All these issues have implications for the revision of the NEGRP Minimum Package and for the configuration of Local Education Units (LEUs) at the palika level, both currently under discussion as part of the SSDP review and the revision of the National Education Act. 142 Implications for NEGRP Minimum Package 1. TLM: EGR TLMs in Nepali are effective but could be more so. Teachers need a GON circular, stating that the EGR workbook is the primary text and the old Nepali textbook should be used, if at all, as a supplement. The official curriculum timetable should be changed to allow for 90 minutes of Nepali daily in Classes 1-3. The EGRP workbook should be redesigned to cover 150-160 school days only. Given the high percentage of non-Nepali speaking students, EGR materials developed in MTs by NGOs should be more widely used. Also, TLMs for Nepali as L2 should be made widely available as soon as possible. 2. Teacher training and classroom support. Teacher training should include more emphasis on time on task, both in and out of the classroom and on teaching Nepali as L2. Given the number of topics to be covered, the initial training period for teachers should be extended to more than 5 days, to be covered in multiple sessions. 3. Community mobilization. Ward chairs, palika mayors, and LEUs could play a larger role but will probably need orientation on multiple topics, in multiple sessions. 4. Student assessment. Our informants were not using CB-EGRA to monitor student learning; rather, they relied on the end of year exams based on the old Nepali textbook. Teachers need to be encouraged to do more individual, informal oral reading fluency tests to monitor student progress. 5. Monitoring and Evaluation. Monitoring and classroom support should be provided by both LEUs at the palika level (see below) and, for those not already burdened with heavy teaching loads, by HTs. Since palika mayors, ward chairs and LEUs are to provide overall support and school observation, the Minimum Package should include a curriculum for palika officials, delivered in multiple sessions over several months. 6. The issue of English medium is not addressed either in the SSDP or the Minimum Package. Yet widespread demand for English beginning in the early grades is a major factor driving enrollments in boarding schools, as well as reducing time for studying Nepali and mother tongue in the official and unofficial timetable. Social mobilization should be built around the message “If you want your child to speak English, an excellent foundation in reading Nepali is the best first step.” The Impact of Federalization 1. Federalization has had the effect of an administrative earthquake and has left the mentoring, monitoring, and evaluation systems for primary education in a shamble.36 2. Much of EGRP’s institutional memory is gone. DEOs & RPs have been eliminated, interim LEU staff have been reassigned and new LEU staff are appointed but not yet in place. Much EGRP orientation and training—in CB-EGRA, in the EGRP instructional model, in integrating EGR support into palika budgets—will need to be redesigned and redelivered for new staff, but some LEU positions may be vacant de facto for some time. 3. LEU staff are being assigned uniformly, independent of the number of schools in palikas 4. Orienting/training mayors and ward chairs is a new challenge. To date, some mayors have chosen not to attend EGRP orientation and many have not stayed for the duration. New ways to motivate their participation and engagement must be developed. 5. Many informants want the RPs reinstated or their teacher mentoring functions replaced at the palika level. Four of our six RP informants were among the best prepared, most knowledgeable and appreciated as mentors/monitors at the classroom level. Some mayors say they 36 Seven months after the release of Claussen, J., Nepal, J., Sharma, N., & Upadhyay, J. (2018). Federal system and changes in risk assessment for the School Sector Development Plan (SSDP) Kathmandu, Nepal: ???? 15 August. 50, we find conditions little changed at the palika level. 143 will try to retain the RPs that have been working with them to date, but whether those mayors will find and disburse the money for the RPs remains to be seen. 6. Teachers are facing uncertainty: Mayors are fighting with MOEST over who hires and fires teachers. This is not likely to be resolved until after the National Education Act is finalized. 7. Ward chairs were among our least informed informants, so a proposal to make them SMC chairs needs re-thinking. 8. Delay in the finalization of the National Education Act prevents many of these issues from being resolved. 144 APPENDIX 5: DISCLOSURE OF ANY CONFLICTS OF INTEREST 145 Appendix 5: Sample of Excerpts, by Evaluation Question, Theme, and Stakeholder Level EXCERPTS RELEVANT TO QUESTIONS 1-2 A. VIABLE: Working well (schools in Cohorts 1-2 CLA) School-level [A.1.S] ESTIMATES OF WHERE EGRP IS WORKING WELL a. 100% of EGRP stakeholders said EGRP is a good program. Students are happy, engaged, speaking better Nepali, better behaved/disciplined and their attendance improves. School summary data, Cohorts 1 and 2. b. 60% of schools in Cohort 1 district are implementing EGRP well. EGRP field staff [A.2.S] We are giving 90 minutes for the Nepali subject in Class 1-3. Teachers and students say they have found it easier to teach and learn using EGR method. This has only been made possible because of the availability of teacher's training, teacher's guide, workbook, storybook, pinwheel chart, words card, window cards and so on. After the implementation of EGRP teachers have realized a better way of teaching and dealing with children. HT (HT), rural school, Cohort 1 [A.3.S] EGR project develops the student’s capacity in recognizing the Nepali letters, and also develops the habit of reading books even in leisure time (lunchtime, free time at home etc.). Students are more attracted towards books with pictures. Students take the storybook to their homes which is helping to develop reading habit among children even at home. HT (HT), rural school, Cohort 1 [A.4.S] This is just the start of the program, but its efforts are effective. Students are studying in a joyful environment; they are progressing academically, and their learning skills are also improving. PTA, town school, Cohort 2 [A.5.S] How much time do you devote for Language teaching every day? 90 minutes. Is it enough? How much time do you really need? 90 minutes is enough. Teacher Focus Group Discussion (TFGD), town school, Cohort 1 [A.6.S] We have been teaching 90 minutes every day. it has been easy to follow teacher’s manual. I teach in Class 1; I do not know how time passes by [so quickly]. TFGD, rural school, Cohort 1 [A.7.S] The 10 days training was more effective than 10 months teacher training in terms of learning. TFGD, town school, Cohort 2 [A.8.S] Some teachers say, “This is the first training in my 25-year career I have received that I am happy about.” GON, Cohort 2 district 146 [A.9.S] Because of TLMs students are enjoying the classes and their attendance has improved. They read storybooks and poems in a group and enjoy them among themselves. Sometimes when teacher is absent or busy in some other work, they sit and read by themselves. This is the development of a very good reading habit among students. TFGD, Cohort 2 [A.10.S] The colorful books have attracted children, and they even study in leisure time. Even the students from upper grade are attracted towards the books and ask with teachers if they can borrow them. HT, Rural school, Cohort 2 [A.11.S] EGR has been helping weak students immensely. We do not punish children, instead we try to appreciate and encourage them which helps them learn better. Children have not only learnt to read but also learnt to behave well. TFGD, town school, Cohort 2 [A.12.S] Students are becoming active; they are making their own learning materials. They are making sentences and reading systematically, understanding the words and able to practice writing in the mud, water and air. They work in groups better than before. They have increased their reading and writing capacity. Since EGRP has started, children who speak only Awadhi at home have started speaking Nepali in class. TFGD, city school, Cohort 1 [A.13.S] If we are to compare private and community schools in terms of Nepali comprehension, the EGRP schools beat private schools. TFGD, town school, Cohort 2 The most useful activity was parental education. HT, Rural school, Cohort 1 Have you noticed any change in the attitude of your children towards school in the last 2 years? [A.14.S] Children these days love to attend the school on regular basis in comparison to earlier. Guardian, city school, Cohort 1 [A.15.S] If we see the annual result, 75% of students have become successful (it is because of EGR program) in comparison to previous result (before the start of project) PTA, rural school, Cohort 2 [A.16.S] In last one year, I went five times in this school. My wife also went three times after EGRP started. We inquire about how our children doing in class in their reading and learning and request teacher to take care of our children in school. Guardian, Town school, Cohort 2 [A.17.S] As per the guidance we received from EGRP staff, we formed an EGR sub-committee in our school which was eligible to receive NRP 140,000. With the help of the budget, we conducted peer education training for parents which also helped parents’ education program. We also conducted Tole Reading Groups and parent-teacher gathering where we discussed learning achievements. We are planning to conduct community reading-a-thon/competition. The most useful program has been peer education program [for parents] and other activities are also useful to some extent. HT, town school, Cohort 2 [A.18.S] [The SMC grant planning and implementation process] represents a great opportunity to mobilize the community. Try to incorporate parents and community into proposal development and in implementation. Helps build a close relationship between the school and the parents. This is a good activity with the right approach. Partner NGO, Cohort 2 district Palika and district level: [A.19.PD] 147 In [palika] this year have NRP 0.2m for reading festival from palika. [EGRP NGO partner] focused on working with SMC grant, nice peer education. All schools with an SMC grant formed reading committees. Grant comes in instalments. Some schools may have already received one instalment. RP, town, Cohort 2 [A.20.PD] Best thing EGRP has done: the message has been absorbed by all basic level teachers and students: to learn, you must first learn to read. New teaching techniques emphasizes lesson plan, how to use teaching materials, stimulated professional enthusiasm. RP, town, Cohort 2 [A.21.PD] Two months ago, EGRP began airing radio messages several times a day in Nepali. Very effective. Very touching, moving. They are translating the messages into Tharu language now. GON, Cohort 2 district [A.22.PD] Radio Laliguras and Radio Makalu are promoting EGR. Social mobilization and communication are good if it results in stakeholders taking ownership of the program. Journalist, Cohort 2 Provincial and central level [A.23.PC] In his entire career he has never seen a program that has had this kind of impact on children’s learning in the lower grade. He says that in many EGRP schools, children did not want to come out to play rather they preferred to read books. Parents say their children are happier in school. Teachers trained and devoted; earlier work with CFS helped with this. Children are reading with comprehension. GON, Province level [A.24.PC] CBEGRA scores indicate that EGRP interventions have positively affected students’ reading with comprehension abilities. During my interviews with the teachers in District X, I had asked their opinion about EGRP. They were all positive. Provincial Education Unit staff [A.25.PC] “[EGRP] has done great things!” Previously he visited a Class 8 where, out of 24 students, only five could write their schools’ name. “Now children in lower grade can read. Children speak more.” EGRP is one of the best programs he has seen and he’s very happy it was brought to the schools. “With local government now responsible for basic education, EGR gives them a tool to make schools more accountable. At a reading festival, students from community [public] schools could say a telephone number in English and Nepali but those from boarding school could say in English only.” Provincial Education Unit staff [A.26.PC] EGR work has improved and the Technical Working Group at the Central level, including NGOs, is really working. NGO, Central level [A.27.PC] During my unofficial visits to the [EGRP] schools, the parents and Local Government representatives tell me that children were reading beyond their grade level, that Class 2 students are reading better than Class 9 students. [Children are reading more, even] I have seen children reading the contents of biscuits and noodle covers in EGRP schools. GON, CLA [A.28.PC] We were recently consulted by the EGRP team whether we can have some sort of EGRP specific teacher competency framework while elaborating the existing framework for primary teacher licensing. We think that level of specialization is probably not necessary, but we can further incorporate EGR components in the framework. TSC, central level 148 [A.29.PC] EGRP has provided great support to TWG. Joint monitoring Terms of Reference is useful given lots of turnover among GON Focal Points. There’s a lot of actors, such as AIN. USAID has helped to provide continuity. TWG has had two workshops including one on the Minimum Package. Education funder, central level [A.30.PC] We adapted what we learned in Cohort I to design different Cohort II community mobilization interventions. Cohort I: Two TRGs/school Cohort II: EGR subcommittee under PTA EGRP central level 149 B. VIABLE: Not working well37 (schools in Cohorts 1-2 CLA) [B.1.S] ESTIMATES OF WHERE EGRP IS NOT WORKING WELL a. 35-40% of teachers fully implement EGRP TLM well when no one is watching. RP, Cohort 1 district b. Based on informal supervision, I estimate only about 30% of schools are using [EGRP] materials well. GON, Cohort 2 district. c. Can’t say everything going well; it all needs improving. Some excellent schools. Of the 22 schools he has visited, none of them are focusing on EGRP. LEU, rural school, Cohort 2 d. 50% of the schools can implement the project now. If the project continues longer for five years, they will see real change. Just can’t change education overnight. EGRP field staff, Cohort 2 e. Only 50% of Cohort 1 and 2 are doing well. EGRP central level School-level [B.2.S] Workload has tremendously increased. 90 minutes class is too much. Also, we have to use two Nepalis books in parallel. Since we have to give too much time for EGRP Nepali book, there’s not enough time for the other Nepali book. TFGD, rural school, Cohort 1 [B.3.S] I have been also confused about which book to use in the classroom. During the training, we understood that both books are in synergy and they should use both books in the classroom but it’s been not clear to us as to what to do. TFGD, rural school, Cohort 1 [B.4.S] Insufficient time because all students do not have same type of learning capacity. Time should be increased up to 135 minutes. TFGD, city school, Cohort 1 [B.5.S] After 90 mins of language class, we do not get enough time for other subjects. TFGD, town school, Cohort 2 [B.6.S] Now the 5 days training for HT has started and he [the HT] has learnt that time for Nepali language class is 90 minutes. Class 3 Teacher, rural school, Cohort 2 [B.7.S] DEO office was planning to send the textbooks, but we have not received them yet. HT, rural school, Cohort 1 [B.8.S] 37 Although the number of “working well” far exceeds the number of “not working well”, the latter raises more separate topics. “Happy families are all alike; every unhappy family is unhappy in its own way.” Leo Tolstoy, Anna Karenina. 150 Not everyone has received books. There are 35 students in Class 3, and they have only 10 books. We were having summer vacation when EGR materials were sent to school. There were only 10 books for Class 3, and we were asked to send the letter to the concerned mentioning the extra numbers we needed. We did send the letter and did all the formalities that was asked for, but we haven’t received the remaining books yet. Class 1 had enough part 1 book but part 2 wasn’t available. There may be too few workbooks for Class 2, too, but since the teacher for Class 2 is absent (pregnant) so cannot confirm it. Class 3 lacks 25 books, Class 2 lacks 7 books and Class 1 has 9 part 1 books but no part 2 books. TFGD, rural school, Cohort 1 [B.9.S] The EGR-trained Nepali teacher quit after her wedding so there have been no classes at all. All the TLMs and other EGR materials are still in a box packed in the office. TFGD, rural school, Cohort 1 [B.10.S] Cartons of TLM materials are not opened because HT is newly appointed (just one month) and he doesn’t know about EGR. The EGR trained teacher, now retired, never shared anything about EGR or implemented it in classes. Class 3 teacher, rural school, Cohort 2 [B.11.S] We have started teaching in English medium for nursery/ECD children from this year as per parent’s demand. TFGD, rural school, Cohort 1. [B.12.S] We need supervision and monitoring in terms of how the school is doing from the beginning of the academic year. TFGD, rural school, Cohort 2 [B.13.S] Class 1-Class 3 is instructed in English. The upper levels are instructed in Nepali. SMC, rural school, Cohort 2 [B.14.PC] HTs see monitoring work as additional and want a financial incentive…[Anyone would] have a hard time telling them this is something that was always in your job description. In fact, many HTs teach as many or more periods than regular teachers and do administration, so they have no time to do EGRP monitoring. ETC staff [B.15.PD] Most of the work on SMC grant proposals done by community mobilization officers, not community members; not sure they have been prepared to apply same proposal writing skills to SIP or to palika budget. EGRP Field Staff Palika-district level [see Question 4: Local Government section, below, for more palika implementation issues]. [B.16.PD] ESTIMATES a. 50% of palikas doing well with EGRP. GON, Cohort 1 district b. 30% of schools understand EGRP but don’t do. 30% don’t understand the teachers’ guidebook. That guidebook contains a lesson plan for each of the 192 school days, the material is well-sequenced, so clearly, the guidebook can decrease teachers’ workload. But if at some point there is something they don’t understand in the guidebook, then they don’t continue and revert to the Nepali textbook. And what happens when there are holidays/absences and the new book is sequenced for one lesson for all 192? days of official school year? RP, city, Cohort 2 [B.17.PD] 151 More rules from CLA would be better. Palikas and schools must implement this program, but some circulars would help. RP, town school, Cohort 2 [B.18.PD] EGRP is good but needs monitoring. LEU, rural school, Cohort 2 [B.19.PD] CSOs need to be allowed to interact with teachers. Right now, RPs and HTs not aware of what is happening in terms of community mobilization. EGRP Field staff [B.20.PD] KTM depends on data from Flash [reporting system] to print/distribute learning materials for coming year. Often mismatched to schools by time they arrive. Even if not mismatched, low attendance rates make them look mismatched. TLM couldn’t be delivered on time for first two sessions. EGRP Field Staff [B.21.PD] In first two years definitely confusion and TLMs came two months after start of school year. Then circular said we can use Nepali book as supplement. In a few schools had incomplete set [RM talked about mismatch between school census and actual enrolment one year later]. This year the books were late by one month as they changed teachers’ guide for Cohort 2. And then 15 madrassas haven’t even opened the TLM boxes. RP, city, Cohort 2 [B.22.PD] Last year, teacher training was delayed by six months due to the delay in release of funds. GON, Cohort 2 district. [B.23.PD] If the program had a stronger communication program, it would be better. He recalls the radio jingle only being used in April 2018 during the start of new session, mainly to increase enrollments. There are 15 radio stations in district and jingle only airs on three. Jingle should be published in newspapers, he says. For now, it is more like an advertisement than behavior change communication. All EGR activities should be linked to communication. Journalist, Cohort 2 [B.24.PD] Most important factor: Enhancing conceptual/practical understanding of government officers at regional [provincial?]/district levels. We insist that it is GON’s NEGRP program and ECDUs, LEUs say that. But they have not internalized. Heads of LEU say NEGRP will only work if EGRP keeps monitoring, providing materials because they do not feel technically empowered. First, they say they need technical capacity development so EGRP provided one-day orientation then three-day follow-up technical workshop. Some mayors, like the one in Palika X, chose not to participate. Now trying to figure out what other technical capacitation necessary. CB-EGRA is one thing. Perhaps community mobilization? EGRP Field Staff [B.25.PD] Although government can implement low intensity, it is the high intensity model that is necessary. Partner NGO, Cohort 2 [see Question 4: Local Government section, below, for more palika/district level implementation issues] Provincial/central level [B.26.PC] TLM for 2019/20 will be distributed by end of April 2019. EGRP central level [B.27. PC] Observed cluster level training in the city in December 2017. Saw 50% of books from book corner had disappeared or were not used. The HT said because the classroom had no doors or windows, books were easily stolen. NGO, Central level. 152 [B.28. PC] Disbursement of G2G money has been delayed by misunderstandings: is this donor or CLA money? What sort of reporting requirements/timing are attached? Transfer of funds to palikas delayed. Initially didn’t even realize ETC funds were in Ministry of Social Development budget, so funds really delayed. This year passed them three months ago. [December] GON, province level [B.29. PC] NCED used to release budget in 2nd trimester and then ETC would have to complete all training for the year in two trimesters. EGRP funds arrived in 3rd trimester; beginning in May 2018 they had to train 940 teachers in 3 months. This created terrible problems for the trainers, the majority of whom were secondary school teachers who needed to be in their schools preparing students for exams or administering them during this period. This may have affected quality of the training. ETC staff [B.30. PC] HT 5-day training takes HTs away from schools too long. Should be scheduled to include a weekend and cut back on school days. For example, could be Fri (half day), Saturday-Tuesday. LEU, rural, Cohort 2 [B.31. PC] Provincial Ministry of Finance raised travel allowance for TPD in Province 5 to NRP 800 pd but EGRP TPD remains NRP 600 pd. “We will convince them to come.” In Rupandehi district some teachers have shown up who were given no travel allowances. ETC staff [B.32.PC] Much effectiveness lost between TOT and when training actually delivered [and another gap between teacher training and teachers could practice whatever they learned in training, i.e., when TLM delivered]. EGRP, central level [B.33.PC] ISSUES WITH CDC a. Still need more change in TLM, but this is difficult for the government of Nepal. CDC wants integrated curriculum, but they are not clear about what that is. They may not sustain 90 minutes for Nepali. They may not integrate all the changes we have suggested in existing materials. EGRP Field Staff b. Lack Nepalese consultants, knowledgeable about the specifics of Nepali and context-based findings. Only 50% of consultants have any chance of being useful in the Nepal context. Lots of arguments about script for Rana Tharu between university and local consultants. Lots of problems with [using] letter frequency [as the sole criteria for the order in which letters are introduced] in a language like Nepali with 136 sounds & symbols. GON central level c. Technical staff (curriculum writers, editors, etc.) and policy makers need more capacity development for curriculum. GON, Central level d. Struggle over whether to introduce letters in order of frequency or in order of traditional alphabet. Traditional alphabet begins with many letters that are infrequently used, making it difficult to develop decodable texts as soon as children learn a couple of letters. Also, over the importance of 90 minutes/day of Nepali. Chief writer in CDC doesn’t believe it is necessary until he sees research conducted in Nepali that proves this. EGRP, Central level e. We are involved in policy formulation, material development and training. EGRP provides us with the services of consultants but we are not always happy with the quality of service received. The consultants are called “experts.” We really need to define what an expert is. GON, central level 153 f. CDC is understaffed and is working on integrated curriculum at the same time it is still not finished producing mother tongue and Nepali as a second language curriculum for EGRP. NGO, central level [B.34.PC] When designing the project, we were very excited that EGRP will positively shift classroom teaching paradigm. But this has not been the case to the extent we expected. GON, Central level [B.35.PC] G2G: USG and GON rules are different. USG pays more so no one wants to work for GON. GON, Central level [B.36.PC] I am not fully satisfied with the level of coordination we (GON) have amongst ourselves. Also, many community level efforts have been conducted under EGRP and are clearly visible, but it is very difficult to measure the individual household efforts in improving children’s reading habits. GON, Central level [B.37.PC] Our NGO had very successful EGR/MT program in an EGRP district, just before EGRP started. But nothing of our program has been integrated into EGRP. Don’t even use our experienced teachers as teacher trainers or mentors in EGRP; when EGRP sent them for training, our teachers knew more than the trainers. Still our teachers are working well, according to Language Commission visit, even without recognition/support from EGRP. NGO, central level [B.38.PC] Two years not enough. Monolingual may need four or five years of intensive support, with many follow ups with all stakeholders. Example: Two years after end of project, palika mayor was willing provide NRP 15,000 for materials, but needed advice from Kathmandu University on what materials. Communities with more than one MT may need seven years of assistance, not just five. NGO, central level [B.39.PC] Lots of turnover among GON Focal Points (often DEOs) Donor partner, central level. 154 C. VIABLE: Helping (schools in Cohorts 1-2 CLA) School level [C.1.S] We have good infrastructure, classrooms are enough, and toilet is in place for girls and boys (with ramp). We serve mainly the children of poor families from the community who would otherwise not be able to send their children to other schools (even the nearby government schools because they charge some money to the students per month as they are starting to teach in English medium). A2: We are good infrastructure wise – we have enough space for children to play. We are serving poor children and its free of cost unlike other schools nearby. A3: We sometimes provide dresses, copy and pencils to children who cannot afford them on their own. TFGD, rural school, Cohort 1 [C.2.S] Teachers from G1-G2 have up to +2/I Ed level education. G3 teachers have done B Ed. We have 5 permanent teachers’ quota from G1-5. We have 3 ECD facilitators, one of which have done MA in English. Our teachers from lower grade can also teach higher grade student. One of the qualified ECD facilitators teach 4 classes for G6 and G7. TFGD, town school, Cohort 1 [C.3.S] Since there are no other community schools nearby, our school is the only one for low economic family to enroll their wards. HT, rural school, Cohort 2 [C.4.S] We teach in Nepali, but we translate it to Maithili to meet the needs to Maithili speakers. Maithili has been taught as an ‘optional’ subject from G1-3. We teach Maithili as a language and we also take written exam on it. We do not have textbooks in Maithili. TFGD, rural school, Cohort 1 [C.5.S] Both Nepali and Abadhi language are used for teaching. 20% of students can talk in Nepali The teacher translates Nepali into Abadhi to order to make the student understandable. HT, city school, Cohort 1 [C.6.S] We have Magar speaking children, so we have appointed one teacher who is from the Magar community. HT, town school, Cohort 2 [C.7.S] The boarding schools are far from here. Parents do not want to send their children [so far away], so they send their children to this school. HT, town school, Cohort 2 [C.8.S] At lower [primary] level, medium of instruction is English which has brought a few of the students back from private schools and others from the private schools are being attracted. HT, rural school, Cohort 2 [C.9.S] DEO office provided some funds to reward parents whose children are good students HT, rural school, Cohort 2 [C.10.S] We have been charging some fee with the parents for teachers’ salaries. HT, town school, Cohort 2 155 [C.11.S] The HT has been working in this school since last five years and after he was appointed as HT, number of students has increased from 40 to 212. This is because he resides locally which has helped him gain the trust of local people, he is actively involved in organizing door to door admission campaign program to send their children to school and he is dedicated and committed to make this school a smart school. Researcher, rural school, Cohort 1 [C.12.S] Safe drinking water proposal, desk bench proposal, boundary walls, doors in the classroom etc are all included in the school SIP. HT, rural school, Cohort 1 [C.13.S] There is a ‘Community Forest’ nearby the school which also encloses the passing by river. They have given permission to the school to use the stones available in the riverbanks. School collected and sold those stones last years (by hiring the daily wage workers) and made NRPs. 30,000/- which was used towards paying the salary of privately hired teachers. HT, town school, Cohort 2 [C.14.S] School is near to the road and is easily accessible for all. This school is near to the village, so students can travel here easily on their own. HT is well disciplined and working for the betterment of the school. This school is better than other school (in terms of study/education) Our school is also forward in extra activities (got second position in football) SMC involved in auditing (every year.) SMC, rural school, Cohort 1 [C.15.S] In the last two years, SMC has been engaged in repairing and improving the physical infrastructure of school (classrooms and boundary wall) and management of school lunch program. SMC, town school, Cohort 2 [C.16.S] During the past 12 months I went at least 10 times to visit this school. During my visits, I asked to improve the teachers’ teaching qualities and to take care of our children’s learning. And I asked teachers to check regular homework of our children in school and I tried to ask the reason about too many local holidays. Guardian, rural school, Cohort 1 [C.17.S] Teacher always send us information about exam schedule, and they call us in school to receive our children’s results. Guardian, town school, Cohort 2 [C.18.S] School is near the community, so it’s easier for small children to go to school. School is near my home, so I keep meeting teachers and asking for my child’s update. Guardian, rural school, Cohort 1 [C.19.S] Since February 1 Sabal Nepal started for Grade 1 to Grade 3 morning tuition class in this school from morning 7 am to 9 am. They are hiring tutor from our own community. Guardian, town school, Cohort 1 [C.20.S] Teachers are active, and they have good teamwork. There is good coordination between SMC and PTA. They bring projects to school together. PTA, rural school, Cohort 2 [C.21.S] This school teaches students coming from 3-4 villages. Good things about this school are: It is near the village and easily accessible to community people. Infrastructure is good (classrooms, toilet, drinking water, boundary wall) 156 School is far from the highway so students cannot run away. Our school is child friendly (furniture and textbooks) PTA, town school, Cohort 2 [C.22.S] We have meetings on how to encourage Muslim community children to go to school. SMC, rural school, Cohort 1 [C.23.S] The compound walls have been under construction from the fund of member of provincial parliament. SMC, town school, Cohort 1 [C.24.S] With the initiative of ward office, door to door program to send children to school and sanitation campaign has been conducted in the community. Ward chairman, rural school, Cohort 1 [C.25.S] According to policies of new government, municipality is providing salary of privately hired teachers and also has budget of NRP 3,00,00,000 to spend on physical infrastructure of schools. Ward chairman, rural school, Cohort 2 [C.26.S] I have provided the cement and also donated money to pay the laborers for construction of school boundary wall. I also provided NRP 28,000/- for construction of school main entrance gate. I have planted the trees around the school area, taken care of it and later cut it down to sell the woods. We sold the wood for NRP 8,19,000 and that money was donated to school for its various uses. Last year, I distributed copies, pens, pencils, erasers and sharpeners to all the students. Benefactor, rural school, Cohort 1 [C.27.S] Most of the RC structures were converted to classrooms. The previous school clusters seem to have been replaced by the local education units of palikas. Some of the HTs informed that their regular meetings are held in the premises of their respective palikas. Various HTs, Cohorts 1 and 2 Palika-district level [C.28.PD] EGRP has helped stem transfers to boarding schools. Mayor, rural palika, Cohort 1 [C.29.PD] When I visited, I found few teachers but very clean, 40 flowers around school, toilets are clean, and the HT takes his turn cleaning them. “I am head person; I must do more than anyone else.” Lower secondary students go around the village promoting cleanliness. Mayor, rural school, Cohort 2 [C.30.PD] Lunch program two years ago helped to stop decline in enrolments and he is thinking of doing it again. Mayor, rural school, Cohort 2 [C.31.PD] The effect of CFS: attendance was better, but no change in achievement/retention. LEU, city, Cohort 1 [C.32.PD] Rural schools doing better than those near market [and private schools] because the latter have large % disadvantaged students. LEU, rural school, Cohort 2 [C.33.PD] 157 He developed a form for school visits: parts 1-3, how well do children read; 4-5 how do they perform in other subjects. LEU, rural school, Cohort 2 [C.34.PD] Palika Education Committee [parallel to old VEC?] got re-established at the beginning of the academic year. Total of 11 people, including HTs of community and boarding schools, meets ~once/mo. Because of them, all schools were visited in the last month. LEU, rural palika, Cohort 1 [C.35.PD] These programs are important, but implementation is the most important and HTs are the most important for implementation. 8-10 schools are implementing well, but these schools were already strong. LEU, rural palika, Cohort 1 [C.36.PD] RE: supervision. Permanent and relief teachers do listen to mayors and RPs. RP, rural palika, Cohort 1 [C.37.PD] Feedback is good. No complaints. That’s why palika is motivated to continue funding. 29 students transferred back to community from private schools. RP, rural palika, Cohort 1 Province-central level. [C.38.PC] This year was a bit better because CEHRD passed funds through MoSD. ETC [C.39.PC] The government requires all teachers are licensed. Most have B.Ed, M.Ed. Full 15-day X 2 Teacher Professional Development (TPD) offered at ETC to permanent teachers only, who receive a certificate at the end. Non-formal, temporary teachers may take 5-day TPD, but it leads to no certificate. ETC [C.40.PC] We are designing an [awareness raising] package for municipal officials, including jingles [promoting mother tongue]. Good response but we had limitations. GON, central level [C.41.PC] Language Commission (LC) has a constitutional mandate to do status analyses of MT and to recommend to GON a model for implementing MT-based Class 1-3 instruction, moving to Nepali/English in Class 4+. GON, central level [C.42.PC] We have kept EGR related questions (a small portion) in primary teacher selection exams – a very small portion within the scope of primary level teacher selection process. GON, central level [C.43.PC] INGOs could facilitate discussion at ward/palika/district around question: what changes do we want to see in schools? For example: in Pyuthan, said that within 9-10 years: we want learning achievement to increase from 35% to 60%, no out of school children, 100% regularity of teachers and students Announced a plan that included most EGR ideas. NGO, central level [C.44.PC] 158 2017 study of TPD/TPS showed most important factors for success of teachers: 1) TPS in school, 2) teacher is motivated and 3) not too much politics in school. Baseline funding from NORAD for a study on what do teachers do in the classroom? found less than 40% teaching well. Then discussed with palika, SMC: What is necessary to motivate teachers and how to support them (create a conducive environment). After two years (2018), attendance audit for teacher and student. Initially 90 minutes very difficult. But DEO supported. In some districts we developed monitoring checklist. Banke DEO tailored it and started using it. NGO, central level [C.45.PC] [Because of NGO EGR program] A FEW students have shifted from boarding schools back to community schools. As a result, in one palika, one boarding school will likely close in 2019/20. Result of very young and energetic HT good SMC. NGO, central level [C.46.PC] Technical Working Group provides good experience sharing. Also helped with Minimum Package. NGO, central level [C.47.PC] [When we were starting new EGR program,] Survey showed parents wanted English but agreed should learn MT first. RCs wanted to implement English in G1-G3. Media mobilization campaign. We developed Tharu TLM. Took rigid government officers to Cambodia, where they saw G8 students speaking good English, though they had only started studying in G6. After that, RP changed emphasis to MT. NGO, Central level [C.48.PC] We cooperated with Rural Women’s Development Center in Dang to teach mothers financial literacy. Developed five modules: EGR, MT, managing adolescent, developing a business. Each module discussed over 2-3 weeks, each mother developed action plan and after 15 days, the group checked on progress on plan. Mothers’ group-initiated reading club. Had mother-child reading competition at end of every month. Annual reading fair disseminated (2012-17) reading results. Some RPs and HTs allowed them to test in the schools. Small award to best-improved schools. Also linked “Welcome to School” initiative to fair. Some children came back to community from boarding schools. NGO, central level [C.49.PC] Our program concentrated in Tharu areas because of the lowest achievement there. Discovered there was high absenteeism because the mothers too busy to get breakfast ready on time and children wouldn’t come without it. So, worked with WFP to do school feeding program. NGO, central level [C.50.PC] Merging schools is a good thing because small schools dragged down average learning. NGO, central level [C.51.PC] Our pilot project had really good results: 8% of children could read with comprehension in baseline, rose to 53% after three years. Some project activities around MT. Also promoting MT during teacher training. NGO, central level [C.52.PC] Priority Minimum Enabling Conditions in SSDP captured a lot of CFS parameters (2014-15). Donor partner, central level 159 D. VIABLE: Hindering [D.1.S] Estimates of the proportion of viable schools able to benefit from any learning improvement intervention a. 35% of schools in palika are dysfunctional. LEU, town, Cohort 2 b. 40% of teachers in this district are not doing their duty in Classes 1-5. Teacher association, Cohort 2 c. ≥ 50% of schools have only nominal change—they are dysfunctional. NGO, central level d. Only 18% of Nepali primary schools meet minimum standards. Maybe 40% are dysfunctional. NGO, central level e. Humanity International study in 40 Nepali schools, using Washington Group Test found: 26% of children had some functional limitation 9-10% had severe disability 70% have learning disability Teachers simply don’t have the capacity to deal with these children. Education donor, central level School level [D.2.S] Reason for under enrollment: lack of birth certificates. Reason for lack of birth certificates: parents not registering marriages Reason for not registering marriages: trend of mothers eloping with other males but still demanding child support/a share of husband’s family’s property. TFGD, rural school, Cohort 1 [D.3.S] Those who come to school regularly are benefitting from EGR the most. The irregular students are the ones who are not benefiting from the program. HT, town school, Cohort [D.4.S] One of our most difficult problems is we have to keep Classes 2, 3 and 5 with ECD students in one class due to lack of teachers’ quota. TFGD, rural school, Cohort 1 [D.5.S] Our participant is an ECD trained female teacher but because its only HT and herself working in school, she has to manage Classes 1-5 and teach them every subject. ECD is run by the school helper. Researcher, rural school, Cohort 2 [D.6.S] Salary and teachers should be made on time. Since the palika has come into operation (last 8 months), we haven’t been paid. TFGD, rural school, Cohort 1 [D.7.S] Parents think that teachers can do everything – improve students’ regularity, fulfil adequate teachers’ quota, and improve parents’ involvement. But these are beyond teachers’ capacity. TFGD, rural school, Cohort 1 160 [D.8.S] End-of-year examinations a. There are three examinations held in school annually which determines the class promotion. TFGD, town school, Cohort 2 b. CLAs say we should do only CAS in primary, but parents won’t trust us if we don’t do final exam. RP, rural palika, Cohort 1 c. There are many students enrolled in classes Class 1-Class 3 but they are not very regular. Interestingly, since they are exam focused, attendance increases during examination time. TFGD, city, Cohort 1 d. The children said that the questions in final exam would come from the textbooks of GON but not from EGR workbooks therefore they would like more time to study textbooks. TFGD, rural school, Cohort 1. [D.9.S] All the members from SMC and PTA are inactive except for their respective chairpersons. They are money minded and if meetings and other activities/program provides with incentive, they happily join the program otherwise we have to reach them for signing the documents at their homes. HT, rural school, Cohort 2 [D.10.S] Parents are demanding for English language. There should be awareness class for parents to teach them the importance of education. We do not have enough classrooms for the number of students who are enrolled. SMC, town school, Cohort 2. [D.11.S] Increase in attraction towards boarding school/private school is not only because of English medium but also due to the availability of transportation facility, safe and clean drinking water, safe and well fenced school premises, toilet facilities etc. HT, town school, Cohort 2 [D.12.S] For the betterment of school, the children of those people who get government facility should study in community school. Ward chair, town school, non-EGRP Palika-district [D.13.PD] 80% of children are in English medium private schools. Among those attending public schools, many disadvantaged families. Lack of parental awareness among Madeshi and Muslims results in not enough nourishment, time to do studies at home for poorer students. Mayor, city, Cohort 1 [D.14.PD] Some older teachers are lazy. And evaluating teachers is local government responsibility, but teachers are not listening. Teachers central govt has appointed must have licenses but they are not willing to go where they are assigned. I can’t find anyone with a license for teaching science. Hopes the “stubborn” teachers will listen to someone appointed from central level. Mayor, town, Cohort 2 [D.15.PD] What activities is this LEU responsible for? Paperwork, salaries of teachers. LEU, town, Cohort 2 [D.16.PD] 161 If you are born in America, and don’t speak English, you are useless. Likewise, if you are born in Nepal and don’t speak Nepali, you are useless. Half of the government schools are teaching in English medium. They do this by hiring a private English teacher. It usually happens where there is an active HT who wants to maintain enrollments. All parents gravitate to higher status schools. GON, Cohort 2 [D.17.PD] The entire [hilly] district X is a bucket with a hole in it. The success of EGRP is not so affected by the infidelity of teachers but by mobility. By the time TLMs reach students, hardly 70 to 80 days left in school year. Still the teachers are trying and doing well inside the classroom, but community mobilization and distribution of materials are a nightmare. Last time I visited district X, I asked the EGRP staff person to point out his home, and he pointed straight up. I could not imagine going up and down there every day. EGRP field staff [D.18.PD] 40% of teachers in Dang not doing their duty in all Class 1-Class 5 classes. 1) Most teachers appointed by SMC and they are political. Hiring temporary teachers should be done through Teachers’ Commission but it has yet to be constitutionalized. 2) Teachers feel disrespected when observed by outsiders [like NGO]. Teachers feel suggestions from mayors/LEUs not always practical. Former SS really didn’t supervise. RPs are better. TA not happy RPs have been dismissed. Teacher association, Cohort 2 district [D.19.PD] CSOs need to be allowed to interact with teachers. Right now, RPs and HTs not aware of what is happening in terms of community mobilization. EGRP Field staff [D.20.PD] Has not heard that Mother Tongue is best foundation for English. LEU, city, Cohort 1 [D.21.PD] The SMCs focus [all their attention and resources] on higher classes. SS, Cohort 2 district [D.22.PD] Current problem in Nepal: lack of ethics. HR and material not the problem in schools. Need to improve understanding of responsibilities, diligence and ethics. If no ethics, nothing will work. Incorporate ethics in TPD. SS, Cohort 2 [D.23.PD] Too many local level holidays are a hindrance to effective implementation of these standards and benchmarks. GON, Cohort 2 [D.24.PD] Guardian awareness and door to door admission program should be conducted regularly. Ward chair, rural school, Cohort 1 Province and Central Level [D.25.PC] A decade ago, education sector was allocated 17 percent of the annual budget. Now it is down to nine percent. The government should substantially increase budget allocation for education. Teachers’ salary should also increase. Teachers’ union, central level 162 [D.26.PC] CDC ISSUES a. CDC overextended: Six CDC under-secretaries and six others transferred to provinces or LEUs, taking institutional memory. Need to revisit Federal Education Act. It is stuck in Ministry of Finance because cost is too high. Proposed additional agencies: higher education governance, accreditation and more. CDC’s role needs to be redefined: do we give others outside CDC the [curriculum] framework and let others [local government? NGOs?] write [and print?] textbooks? GON, central level b. CDC supposed to have 9 directors, but 3 have been transferred. Now CDC is asking for comments on books they have prepared but they will not have enough time to incorporate comments. Time frame for producing materials is too short. NGO, central level. c. There is no formal communication between the Language Commission and CDC and CDC is not necessarily listening to their recommendations. GON, central level c. All children should be learning Nepali in both government and private schools. However, NASA clearly indicates a problem in Nepali; there has been no improvement in learning. Achievement in Nepali is eroding in the rush to English medium. CDC team needs to be reoriented. GON, central level e. No child development/early reading experts in CDC/CEHRD. People who have some technical expertise are transferred; others are transferred before they can develop technical expertise. EGRP, central level f. Need better understanding of child development and neuropsychology of reading in CDC. NGO, central level [D.27.PC] Teacher training is the most challenging: delivery is dreadful. Teachers are randomly selected; they are not necessarily teaching Class 1-Class 3 or focused on Nepali. They only get 1 ½ days out of 3 days of training. Only permanent teachers supposed to be trained at ETC [But we heard the teachers got 5-7 days of training.] Many teachers then got transferred, lost job, started teaching another subject. When temporary teachers fail teacher service commission test, can’t go back to work and EGRP must train more. NGO, Central level. E. SUSTAINABLE: Helping School level [E.1.S] If we can receive TLM/SRM and refresher training for teachers, we can continue this program without any problem. HT, rural school, Cohort 2 [E.2.S] Even in absence of external funds, the program should continue and will continue in our school as we can manage the funds from the sources we have. HT, rural school, Cohort 2 [E.3.S] The school is planning to continue EGRP and is talking about new ways to raise funds. HT, city school, Cohort 1 [E.4.S] Even if we will not get any workbooks, we can follow the teaching techniques. Also, “Maula ke daud masjid tak (priest goes running to the temple)”, we will go to palika and district level (District coordinating committee) and ask for the materials. TFGD, rural school, Cohort 1 163 [E.5.S] Yes, there is a Tole Reading Group in our community. I try to attend it and am learning to write my name. We also talk about sending our children regularly to school. Guardian, rural school, Cohort 1 [E.6.S] Every Saturday Tole Reading Group (TRG) is organized, where students from government and private school come together and read. Rural municipality allocated NRP 50,000 for 33 TRG (NRP1500 per TRG) and distributed that money. SMC, Rural School, Cohort 1 [E7.S] Civil [Palika? Former Village?] Education Committee has been formed. Assigned the ward member to look after at least one school and report to the ward chair about the achievement of school. There are regular visits from the ward member to the assigned school. Started the anti-drugs campaign in schools in coordination with the Police so that the students are not into the drugs. Started to provide Day Meal to one of the primary schools as there was demand from the parents. Talked with Mayor for the increment of salary of the ECD teachers. Ward Chairman, town, non-EGRP Palika-district level [E.8.PD] Palika has planned some community outreach, has funded reading festival for next school year during which schools will be encouraged to make their own learning materials. Formerly 62 TRGs; about half are active with help from ward. Has talked to ward chairman and is hopeful. RP, rural palika, Cohort 1 [E.9.PD] It is a GON project: Local Govt will continue to support the good things from EGRP, even after external assistance phases out, we must do it. LEU, rural palika, Cohort 2 [E.10.PD] As soon the project finishes, teachers will need refreshing and new materials. Maybe 10 highly motivated teachers can keep going in schools that have good HT and are financially sound. LEU, town palika, Cohort 2 [E.11.PD] HT can coordinate with ward to have Social Mobilization. GON, Cohort 1 [E.12.PD] Schools capable of sustaining EGRP in this district: 50% (Another district: 40%), NGO, Cohort 2 [E.13.PD] Of 121 government-supported madrassas in this district, 15 are very good, including one that has expanded school year to 242 days. EGRP field staff [E.14.PD] [What survives after end of intensive intervention?] TRGs continued for 6-7 mos. Ward provided NRP 1000 incentive for 2- 3 hours for teacher. School provided room, carpeting, and books. Guardian association, Cohort 1 [E.15.PD] About 50 percent of SIPs have included EGR matters. All SMC grant recipient schools have committed to include EGR matters in their SIPs. Honoring of parents, reading fairs and reading competitions are most likely to continue. EGRP field staff, Cohort 1 [E.16.PD] Projects like World Education demonstrated it previously in the district and is necessary. Journalist, Cohort 2 164 [E.17.PD] EGR is not new. It’s already in the curriculum, part of CFS. It simply involves giving children materials and helping them to study. EGR can make CAS better. EGR involves some behavioral change: how much does the student says he understands? GON, Cohort 2 [E.18.PD] There is a long tradition of emphasizing reading in our culture. With minimum support and we will continue. Not much cost to organizing parent meetings peer-to-peer work training and community mobilization should not be a problem. GON, Cohort 2 [little experience with EGRP] [E.19.PD] Some history of benefactors. Most funds from private donors go to the secondary level. But perhaps could be persuaded to invest in library if some way to publicize donors could be found. [I suggested listing names of benefactors on school walls, as in Salyan.] EGRP Field worker [E.20.PD] We have philanthropist and private donors many of them. But initially probably wouldn’t it be interested in TLM. The EDCU could make it happen. We could try putting their names in the front of books. If they get publicity, they will be satisfied. It is possible with counselling and motivation that they will donate for library books. GON, Cohort 2 [E.20.PD] I am confident that the government will mobilize resources for EGRP’s continuation. GON, Cohort 2 [E.21.PD] Existing arguments for MT: However, if you want to learn a language, wait until you get there. All the Nepalis now working in Korea did not learn Korean before they went to work there. Language can be learned as the need arises. GON, Cohort 2. Province-central level [E.22.PC] Although his primary role is to develop policy and planning, sees a role for his office strengthening capacity of 109 LGs in province 5 to implement EGRP, for example, budgeting and planning with ETC. Will bring EDCUs to the province level or MOEST will reinstate them at district level. GON, Province level [E.23.PC] World Education did EGR in two VDCs, finished in 2016. Quality in all aspects improved. Even the working style of teachers improved as they were trained on mental health of students. Big increase in reading capacity compared with other VDCs. In participating schools, level of reading in Class 2 higher than in Class 6 in the school of other VDCs. NGO, Central level. [E.24.PC] In 2018, 37 palikas asked us for help to orient and develop strategies and standards in ECD/EGR. NGO, Central level [E.25.PC] [In response to EGR program results in public schools,] A FEW students have shifted from boarding schools back to community schools. As a result, in one palika, one boarding school will likely close in 2019/20. Result of very young and energetic HT, good SMC. NGO, Central level [E.26.PC] Believes committed community leaders, HTs and govt officers should organize more awareness-raising/motivational messages to parents [Witness his 2.5-hour speech to guardians and PTA members in his home palika the previous week. Contained little EGR substance, mainly folksy anecdotes about treating children better.] GON, Province level 165 [E.27.PC] After EGRP finishes, EGR activities will be maintained, just not as intensively. GON, Province. [E.28.PC] Because the teachers are positive about EGRP, the chances of sustaining current level of activities are high. GON, Province [E.29.PC] I am hopeful that there will be no dearth of donors to fund a good program like training in EGR. ETC [E.30.PC] Incorporating core EGR documents into integrated curriculum. SSDP has clearly mentioned commitment to EGR. To finish developing & testing Nepali & Mother Tongue materials: Need EGRP Phase 2. GON, central level [E.31.PC] We understand some HTs have been overburdened by the number of classes they need to take each day. So, we are thinking about creating a HT position in such a way that their primary role becomes leading and monitoring. We are thinking of starting this with bigger schools first and implement it nationwide. Since the government has decided to remove Resource Centers, HTs have bigger role to play. GON, central level [E.32.PC] Up to 60% of TRGs could be associated with a CBO. Government thinks community mobilization ends when project ends. Need to engage palika government and get them to engage all stakeholders EGRP central [E.33.PC] Sustainability: CB-EGRA and NASA, definitely. Training, maybe. EGRP central 166 F. SUSTAINABLE: Hindering Estimated sustainability levels: Banke and Bardia: 35 to 40% [NB: Same estimate given by former RP in Nepalgunj sub-metro palika) Dang and Surkhit: Will struggle more. EGRP, field staff F.1 Overarching issue: What is a good school? [school level, Cohorts 1-4] a. If the school has kept students’ learning in the centre, that school is a good school to me. Rest of the other things are secondary. My school is an average school. Infrastructure-wise, we are lucky. But learning wise we are just average. It used to be a good school before, but now this has not been the case as the number of students has greatly decreased. It is one of the oldest schools of this area (I have been a student myself). Unfortunately, we couldn’t keep up with the demand of present times. HT, rural school, Cohort 1 b. A good school needs to have active SMC, palika, teachers, and parents. A good school also has quality teachers, good students and active parents. HT, rural school, Cohort 1 c. Our school is not good as we don’t have enough teachers, and the students and teachers are not regular to school. SMC, rural school, Cohort 1 d. Teachers in good school should teach in a joyful environment so that students do not feel bored. Guardian, town school, Cohort 2 e. Our school is doing satisfactory – we do not have enough physical infrastructure, adequate number of teacher training and active parents. These all are necessary to improve the quality of school. TFGD, town school, non-EGRP f. A school should have enough infrastructure such as building, library, classrooms. Teachers should be well trained, capable and adequate. School should focus on parents’ education to make them aware about their role in children’s education. A school should also focus on practical education. A good education should keep students’ talents in the centre and work towards grooming them. HT, town school, non-EGRP g. A good school should have good teamwork between its HT, teachers, SMC and PTA. Teachers in a god school should use modern technologies to teach students (they can be trained to use it). A good school should have active parents. A good school should have good supervisor to look after their teachers and advise them whenever needed. A good school should aim to complete their curriculum in given time. A good school should also be child friendly and provide quality education to their students (well trained teachers, provide learning materials to the students). A good school should also promote English language because recently parents are demanding English language. There should be no politics or any other pressure in a good school. HT, rural school, non-EGRP School level [F.2.S] The children from Magar and Dalit communities come to our school, but the children from privileged families go to boarding school. HT, town school, Cohort 1 [F.3.S] For the betterment of school, the children of those people who get government facility [job] should study in community school. Ward chair, town school, non-EGRP 167 Palika-district level [F.4.PD] Many SMCs do not want to invest in print rich rooms, to buy TLM with EGR grants. Instead, they are focused on infrastructure development. EGRP field staff [F.5.PD] If parents can be made aware of importance of reading, they will take more initiatives. All students in government schools have low-economic background and those parents need more awareness. RP, town, Cohort 2 [F.6.PD] We have no plans for funding the EGR activities when it ends next year. Our NGO is specially focused on sanitation activities and our program is already phased out in XX School. We have started and set the practice of working in group among the students and until now this practice is actively on. We suggest NGOs should conduct programs that encourage the parents to get involved in their children’s learning. NGO, Cohort 1 [F.7.PD] After June 2018, 0 TRGs in the city palika; only 15% survived in other palikas. NGO, Cohort 1 [F.8.PD] 22 months of social mobilization implementation was too short. NGO, Cohort 1 [F.9.PD] Period very short. Social mobilizers were helpful but when they [NGO] tried to hand responsibilities off to Ward Chairs and HTs, not very successful. Guardian Association, Cohort 1 [F.10.PD] If the CLA sends us directives, if the CLA makes it our responsibility, if the CLA sends us the resources we will continue the program. It will also be better if there is something sent to us from the centre in paper and ownership is given to us. GON, Cohort 1 [F.11.PD] District officials say nothing substantial has been done here; they are interested in but don’t feel technically capable of doing CB-EGRA. EGRP field staff Provincial-central level [F.12.PC] MT materials no longer a priority; CDC focused on producing “Integrated Curriculum”. Minister got the idea from the Finns, but Finns say it takes 20 years to do it right. Finnish technical assistance support about to end. EGRP central level [F.13.PC] Need to revise EGR material to fit into new integrated curriculum (the draft is currently under review with the Curriculum Council). Hope to design the entire curriculum along the lines of the EGRP all-in-one textbook/exercises/workbook. Would develop 1 book from 2 current volumes of EGR. Next week there is a four-day workshop to finalize. GON, Central level [F.14.PC] Now just a four-day workshop to develop materials, before integrated curriculum is ready. Donor Partners are providing now help with content. CDC has asked for help with design and illustration only. What is driving force? Minister said they would implement integrated curriculum by 2019… NGO, Central level. [F.15.PC] In April 2019, integrated curriculum will be piloted in 100 schools. EGRP, central level 168 [F.16.PC] Might not integrate EGRP into TPD until 2025. EGRP, central level [F.17.PC] [What are the big problems?] Irregular teachers. Irregular students. Difficult to hire teachers locally. Suggestion: INGOs could? facilitate discussion at ward/palika/district around question: what changes do we want to see in schools? For example: in Byuthan, said that within 9-10 years: we want learning achievement to increase from 35% to 60%, no out of school children, 100% regularity of teachers and students. Announced a plan that included most of SC/LB ideas. NGO, national level. [F.18.PC] Nepalis understand that USAID was putting procurement through RTI to reduce corruption; not sure how program will do when it is entirely run by GON. NGO, central level 169 G. SCALABLE: Helping [G.1.S] We give 45 minutes for language class which is not enough. We need to at least give 1 hour for it. TFGD, rural school, non￾EGRP [G.2.S] Through the support from Room to Read (RtR), we have been allocating 90 minutes for Nepali language teaching in Class 1. This is also not sufficient. It should be 120 minutes. TFGD, rural school, non-EGRP [G.3.S] First priority is we want to increase the use of TLM while teaching. We are also prioritizing child friendly classrooms. We also plan to make parents aware to improve students’ regularity to school. We also want to teach students on the basis of what they already know, and we also want to conduct extracurricular activities alongside quality teaching. HT, town school, non-EGRP [G.4.S] [PLAN’s] training our teachers received has contributed to our children’s reading skill primarily because they use TLM while teaching. Students understand the teaching well. HT, town school, non-EGRP [G.5.S] [PLAN supported school] Yes, children’s learning has improved compared to previous years. PTA, town school, non-EGRP [G.6.S] [PLAN supported school] The children were very active and were very keen on interacting with the evaluation team. The children were on time to come to school and were very interactive. Interviewers, non-EGRP school [G.7.S] Students’ learning achievement was the point of discussion in last meeting. HT, town school, non-EGRP [G.8.S] If we can divide the weaker students in different group, it would be helpful to improve their achievement rate. HT, rural school, non-EGRP [G.9.S] We are enrolling students in Class 1-3 without birth certificate (they can take their time to register for it, meanwhile students don’t have to miss studying). HT, rural school, non-EGRP [G.10.S] I visited school to attend meetings, they were about improving learning behaviour in students and about school lunch program. Guardian, town school, non-EGRP [G.11.S] My child was in private school as the study of that private school was not satisfactory, I admitted my child to this school, and she is performing well in this school. Guardian, rural school, non-EGRP [G.12.S] Our SIP was made by me, SMC chair and teachers. We made it recently in January and passed it. We have made an action plan for implementation. The priorities are evaluation of students’ attendance on a monthly basis and improvement in their regularity to school. More time allocation for teaching-learning activities. Developing lesson plan and using it. Use of locally available TLM. HT, rural school, non-EGRP [G.13.S] 170 Development of SIP: There was visit from the RPs and SS Yearly plan was developed for school. Meeting between SMC and teachers was held. There was donation from the palikas for the office materials. All the activities were useful but the meeting between the SMC and teachers was very useful as in the meeting we discussed how the learning of student can be improved and also discussed about the infrastructure to be constructed. SMC, rural school non-EGRP [G.14.S] Our budget [for how many years?] includes the following: Community user Group: NRP 25,000 (sitting arrangement in classrooms) NGO: NRP 22,000 + 14,000 VDC: 24,000 (construction of boundary wall and slide for students) Nagarpalika: NRP 19,000 (textbooks) NRP 13,000 (scholarships) NRP 15,000 (SMC activities) NRP 1,000 (Audit fee) NRP 15,000 (development of SIP) NRP 96,000 (for construction works) EDCU: NRP 50,000 (paint and compound gate construction). HT, town school, non-EGRP [G.15.S] Our budget includes funds from: Ministry of Education: NRP 36,00,000 for school building and 30,000 for water tank. Gaunpalika: NRP 10,00,000 +5,00,000 for tiles in playground and school building. NRP 5,00,000 and again NRP 6,50,000 for science lab. They gave us NRP 50,000 gain for being a good school and NRP 45,000 +11,000 because we ranked first place in district level sports competition. Provincial government: NRP 10,00,000 for extending school field. Ministry of Peace: NRP 5,00,000 for construction of 2 classroom and some used in school building too. HT, rural school, non-EGRP [G.16.S] We wish that EGR would be implemented in our school as well. HT, town school, non-EGRP 171 Palika-district Level [G.17.PD] Cohort 1 and 2 have gotten inquiries about EGRP a. EDCU transferred from Cohort 1 to non-EGRP district. He had TOT training and requested help from EGRP-DC for 10- day EGRP training for HTs. When? Also asked for help with 2nd training for rest of HTs in district. EGRP-DC, Cohort 1 b. Siraha is interested in the program. It is closer to Saptari therefore teachers and HT from that district have frequently shown their interest in the program Okhaldhunga is also interested. EGRP-DC, Cohort 1 c. The Deputy Mayor from a municipality of neighboring Bhojpur has expressed his interest in introducing NEGRP in his municipality. NGO, Cohort 2 district d. Teachers from Tehrathum and Bhojpur have expressed their interest in EGRP. EGRP-DC, Cohort 2 e. Rolpa District teachers have requested ”When is EGRP coming to our district?”. This happened when they came to our districts to visit their friends and heard about the project from them. EGRP-DC, Cohort 2 f. Before high intensity could start in one Cohort 2 district, EDCU kept asking how to start High intensity program. He picked to 18 Schools that most needed help and try to implement. EGRP-DC, Cohort 1 [G.18.PD] Yes, we are aware of EGRP implementation in [adjacent EGRP districts]. It is an intensive program with constant interventions. It will be useful for this district as well. NGO, non-EGRP district [G.19.PD] [In terms of raising student learning achievement, if you could find funding for one intervention, what would you invest in?] Improving student attendance. RP, rural palika, non-EGRP [G.20.PD] We now have 90% attendance. [ Centrally funded] School lunch program has increased enrollment/attendance. Lunch program approved in budget and got some orientation last June-July and also in September. We have been doing program for seven months and still waiting for money from center. For now, children bring lunch from home (without junk food) and they get cash to take home. SS, town palika, non-EGRP [G.21.PD] Strong schools include XX Primary in YY palika and ZZ. Both benefit from NGO program but XX was doing well before NGO came. The parents in that school their own teachers, who are poor and have no other job prospects. Parents are determined to invest in that school and improve it. GON, non-EGRP district [G.22.PD] The scores on final Nepali exams have gone up in my district: 2017: 37, 2018: 43, 2019: 50. GON, non-EGRP district [G.23.PD] 50% of primary schools are of good or excellent quality. GON, non-EGRP district. 172 [G.24.PD] Our NGO has worked for four years in this town’s public schools and now finishing up. Teachers are continuing with the program [on their own]. We have tried to raise awareness among parent about children’s mental health, what constitutes good parenting; with teachers try to work on 1) classroom management, 2) positive disciplinary approach and 3) child counselling. Very satisfied with outcomes. Has helped with holistic development of children. It has changed teachers. If other programs ask parents to come, they don't. But they ask our program to for longer, more frequent training. Parents see that they don’t give enough time to children and they have learned to control their stress and to discipline positively. Now parents have learned to encourage children, to express their care for children, counsel them. They used to feel ashamed to play with children. Parents report these things to teachers and teachers verify with children. NGO, non-EGRP [G.25.PD] We are appointing more teachers based on classroom requirements. Rural municipality has managed the budget to pay the salary of privately hired teachers/private teachers. Rural municipality has also started the provision of day lunch in primary level classes. Ward chair, rural palika, non-EGRP [G.26.PD] What factors does your association think would bring about the biggest improvements in education in this district? Consciousness about their duties on the part of stakeholders and honest adherence to their duties. Guardian Association, non-EGRP district [G.27.PD] In palika, where Save the Children was working, government schools used to win palika-wide reading competition every Friday. Ward has one private school and it had to adopt EGR [90 minutes of Nepali daily] into order to compete for students. Journalist, non-EGRP [G.28.PD] [Having watched Save the Children and Room to Read in this district, he estimates] 40-50% of schools in palika could run EGR without intense NGO monitoring. Working with ward/palika government on sustainability. Journalist, non-EGRP [G.29.PD] There are 250 journalists in this district. Education is a priority of all the journalists. Journalist, non-EGRP Province-central level [G.30.PC] Yes, it’s a good model for many districts, with local adaptation, particularly for non-Nepali languages. GON Province level [G.31.PC] CFS was implemented in a better way than EGRP: Focused stakeholders on “what should a good school look like? Biggest achievement: got rid of sticks and corporal punishment. Still seeing results: teachers no longer being engaged in politics. NGO, Central level 173 H. SCALABLE: Hindering School level [H.1.S] Students aren’t very active. The attendance chart, book corner and student job chart are all present in classroom but not in use since the NGO finished its program in the school. RCA, town school, non-EGRP [H.2.S] Parents are attracted to send their children to boarding school as their living standards are gone up. Parents pay less attention to community school as we teach in Nepali medium. This is the reason why we do not have enough children. Only the children who are from low income families come to our school. HT, town school, non-EGRP [H.3.S] Our hands are tied due to lack of proper and adequate funding and we are compelled to distribute the salary of one teacher among 5 of them. SMC, rural school, Cohort 1 [H.4.S] We are not able to increase the number of students due to geographical location; students walk about 5-6 hours daily to come to school, minimum distance of students is 90 minutes, but we cannot help it. Due to increasing demand of parents and other community people, we started English medium in Class 1-5 in BS 2065 but it was too difficult for students to understand the language, so, we stopped it and introduced English medium only to Class 1 and have been increasing it class wise every year (G2 next year and Class 3 after that). HT, rural school, non-EGRP [H.5.S] The school is not good school because infrastructure is not good, and the teaching is also not good. The teacher should repeat the lessons until the students understand it The teacher should ask questions to improve the learning. Guardian, town school, non-EGRP [H.6.S] Our school is doing satisfactory – we do not have an active SMC, physical infrastructure is not in good shape, teachers’ quota is not adequate – the teachers hired for lower grade are teaching in upper grade. we need more teachers. We need more furniture and better managed classrooms. HT, rural school, non-EGRP [H.7.S] Our main problem is managing with students’ absenteeism. We are also finding it difficult to balance between normal and weaker students. We have certain goals to reach in given time, we always seem to be running behind the timeline. NGO, town school, non-EGRP. Palika-district level [H.8.PD] After assuming the position of mayor in this municipality I conducted a survey of both community and private schools. Even though the community schools are well-facilitated, and teachers are drawing handsome salary students are not learning primarily because the teachers are irregular in classroom teaching. Mayor, town, non-EGRP [H.9.PD] There’s a conflict between old and new teachers; the old teachers that are not retiring need training. Mayor, town palika, non-EGRP. 174 [H.10.PD] On average schools only met [or students only attended?] 174/192 school days in 2017/18. Several sources of problem: 1) Parents take children out of schools for festivals/marriages, religious ceremonies, 2) Political strikes Teachers are not allowed to repeat lessons. RP, rural palika, non-EGRP [H.11.PD] In X palika found only 9 schools that could meet NGO selection criteria. In Y palika, out of 84 schools, found only 34 have > 12 students in Class 1. The palikas are still struggling with merging/non-merging pressures [H.12.PD] Many had no extra room for a library and many more had less than five teachers and had no spare teachers to mind library. There is a problem: many parents sending children to private schools then fewer children in public schools can’t meet school/class size criteria for NGO program. NGO, non-EGRP district [H.13.PD] Only 25 of 200 schools in the six palikas where this NGO works could implement EGR effectively without lots of support. NGO, non-EGRP [H.14.PD] 25% of schools in this district are dysfunctional a) Nationwide, not enough teachers appointed. He teaches 11 classes and seven periods. No break, no time for lesson planning. b) Primary level teachers have been reassigned by principal to secondary classes because no teachers appointed for secondary level. One teacher may have 120 students [at secondary level?]. Can we teach them? Can we review their homework? 5% of schools have too few students, 15% are ok, and 80% are overcrowded. Teacher Association, non-EGRP [H.15.PD] 8-10% of teachers are qualified but lazy. Out of 1500, 500 are MA level and don’t take teaching seriously. Yes, previously there were people who drank alcohol during the day, but the association has been monitoring and reducing that. Teachers’ Association, non-EGRP [H.16.PD] Teachers’ salaries/benefits are inadequate. Get NRP 30,000/year? month? for tea, markers, lunch/snack, logistics. If HT travels, he must use his own money. If school appoints teachers internally, very hard to pay salary. So, no funds for little morale raising things, like giving prizes for student extracurricular activities on Fridays. Teacher Association, non-EGRP [H.17.PD] The message “Your schools are weak in Nepali” was not well-received by the EDCU. Interviewer, non-EGRP. [H.18.PD] [Central] government has not allocated enough teachers. We also have problems with roads and untrained teachers. If LGs hire local teachers, they leave in 6 months to join boarding schools. Lack of awareness among guardians. District capitol has high achievement, but remote areas are very low. GON, non-EGRP district [H.19.PD] Schools are like hikers wandering aimlessly without a goal or target. They don’t know how students are reading. Existing assessments are a problem: if you fix a race between a cheetah and a monkey, you know the cheetah will win. Journalist, non-EGRP district Province-central Level. See Question 5, below. 175 EXCERPTS RELEVANT TO QUESTION 4 I. Local government/federalization: factors helping EGRP School level [I.1.S] We received for following grants from palika: Scholarships: NRP 52,000 Textbooks: NRP 50,000 SIP preparation cost: NRP 8,000 Miscellaneous: NRP 15,000 HT, rural school, Cohort 1 [I.2.S] Received from MOEST : NRP 36,00,000 for school building and 30,000 for water tank. Gaun palika: NRP 10,00,000 +5,00,000 for tiles in playground and school building. NRP 5,00,000 and again NRP 6,50,000 for science lab. They gave us NRP 50,000 gain for being a good school and NRP 45,000 +11,000 because we ranked first place in district level sports competitions. Provincial government: NRP 10,00,000 for extending school field. Ministry of Peace: NRP 5,00,000 for construction of 2 classrooms and some used in school building too. HT, rural school, non-EGRP [I.3.S] We have received NRP 200,000 from ward office for boundary wall, library and child friendly classroom management. However, we have used only 40% of the total amount for the given topics, 60% of the amount has been utilized for private quota teachers’ salary with consensus from everyone concerned. HT, rural school, Cohort 2 [I.4.S] Rural municipality provided grant to four schools for construction of their boundary wall, namely: Nepal Rastriaya Secondary School- NRP 5,00,000/- Nepal Rastriaya lower Primary School- NRP 6,00,000/- Kalika Primary School- NRP 4,00,000/- Amrit Primary School- NRP 1,00,000/- It has also provided uniforms for few students whose family are very poor (he doesn’t know the exact number). Rural municipality is also managing to pay the salary to additional number of teachers that is required in some schools. Ward chair, rural palika, Cohort 1 Palika-district Level [I.5.PD] NARMIN insisted proposed Federal Education Act, due to be approved on March 6, would infringe upon 23 education responsibilities assigned to palikas by new [2015] constitution. NARMIN made three demands: 1) the Act be sent back into committee and not revised until after National Education Policy finalized (initial draft recommendations already submitted to MOEST by National Education Commission), 2) the portions of the Acts related to technical areas must be submitted to LGs for consultation and 3) senior lawyer to helped to draft the local level acts and regulations necessary to operationalize Act. Mayor, rural palika, Cohort 1 [I.6.PD] With federalization, won’t need help from NGOs anymore. Palika Education Committee [parallel to old VEC?] got re￾established at the beginning of the academic year. Total of 11 people, including HTs of community and boarding schools, meets ~once/month Because of them, all schools were visited in the last month. Mayor, rural palika, Cohort 1 176 [I.7.PD] With a view to improve teaching and learning in community schools, we have developed a 54-point code of conduct for the community schools to follow. We hope to see improvements in the months to come. Mayor, rural palika, non-EGRP district Province-central Level [I.8.PC] CLA indeed thinks federalization is a destructive administrative earthquake but those of us stationed at the province and district level think it is a good thing, it’s what the people want. GON, Province level [I.9.PC] Informant wants to reinstate ECDUs under the Ministry of Social Development (MoSD) at provincial level, as other technical ministries have placed their equivalent office there (almost all including health, agriculture, etc.). In his mind, EDCUs should coordinate education implementation with palikas while DCCs can work on planning/budgeting. Majority of palikas have not prioritized education because EDCUs have evaporated/stopped working. MOEST cannot understand provincial-level activities and needs to place the EDCUs under MoSD, which specializes in working at the district level. All seven PEUs sent request to CLA recently to restructure EDCU to cover more topics, such as sports. GON, province level. [I.10.PC] The 10 EGRP Cohort II districts are comprised of 99 palikas total By May 2019, EGRP will orient and have in place 17 District s to work directly with palikas and DECs Will support local level planning Help formulate education policy at provincial level Orientations and review and capacity development at district level Mobilizing through NGOs, encouraging them to advocate through teachers Helping HTs do TPS through capacity development and support at local level Developing classroom monitoring tools for HTs EGRP, central level [I.11.PC] Education sector is going through a bigger transition than any other sector – the new structure is yet to be tested and worked. In other countries, the local governments have done great jobs – it’s our time to wait and see our local governments. GON, central level. Local Government/Federalization: Factors Hindering EGRP School level [I.12.S] RP and palika people have visited but they have not observed classroom teaching learning – they just looked at our attendance register and went away. TFDG, rural school, non-EGRP [I.13.S] He couldn’t get any help from rural municipality because of political issues (he and ward chair are from different political parties). We feel like rural municipality has discriminated this school. SMC, rural palika, Cohort 1 [I.14.S] The mayor, Nagar education committee members and RP have come to visit our school from time to time. They don’t talk about quality but blame us teachers when there are problems. They also alarm us by talking about merging our school to another school. TFGD, town school, non-EGRP 177 [I.15.S] After the restructure of government in the country, responsibility of taking decision on educational matters is given to educational unit at municipality and we do not have anything at ward level. Ward chair, rural palika, Cohort 2 [I.16.S] After the restructuring of government in the country, local governments haven’t received any guidelines for educational sector, and they say it has made their work difficult. Ward chair, rural, non-EGRP [I.17.S] Ward office hasn’t done anything towards improvement of quality of education because no guideline has been sent to us related to sector of education from the central level. Ward chair, rural, non-EGRP Palika-district Level [I.18.PD] Now a problem because all the people are not accepting new allocation of responsibilities under federalism. Mayor, town, Cohort 2 [I.19.PD] RP received transfer letter yesterday. Mayor asked him not to leave right away because he needed to arrange Class 8 exam! Tell CLA: “This is the wrong decision.” Mayor, rural palika, Cohort 1 [I.20.PD] Quality of primary education and health were big issues in mayoral election. Drastic change since election: instructed all 57 schools to convert to English Medium. Distributed [English?] textbooks for free. Began EM in Class 1 and fired teachers who couldn’t teach English at the Class 1 level. 25-30 teachers came for 7 days of training in teaching English language. Will expand to one grade each year. He says enrolment has gone up. [But could not give us statistics.] Mayor, rural palika, non￾EGRP [I.21.PD] We have LEU in place, but the municipality has no authority to recruit, transfer or fire teachers. This is a hurdle in the way of educational reform. Mayor, rural palika, non-EGRP [I.22.PD] What other support does the LEU need to do its job effectively? Computer operator Vehicles to visit schools Clear policy direction about the role and responsibilities of LEU. LEU, town palika, non-EGRP [I.23.PD] In light of federalization, there are no clear policy directives on the roles and responsibilities of LEUs. LEU, rural palika, Cohort 2 [I.24.PD] Orientation of mayors is very important. High level of participation from gaunpalikas but education has less priority in sub￾metro and nagar palikas. LEU, rural palika, Cohort 1 [I.25.PD] Mayors don’t know anything yet; they want to prioritize education but knowledge of EGRP is limited. Need more than one day orientation. Training needs to cover budgeting for education. RP, rural palika, Cohort 1 [I.26.PD] Most important action necessary to strengthen schools: reinstate RPs. LEU, town, Cohort 2 178 [I.27.PD] Under centralized government structure, had all the necessary guidelines, procedures; now need some for local government. Last month held advocacy meetings for all 10 mayors and their LEUs. Eight or nine agreed to dedicate more funds for education than for health or education. All? Most? agreed to raise ECD facilitator salary from NPR 7000 to 10,000. EGRP partner NGO, Cohort 2 [I.28.PD] 8-10 palikas proudly allocated NPR 200,000 for TRGs but in fact TRGs were never consulted on planning this allocation. The TRGs were never asked to prepare a proposal. Without telling the TRGs, the palikas spend those funds on whatever they wanted. EGRP field staff [I.29.PD] Previously in this district, there were 15 resource centers, now just 10 LEUs. Not enough Palika education officers to really replace resource centers. Tulsipur and Ghorahi had three resource centers each; now just one LEU each. EGRP, field staff [I.30.PD] LEUs have proposed ward chairpersons should be chairs of SCMs. Would politicize. Need guidelines for this. Journalist, Cohort 2 Provincial-central level [I.31.PC] LGs have a right to retain RPs, but they will have to spend their own money to do so and it is not clear many will be willing to do so. GON, province level [I.32.PC] Education sector experienced a setback due to federalization. The local government is given the authority to look after education through grade 12. The Confederation of Nepali Teachers strongly opposes this decision of the central government. The local governments are incapable of taking this responsibility. They have no knowledge and ability to look after educational matters. Teachers’ union, central level. [I.33.PC] Until recently, about 50% of school [have such serious problems that no intervention at the level of intensity that government can provide is likely to make a significant difference in learning]. But this is where palikas can make a difference. Now can’t hide. But until monitoring/mentoring mechanism is fixed, the palikas can’t make a big difference. NGO, central level. [I.34.PC] Some palika-level officials have independently formulated Local Education Act, including proposal to make ward Chairman the chair of SMC. EGRP staff, central level [I.35.PC] LG currently has control over 40-60% of curriculum but no ideas what to do with it. Still under discussion. GON, province [I.36.PC] As of April 2019, overall 30% of LEUs are vacant. EGRP, central level [I.37.PC] Mayors made campaign promises about education before we could orient them to EGR. GON, Central level. [I.38.PC] Some LGs are hiring people who are jobless but important to the political party. There is simply not enough readiness to fully engage in EGRP by most teachers. GON, central level 179 [I.39.PC] Recently, it took me 2 full days to send an invitation to the palika officials. There are no clear directives as to how to go about it. The reporting channel is also not clear. GON, central level. [I.40.PC] In Dhulikhel nagarpalika in Kabhre, there are 35 schools. In Roshi gaupalika in Kabhre again there are 60 schools. But because Roshi is a gaupalika, we are only providing section officer even if the number of schools under his portfolio is higher. And, because Dhulikhel is a nagarpalika, we are providing an under-secretary to head the LEU even if the number of schools is lower than that of a gaupalika. We need to really figure this out as soon as possible. GON, central level [I.41.PC] Last time, we wanted to invite the education officer from Dhulikhel nagarpalika, but we had to send the letter first to MOEST, then to MOFAGA, to PM office to Province 3 office and then they sent the letter finally to Dhulikhel. It almost took us 1-2 days. This is not practical. We have worked on the basis of a phone call in the past, now this process takes so much time. Similarly, the reporting mechanism from local to federal is lacking. Yes, there are people who we know personally, and they send us reports. But some local governments have started saying that we cannot ask them for reports. GON, central level [I.42.PC] At the provincial and LG levels, [he has? they have?] no idea how to develop curriculum. Could perhaps give authority/empower them to do more supplemental materials. GON, central level [I.43.PC] The number of ETCs is reduced to seven. The new structure is going to negatively affect training activities. The number of trainers is already small. These trainees are not able to meet the needs of the palikas. GON, central level 180 EXCERPTS RELEVANT TO QUESTION 5 J. NEGRP Minimum Package and Scaling Up [J.I.PC] The efforts made by NGOs are encouraging. But we have not been able to map their initiatives. It is very important to track/map their work and set the minimum standards/benchmarks for them, such as in terms of teacher training standards, materials, classroom activities among others. We, as the managers, have not been able to create those benchmarks. We have asked the RTI TA team to do this. Our main intention is to avoid duplication. GON, Central level [J.2.PC] NEGRP Technical Working Group provides good experience for sharing experiences. TWG also helped with Minimum Package. [J.3.PC] GON’s approach to the Minimum Package was largely guided by what GON hoped USAID would provide. May not reflect most pragmatic model for Nepal. Also, Minimum package requires complete coverage of palika; too much for most NGOs. Donor partner, central level. [J.4.PC] NEGRP Minimum Package had lots of input from many stakeholders, but it is all about inputs, not process. EGRP, Central level [J.5.PC] Minimum Package is OK but some problems for NGOs: NCED insisted on 10 days of training but most NGOs and their teachers can afford more like 5 days. NCED said it would treat 5 days as refresher only. GON will not share/allow NGOs to purchase? EGRP SRM (CDC) even though student practice book makes explicit reference to books in the EGRP SRM. Costs NRP 25,000 to obtain 2-4 copies of 153 CDC-approved ASR. Talked to Jannie. ERO has turned EGRA into something standardized and will not share. NGO, Central level [J.6.PC] Still a lot of NGOs who don’t have enough resources to carry out Minimum Package. And the private schools are left out. NGO, central level 181 J.7.PC Four new districts for EGRP/NEGRP Under SSDP, we have agreed that we will roll out to all 77 districts by the year 2020. We are preparing for 4 additional districts in the upcoming academic year. GON Central level So far, no report on GON’s efforts to expand on its own into 4 districts and how it compares with the way RTI has been implementing EGRP in 16 pilot districts. Donor partner, central level Last week there was NEGRP review workshop – we were told that the budget has been released for 4 additional districts. But there was a doubt that the budget may not be sufficient. GON, Central level In four new government-led districts, will train teachers in April. EGRP central level We need technical assistance if we are to expand it to 4 additional districts. GON, Central level Local agencies are given the responsibility of implementing NEGRP. But they are not ready for it yet. These agencies need short-term training to enhance their capacity. GON, central level District selection of GON’s 4 new districts in 2018-19 paid no attention to Minimum Package. MOEST put District X on list, but under Minimum Package, an NGO had already proposed to do more than 100 schools in that district. NGO, Central level [J.8.PC] Need to vary model by geographic area. Have to plan differently for hills versus Terai. GON, non-EGRP district [J.9.PC] Need to develop EGR readiness measures for districts and palikas and start with those that are ready. Cover ECD-Class 5, not just Class 1-3. NGO, central level [J.10.PC] Until now all planning for EGRP has been at CLA level, leaving the implementation headaches to provincial and local level. Can’t have uniform approach across all geographic areas. Many unique issues at the palika level. GON, central level [J.11.PC] The whole EGRP is linked with the governance of the sector. The civil servants re-adjustment/reshuffling process has come up as an important issue. The practical issue is that we cannot keep one staff in one position for years and that has [negatively] affected the ownership and accountability. I also feel that we may have been ambitious in terms of setting policy benchmarks. We probably were not ready in terms of available resources and capacity of the ground level agencies. GON, Central level [J.12.PC] I think the best approach is to increase the number of NEGRP districts slowly given our resources and capacity. GON, central level [J.13.PC] NEGRP should expand even if model not “ready” or thoroughly tested because people learn by doing, not by preparing a model for them and asking them to implement it. Now LGs are implementing SSDP. CLA and experts must find a way to build capacity while implementing. By doing it, they will learn. Wants to expand EGRP to nine more districts next year and nine the following. Donor partner, central level