DISCLAIMER: The views of the expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government USAID/INDIA EX-POST PERFORMANCE EVALUATION NURTURING EARLY LITERACY PROJECT (NELP) Final Report Contract No. # 72038619D00001 Task Order No. # 72038620F00003 June 8, 2023 USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | i ABSTRACT NELP aimed to build foundations of emergent and early literacy competencies in more than 93,000 students in selected blocks of Rajasthan, Maharashtra, and Karnataka by 1) Equipping teachers with knowledge and skills, 2) Ensuring access to age-appropriate, quality children’s literature through classroom/community libraries and 3) Demonstrating a viable model to improve foundational literacy skills in elementary school children and emergent literacy skills in 5+ preschoolers. The evaluation found extensive evidence of sustained progress across these objectives where 1) NELP components are being sustained and scaled beyond original intervention areas and 2) NELP influenced policy and other decision-making in regional education offices. Given this, and NELP's success during its funded period of performance, there is a higher probability of a significant return on the original USAID investment. ii | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV USAID/INDIA EX-POST PERFORMANCE EVALUATION NURTURING EARLY LITERACY PROJECT (NELP) Final Report Contract No. # 72038619D00001 Task Order No. # 72038620F00003 June 8, 2023 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development of the United States Government. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | iii TABLE OF CONTENTS ABSTRACT ............................................................................................................................... I ACRONYMS...........................................................................................................................VI EXECUTIVE SUMMARY...................................................................................................... VII BACKGROUND ..................................................................................................................................................... VII EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS............................................. VII FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ...................................................................VIII EQ1. TO WHAT EXTENT AND HOW DID THE ACTIVITIES ACCOMPLISH THEIR OBJECTIVES DURING IMPLEMENTATION?..........................................................................................................................VIII FINDINGS.......................................................................................................................................................VIII CONCLUSIONS .............................................................................................................................................IX RECOMMENDATIONS................................................................................................................................. IX EQ2. HOW HAS THE PROJECT IMPROVED THE STUDENTS’ LEARNING IN TERMS OF BASIC READING SKILLS?.................................................................................................................................................... X FINDINGS.......................................................................................................................................................... X CONCLUSIONS .............................................................................................................................................. X RECOMMENDATIONS.................................................................................................................................. X EQ3. TO WHAT EXTENT HAS THE PROMOTED BEHAVIOR AT SCHOOL AND SYSTEM LEVEL CONTRIBUTED TO THE EARLY LITERACY COMPETENCIES AMONG THE TARGET GROUPS? IF SO, HOW? IF NOT, WHAT ARE THE FACTORS THAT HAVE HINDERED THE PRACTICE TO CONTINUE? .............................................................................................................................................................. X FINDINGS.......................................................................................................................................................... X CONCLUSIONS .............................................................................................................................................. X RECOMMENDATIONS.................................................................................................................................. X EQ4. HAVE THE IMPLEMENTING PARTNERS/STAKEHOLDERS STRENGTHENED BY USAID FUNDING CONTINUED TO IMPLEMENT THE ACTION? HOW ARE THE SCHOOLS, COMMUNITIES AND OFFICIALS CONTINUING TO MAINTAIN THE EARLY LITERACY SYSTEM IMPROVED BY USAID FUNDING? WHAT IS THE BENEFICIARIES’ PERSPECTIVE / FEEDBACK ABOUT THE IMPACT OF THE PROJECT?...............................................................................XI FINDINGS.........................................................................................................................................................XI CONCLUSIONS .............................................................................................................................................XI RECOMMENDATIONS................................................................................................................................XII EQ5. WHAT INTERVENTIONS WERE MORE SUCCESSFUL AND/OR HAD A GREATER CONTRIBUTION TO ACCOMPLISH THE OBJECTIVES? .........................................................................XII FINDINGS........................................................................................................................................................XII CONCLUSIONS ............................................................................................................................................XII RECOMMENDATIONS................................................................................................................................XII EQ6. WHAT ARE THE FACTORS THAT CONTRIBUTED TO OR IMPAIRED THE LONG-TERM SUSTAINABILITY OF THE ACTIVITIES’ OUTCOMES AND OUTPUTS?.............................................XIII FINDINGS.......................................................................................................................................................XIII CONCLUSIONS ...........................................................................................................................................XIII RECOMMENDATIONS...............................................................................................................................XIII EVALUATION PURPOSE AND EVALUATION QUESTIONS ......................................... 1 iv | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV EVALUATION PURPOSE ........................................................................................................................................ 1 EVALUATION QUESTIONS................................................................................................................................... 1 BACKGROUND....................................................................................................................... 1 PROJECT BACKGROUND..................................................................................................................................... 1 EVALUATION METHODOLOGY AND LIMITATIONS.................................................... 4 SAMPLING................................................................................................................................................................... 4 DATA COLLECTION............................................................................................................................................... 4 DATA ANALYSIS....................................................................................................................................................... 5 COLLABORATION, LEARNING, AND ADAPTION ..................................................................................... 5 LIMITATIONS ............................................................................................................................................................. 6 THIRD-PARTY STUDY REFERENCES ................................................................................................................. 6 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ............................................ 8 EQ1. TO WHAT EXTENT AND HOW DID THE ACTIVITIES ACCOMPLISH THEIR OBJECTIVES DURING IMPLEMENTATION?.............................................................................................................................. 8 FINDINGS........................................................................................................................................................... 8 TABLE 1: NELP IMPLEMENTATION PARTNERS.................................................................................... 8 TABLE 2: NELP FINAL INDICATOR REPORTING................................................................................. 9 FIGURE 1: NELP INTERVENTION AREAS..............................................................................................11 FIGURE 2: SCHOOLS AND ANGANWADIS REACHED BY NELP PER YEAR (EGRA STUDY) ..............................................................................................................................................................................12 CONCLUSIONS .............................................................................................................................................13 RECOMMENDATIONS................................................................................................................................. 13 EQ2. HOW HAS THE PROJECT IMPROVED THE STUDENTS’ LEARNING IN TERMS OF BASIC READING SKILLS?...................................................................................................................................................14 FINDINGS.........................................................................................................................................................14 FIGURE 3: AWC PHYSICAL ENVIRONMENT (AW STUDY)...........................................................17 FIGURE 4: COMPREHENSION RESULTS (EGRA STUDY)................................................................. 18 FIGURE 5: READING FLUENCY (EGRA STUDY).................................................................................18 CONCLUSIONS .............................................................................................................................................19 RECOMMENDATIONS................................................................................................................................. 19 EQ3. TO WHAT EXTENT HAS THE PROMOTED BEHAVIOR AT SCHOOL AND SYSTEM LEVEL CONTRIBUTED TO THE EARLY LITERACY COMPETENCIES AMONG THE TARGET GROUPS? IF SO, HOW? IF NOT, WHAT ARE THE FACTORS THAT HAVE HINDERED THE PRACTICE TO CONTINUE? .............................................................................................................................................................19 FINDINGS.........................................................................................................................................................19 FIGURE 6: BEHAVIOR-FOCUSED LIBRARY TRAINING....................................................................20 FIGURE 7: SOCIAL ENVIRONMENT (AW STUDY)............................................................................21 FIGURE 8: SELF-REGULATION (AW STUDY) ......................................................................................22 FIGURE 9: ORAL LANGUAGE (AW STUDY)........................................................................................22 FIGURE 10: LISTENING SKILLS (AW STUDY) ......................................................................................23 FIGURE 11: READING (AW STUDY).......................................................................................................24 CONCLUSIONS .............................................................................................................................................24 USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | v RECOMMENDATIONS................................................................................................................................. 25 EQ4. HAVE THE IMPLEMENTING PARTNERS/STAKEHOLDERS STRENGTHENED BY USAID FUNDING CONTINUED TO IMPLEMENT THE ACTION?.......................................................................25 FINDINGS.........................................................................................................................................................25 FIGURE 12: RAJASTHAN MOU KPI TARGETS .....................................................................................26 TABLE 3: MOU PROJECT -RSLPP PROGRESS UPDATES: PHASE-I (1/2).......................................28 FIGURE 13: MOU PROJECT - OUTCOMES (EGRA STUDY)............................................................29 CONCLUSIONS .............................................................................................................................................30 RECOMMENDATIONS................................................................................................................................. 30 EQ 5. WHAT INTERVENTIONS WERE MORE SUCCESSFUL AND/OR HAD A GREATER CONTRIBUTION TO ACCOMPLISH THE OBJECTIVES? ..........................................................................30 FINDINGS.........................................................................................................................................................30 FIGURE 14: SCALE UP PATHWAY ...........................................................................................................30 CONCLUSIONS .............................................................................................................................................32 RECOMMENDATIONS................................................................................................................................. 32 EQ 6. WHAT ARE THE FACTORS THAT CONTRIBUTED TO OR IMPAIRED THE LONG-TERM SUSTAINABILITY OF THE ACTIVITIES’ OUTCOMES AND OUTPUTS?...............................................33 FINDINGS.........................................................................................................................................................33 CONCLUSIONS .............................................................................................................................................33 RECOMMENDATIONS................................................................................................................................. 34 ANNEXES............................................................................................................................... 36 ANNEX I: EVALUATION STATEMENT OF WORK.....................................................................................36 ANNEX II: EVALUATION INCEPTION REPORT..........................................................................................40 INTRODUCTION ..........................................................................................................................................40 OVERVIEW.......................................................................................................................................................40 EVALUATION PURPOSE ......................................................................................................................................40 EVALUATION QUESTIONS................................................................................................................................. 40 PROJECT BACKGROUND..........................................................................................................................41 LITERATURE REVIEW...................................................................................................................................42 METHODOLOGY ..........................................................................................................................................42 EVALUATION DESIGN MATRIX...............................................................................................................44 SAMPLING PLAN............................................................................................................................................46 DATA COLLECTION METHODS .............................................................................................................47 DATA ANALYSIS METHODS .....................................................................................................................50 GUIDING PRINCIPLES..................................................................................................................................50 LIMITATIONS AND POTENTIAL BIASES ...............................................................................................50 RISK MANAGEMENT....................................................................................................................................51 ETHICAL PROTECTION AND DATA MANAGEMENT PLAN........................................................51 WORK PLAN...................................................................................................................................................52 TEAM COMPOSITION ..........................................................................................................................................52 EVALUATION TIMEFRAME..................................................................................................................................52 ANNEX III: SOURCES OF INFORMATION ....................................................................................................59 ANNEX IV: DISCLOUSRE OF ANY CONFLICTS OF INTEREST .............................................................61 ANNEX V: EVALUATION TEAM MEMBERS..................................................................................................61 vi | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV ACRONYMS ACDPO Assistant Child Development Project Officer ADP Aspirational District Programme ADS Automated Directive Systems AWC Anganwadis CDPO Child Development Project Officer CmF Centre for microFinance DPSC District Project Steering Committee ECE Early Childhood Education EGRA Early Grade Reading Assessment ET Evaluation Team EQ Evaluation Question FGD Focus Group Discussion IPP India Partnership Program KII Key Informant Interview KPI Key Performance Indicator NCE No-Cost Extension NEP National Education Policy 2020 NELP National Education and Literacy Program PEEO Panchayat Elementary Education Officer PSS Pragat Shikshan Sanstha RtR Room to Read RSLPP Rajasthan School Library Promotion Project SMC School Management Committee SOW Scope of Work SRG State Resource Group TLM Teacher Learning Materials USAID U.S. Agency for International Development USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | vii EXECUTIVE SUMMARY BACKGROUND NELP was approved under USAID’s India Partnership Program (IPP), which focuses on co-funded interventions to promote early-grade reading. The Centre for microFinance (CmF) is responsible for the overall implementation of the project. Field-level support was provided by Kalike, Bangalore, another Associate Organization of the Tata Trusts and three non-profit partners, namely, Pragat Shikshan Sanstha (PSS), Phaltan, Maharashtra, Room to Read India Trust, New Delhi and Bodh Shiksha Samiti, Jaipur. NELP aimed to build foundations of emergent and early literacy competencies in more than 93,000 students in selected blocks of Rajasthan, Maharashtra, and Karnataka. Three general objectives contributed to this goal: 1. Equip teachers with knowledge and skills to improve classroom literacy instruction by moving from rote learning to meaningful engagement with print. 2. Ensure access to age-appropriate, quality children’s literature through classroom/community libraries. 3. Demonstrate a viable model to improve foundational literacy skills in elementary school children and emergent literacy skills in 5+ preschoolers. During the no-cost extension (NCE) period from April 2020 to June 2021, CmF directly implemented the field-level implementation of the following interventions: • Early Childhood Education (ECE)- CmF scaled up the ECE intervention with block-level saturation in Abu Road and Pindwara block of Sirohi district covering 433 Anganwadi centers. • Rajasthan School Library Promotion Project (RSLPP)- In Rajasthan state, CmF signed an MoU with the Government of Rajasthan for strengthening school libraries across all 33 districts in the Rajasthan State covering 3300 Government Schools along with the set-up of 34 district-level model libraries, i.e., one model library in each district and one in RSCERT Udaipur. EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS This end-of-project performance evaluation was designed and implemented to answer the following questions using the best evidence available: 1. To what extent and how did the activities accomplish their objectives during implementation? 2. How has the project improved the students’ learning in terms of basic reading skills? 3. To what extent the promoted behavior at school and system level has contributed to the early literacy competencies among the target groups? If so, how? If not, what are the factors that have hindered the practice to continue? 4. Have the implementing partners/stakeholders strengthened by USAID funding continued to implement the action? How are the schools, communities and officials continuing to viii | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV maintain the early literacy system improved by USAID funding? What is the beneficiaries’ perspective / feedback about the impact of the project? 5. What interventions were more successful and/or had a greater contribution to accomplish the objectives? 6. What are the factors that contributed to or impaired the long-term sustainability of the activities’ outcomes and outputs? FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The evaluation team has summarized the results for each evaluation question below. When examined holistically, the results show positive NELP outcomes achieved through effective adaptive management, evidence-driven technical approaches, and high-level expertise by the implementing partners. The significance of NELP performance increases considering the barriers encountered by the COVID pandemic and the complexity of evolving national policies and regional dynamics. The evaluation found extensive evidence of progress beyond original NELP programing where 1) NELP components are being sustained and expanded in original intervention areas, 2) NELP components are being replicated/duplicated in other geographic areas, and 3) NELP influenced policy and other decision-making in regional education offices. Given NELP's performance against its Key Performance Indicators and progress in multiple avenues of scaling beyond its initial programming, there is a higher probability of a positive return on the USAID investment. EQ1. TO WHAT EXTENT AND HOW DID THE ACTIVITIES ACCOMPLISH THEIR OBJECTIVES DURING IMPLEMENTATION? FINDINGS NELP could not achieve all of its Key Performance Indicators (KPIs) due to COVID but surpassed expectations in its progress after program completion. NELP contributed to a culture shift from a context in which libraries and digital learning materials were devoted to younger students in building the necessary literacy skills to facilitating other learning outcomes in math, science, etc., as they get older, NELP’s cumulative success, stemming from its original USAID funded activities as well as its scaled performance post-funding, cuts across four areas as documented in its progress reports. 1) NELP Capacity Development Success: • Trained 11,400+ Teachers, Headteachers, and PEEOs on Literacy, Library, and Numeracy (target- 3,460 teachers) • Trained 1500+ AW Workers, AW Helpers, and AW Supervisors (target- 1,034 AWWs & AWHs) 2) NELP Student Learning Success: • 1,47,000+ unique children reached • 46% project children achieved reading and comprehension against 19.8% in control (target- 60%) USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | ix 3) NELP Resource Development Success: • 60 multi-lingual resources (multilingual stories, storybooks, conversation charts, etc.) • 13 Supplementary readers (PSS, Maharashtra) • 25 songs & rhymes under ECE (Kalike) • 10 Animated story videos with SLS (eLibrary) • TLM kits of ECE and Literacy • Instruction manuals, handbooks, and training modules 4) NELP Community Engagement Success: • 450+ SMCs/SDMCs strengthened (4,000+ SMC members) • 35,000+ Parents & Caregivers were reached through meetings, home visits, and community events. CONCLUSIONS 1) NELP facilitated an enabling environment for literacy by introducing age-appropriate learning materials into a library environment managed by an evidence-driven pedagogy. It facilitated a culture shift that supports literacy by using model libraries to create an enabling ecosystem demonstrating sustainable progress after the completion of the NELP. 2) There is overachievement in the capacity development of Teachers, Headteachers, PEEOs AW Workers, AW Helpers, and AW Supervisors. 3) NELP ECE performance was driven by evidence-supported pedagogy, which enabled a culture shift from rote (memorization) learning to meaningful learning for the student. RECOMMENDATIONS USAID Convening Authority: USAID could use its convening abilities to discuss NELP successes with Indian government stakeholders to identify entry points for continued support. GOI and USAID Actions to Mitigate Negative COVID Effects: The degree of disruption in ECE caused by COVID is hard to understate. There could be more opportunities to mitigate the adverse effects on literacy from COVID, including finding resources for NELP activities that were halted due to COVID, especially now that NELP has produced substantial learning from its internal performance management. GOI Reforms for Teacher Transition: Frequent schoolteacher transfers were a barrier to sustaining the impact of NELP. Introducing capacity development activities for teachers in their new schools was not part of NELP. Still, it could be included in future programming to extend the exposure to literacy activities and cultivate champions in new schools. x | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV EQ2. HOW HAS THE PROJECT IMPROVED THE STUDENTS’ LEARNING IN TERMS OF BASIC READING SKILLS? FINDINGS NELP contributed to improved student literacy outcomes. Third-party EGRA studies showed significant improvement in children's learning levels. Students in NELP schools demonstrated improved reading fluency and comprehension of grade-level text at the end of grade 2. The evaluation used observational checklists assessing these dimensions at each school and Anganwadi site and found them satisfactory in all locations. CONCLUSIONS The behavior change approach taken by NELP contributed to its overall success. Statistical evidence for the behavior change for this comes from quantified pre-post third-party studies that, while not having a pure statistical counterfactual, are still very relevant. This ex-post evaluation confirmed that all envisioned behavior changes were present in the schools and AWCs sampled. RECOMMENDATIONS GOI and USAID Support for Listening, Observation, and Community Outreach: Listening and Observation activities that capitalize on the students' improved access to reading materials could be included in future programing and policies. Likewise, given the success of the children’s reading clubs, this low-cost effort has much potential to engage the community in recognizing the importance of literacy and cultivate them as stakeholders in improving students reading. EQ3. TO WHAT EXTENT HAS THE PROMOTED BEHAVIOR AT SCHOOL AND SYSTEM LEVEL CONTRIBUTED TO THE EARLY LITERACY COMPETENCIES AMONG THE TARGET GROUPS? IF SO, HOW? IF NOT, WHAT ARE THE FACTORS THAT HAVE HINDERED THE PRACTICE TO CONTINUE? FINDINGS NELP focused on behavior change at all levels (from students to administrators, etc.) and used numerous tools to achieve the desired behavior (from capacity development to policy reform). NELP is still progressing, with some components being scaled by different actors and/or influencing improved policies and practices. Given NELP’s success promoting envisioned Pre-Primary behavior change, NELP’s success in AWCs was widely credited as being transformative for subsequent student learning outcomes. CONCLUSIONS NELP’s behavior-focused approach had measurable positive effects in 1.) preparing children in AWCs to become students in early primary education, 2.) primary education students to excel across school subjects, 3.) community members to engage in literacy promotion, 4.) teachers to use best practices in doing so, and 5.) policymakers to create optimal enabling conditions. RECOMMENDATIONS USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | xi USAID Support for GOI Expanding the Enabling Environment: Capacity building of block￾level education officials, cross-pollination/dissemination of learnings, and adaptation of best practices among the consortium members and schools could improve performance and additional scaling opportunities. GOI Improvements for Inclusivity: Audio books, brail books, and other materials for differently￾abled students could be introduced to improve inclusivity and help marginalized students to develop the same behaviors as their peers. EQ4. HAVE THE IMPLEMENTING PARTNERS/STAKEHOLDERS STRENGTHENED BY USAID FUNDING CONTINUED TO IMPLEMENT THE ACTION? HOW ARE THE SCHOOLS, COMMUNITIES AND OFFICIALS CONTINUING TO MAINTAIN THE EARLY LITERACY SYSTEM IMPROVED BY USAID FUNDING? WHAT IS THE BENEFICIARIES’ PERSPECTIVE / FEEDBACK ABOUT THE IMPACT OF THE PROJECT? FINDINGS NELP’s success in achieving scale across states through partnerships with government offices, communities, and other stakeholders after program completion is perhaps its most substantial achievement. Karnataka: Kalike partnered with district administration under the Aspirational District Programme (ADP) to expand the early literacy-numeracy work to 13300+ AWCs of Kalyan Karnataka region (consisting of seven districts of north Karnataka). Maharastra : ECE activities in Maharashtra were awarded a citation by the Chief Executive Officer of Satara Zilla Parishad for high-quality work in 150 project schools. The implementing partner in Maharashtra, PSS, received a request letter from the CEO (Satara district) to scale-up the early literacy interventions in the district. Rajasthan: Before NELP, even if some schools had had a library, they were not meant for primary grade students. Through NELP, students received age-appropriate, gender-sensitive, and relevant literature. The state’s academic calendar now consists of specific time slots in the library for primary-grade children in Rajasthan. CmF and the Government of Rajasthan have signed an MoU for scaling up NELP activities, especially those related to reading promotion through school libraries, in which CmF is providing technical and monitoring support to RSCERT and DIETs. CmF also partnered with district administration under the Aspirational District Programme (ADP) to expand the ECCE and FLN work to 1450+ AWCs and 800+ schools of Sirohi and Karauli districts of Rajasthan. CONCLUSIONS The evaluation found that NELP surpassed expectations through its IPs collaborating with government officials in their regions and motivating culture shifts through the positive influence of NELP activities. This positive influence has led numerous individual schools, districts and even entire states to adopt NELP practices and lessons for specific classroom activities and reform education policy. The most significant contributing factor to this success could be the performance management of NELP by the IPs, where NELP results and lessons learned were communicated xii | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV through IP professional networks, which, along with the strong reputation of the IPs, created confidence in the NELP model by government offices, schools, etc. RECOMMENDATIONS USAID CLA Driven Planning for Scale: NELP never intended to scale in the way and to the degree it has. The scaling is primarily due to the innovations and commitment of CmF and other NELP implementers. The positive effects of these efforts could be enhanced with better initial planning for scaling during co-creation processes in future programs. Accounting for scaling objectives during program design in a way that invites participation from those actors (government, school administrators, etc.) that are critical to scaling success is a best practice in achieving progress after programs. EQ5. WHAT INTERVENTIONS WERE MORE SUCCESSFUL AND/OR HAD A GREATER CONTRIBUTION TO ACCOMPLISH THE OBJECTIVES? FINDINGS The evaluation found the Model Library component to be highly successful, with higher probabilities of continued successful scaling compared to similar early pre-primary literacy interventions. CONCLUSIONS • NELP cultivated a culture shift the enabled improved literacy outcomes by advocating for the effective use of libraries, Anganwadi reform and Activity Based Learning. • The Model Library is more likely to continue scaling given specific enabling conditions, but the E￾Library also has a substantial opportunity for scaling, assuming certain factors like government support. RECOMMENDATIONS GOI and USAID Support to Build on Scaled Success: The Model Library has demonstrated success and lessons learned that could be very cost-effective to scale through reasonable funding, communications and other support by donors and Indian government officials. The critical data for informing future scaling could reside in the institutional knowledge of the NELP IP’s who could be the formative factor in informing future scaling efforts. GOI Consideration and USAID Support of Further Testing of E-Tablets: The E-tablet pilot was very successful for its size. The end of donor funding and a government emphasis on desk top computers in schools has created barriers to further testing this innovation. Given the pilot's success, coordinated efforts to further test and iterate the pilot could provide the success in literacy outcomes seen in NELP and provide the knowledge necessary to determine how best to use tablets in schools for learning outcomes in general. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | xiii EQ6. WHAT ARE THE FACTORS THAT CONTRIBUTED TO OR IMPAIRED THE LONG￾TERM SUSTAINABILITY OF THE ACTIVITIES’ OUTCOMES AND OUTPUTS? FINDINGS NELP had three probable contributing reasons for its success and sustainability: 1. Alignment with government policy, specifically on the transition from rote learning to meaningful engagement. 2. Skilled implementing partners who know the local context, actors, and culture, which facilitates strong performance management. 3. Evidence-driven approaches and focus on enabling environments. CONCLUSIONS • The success, including the post-program progress of NELP, was primarily a result of having the right partners (IPs) in the right places at the right time. Out of these three factors, having the correct IPs is possibly the most significant contributor to success, given that, in many ways, NELP IPs used their capacity to maximize the potential of being in the right place at the right time. • NELP’s work in the AWCs transformed how children spend their time in these centers. Before, these centers were seen as childcare and meal service facilities. Now, they prepare children to become effective students by cultivating their cognitive and behavioral skills to excel in and out of school. This success motivated government officials to support the initiative through increased resources and enabling policies. RECOMMENDATIONS FOR DONORS: • Projects that emphasize shifting from rote learning to learning with comprehension/meaning￾making could have more extended performance periods for the 3 to 9 years age group. • USAID could use its convening ability to disseminate NELP's lessons learned and success through broader publications, events, etc., to ensure that its success is built on in India and beyond. FOR IMPLEMENTING PARTNERS: • District and block-level officials could be included more in future project training/capacity-building plans. • Training components could also be co-created/designed with teachers and officials to increase ownership and effectiveness. One training approach (like activity-based learning) may not suit all the participating teachers; some may prefer reflective or Edtech-based sessions. FOR POLICYMAKERS: • AWCs could be resourced and managed as a ‘preschool’ with forward linkages to school as opposed to a daycare service with meal options, and could include a uniform report card could be introduced at the AWCs. Mother’s education is equally critical to achieving holistic development of the children. Including mothers in literacy promotion activities could improve literacy-friendly behaviors in the household. For language promotion activities, listening and speaking could also be emphasized along with reading xiv | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV and writing to advance the cycle of language learning. These activities could also be extended to grade three. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE This Ex-Post Performance Evaluation answers the below evaluation questions and provides USAID with an end-of-project performance summary of the Nurturing Early and Literacy Project (NELP). This Ex-Post evaluation is heavily qualitative and unsuitable for any quantitative cost-benefit analysis, but its results have substantial formative potential for future USAID decision-making. EVALUATION QUESTIONS USAID posed six evaluation questions in the evaluation scope of work (SOW): 1. To what extent and how did the activities accomplish their objectives during implementation? 2. How has the project improved the students’ learning in terms of basic reading skills? 3. To what extent the promoted behavior at school and system level has contributed to the early literacy competencies among the target groups? If so, how? If not, what are the factors that have hindered the practice to continue? 4. Have the implementing partners/stakeholders strengthened by USAID funding continued to implement the action? How are the schools, communities and officials continuing to maintain the early literacy system improved by USAID funding? What is the beneficiaries’ perspective / feedback about the impact of the project? 5. What interventions were more successful and/or had a greater contribution to accomplish the objectives? 6. What are the factors that contributed to or impaired the long-term sustainability of the activities’ outcomes and outputs? BACKGROUND PROJECT BACKGROUND NELP was approved under USAID’s India Partnership Program (IPP), which focuses on co-funded interventions to promote early-grade reading. Centre for Microfinance is overall responsible for the implementation of the project. Field-level support was provided by Kalike, Bangalore, another Associate Organization of the Tata Trusts and three non-profit partners, namely, Pragat Shikshan Sanstha (PSS), Phaltan, Maharashtra, Room to Read India Trust, New Delhi and Bodh Shiksha Samiti, Jaipur. NELP had an original period of performance from October 1, 2015, to September 30th, 2019. USAID granted a nine-month extension due to delays introduced by the COVID pandemic that extended NELP to June 29, 2021. However, the nine-month extension only focused on the scale-up Library component with the partnership of the Rajasthan State Government and was implemented by CmF. All Sub-partner activities outside the Model Library activity in Rajasthan closed in March 2020. 2 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV Activities in six core areas contributed to these objectives. 1. Capacity Development- Building the capacity of key stakeholders to include: a. Government schoolteachers, primary and upper primary school teachers. b. Anganwadi workers/ facilitators. c. Community workers and other staff members supporting intervention school sites. d. Administrators and officials at the state/district/block level. 2. Literacy Foundation- Improve student learning by developing a solid foundation for literacy and reading development over four years through: a. Explicit literacy instruction to build foundational literacy skills in school languages. b. Remedial teaching for at-risk pupil studying in primary & upper primary grades. c. Establishing a structured print-rich environment. 3. Teaching-Learning Materials- Development of contextual and relevant teaching￾learning materials for students and teachers by: a. Implementing the first of its kind pilot on tablet-based access to children's literature in multiple formats for reading promotion. b. Teacher training manuals to assist the transition from rote learning. 4. School Libraries and Village Libraries- Setting up well-functioning libraries in schools and villages by: a. Developing student library committees. b. Organizing community libraries with School Management Committee (SMC) input. 5. School Governance- Strengthening of school governance structures through: USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 3 a. Community mobilization around learning outcomes and community libraries in specific blocks b. Working with SMCs on monitoring progress and barriers toward learning outcomes and the transition from rote to meaning-making. 6. Learning- Document and disseminate learnings from the project by: a. Engaging communities around barriers and progress toward key learning objectives. b. Sharing evidence on what works through teacher and school management workshops and other learning events. During the no-cost extension (NCE) period from April 2020 to June 2021, CmF was the sole Implementer in field-level implementation of the following interventions- • Childhood Education (ECE)- CmF scaled up the ECE intervention with block-level saturation in Abu Road and Pindwara block of Sirohi district covering 433 Anganwadi centers. • Rajasthan School Library Promotion Project (RSLPP)- In Rajasthan state, CmF signed an MoU with the Government of Rajasthan for strengthening school libraries across all 33 districts in the Rajasthan State covering 3300 Government Schools along with the set-up of 34 district-level model libraries, i.e., one model library in each district and one in RSCERT Udaipur. The scaled-up intervention project focused on strengthening school libraries with infrastructure, children’s literature, trained teachers, child-friendly and enriched learning environments across the state. The project aligned with the National Education Policy (NEP 2020), which prioritizes attaining foundational literacy and numeracy for all children in early grades and expanding school libraries to build a culture of reading nationwide. 4 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV EVALUATION METHODOLOGY AND LIMITATIONS SAMPLING Given NELP ended approximately 18 months ago for CmF activities and almost 34 months for the remaining sub-partner activities, there were significant barriers in identifying direct NELP recipients (teachers, SMC members, etc.) who can be respondents for data collection. A significant risk of recall bias compounds this attrition (further elaborated in the Limitations section below). Because of these two risks, after consultations with USAID and CmF, the evaluation team decided to 1) focus primary data collection on Rajasthan and Karnataka (with a specific focus on ECE and RSLPP) and 2) collect no primary data from students. Focusing data collection in Rajasthan and Karnataka allowed for easier identification of respondents as various blocks there are continuing specific NELP components. The evaluation team and USAID decided to remove students from data collection, given the difficulty in identifying student recipients and the high potential for recall bias given their age. Students did not receive NELP interventions (perhaps outside of some learning materials like books); the direct recipients were the teachers and administrators whose capacity was developed, etc. Hence engaging indirect beneficiaries introduces additional risk when asking about direct assistance, especially 18 months post-completion. DATA COLLECTION While this was a mixed methods evaluation, the primary data collection was qualitative, consisting of: • Document Review (to include a simple descriptive analysis of KPIs and third-party studies) • Key Informant Interviews (KII’s) • Focus Group Discussions (FGDs) • School Site Observations • Pause and Reflect Sessions The Pause and Reflect sessions consisted of semi-structured discussions with CmF on 1) data collection and 2) results interpretation. Both were of high value to the evaluation. These sessions are expanded in the CLA section below, but both helped validate results and expand on their meaning with the context-specific knowledge of CmF. The Evaluation Team (ET) took steps to mitigate any bias introduced by these pause and reflect sessions, including verifying observations with empirical data. The evaluation team designed the evaluations’ semi-structured key informant interviews (KIIs) and focus group discussions (FGDs) with a mix of closed and open-ended questions. This approach allows for more flexibility in probing for more direct and/or accurate responses (as mitigation for recall bias) and exploring emerging issues (especially in schools where NELP components continue). The quantitative component of the evaluation primarily consisted of statistical analysis of NELP performance indicators. Full details on the dates, locations, data sources, and collection methods are included in Annex I. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 5 DATA ANALYSIS The evaluation team used indicator analysis from NELP reporting (primarily the AMELP) across all Evaluation Questions (EQs) but primarily for EQ #1 to report quantitative data for performance on NELP objectives. Likewise, the team used Qualitative Content Analysis for all EQs, particularly EQ2-EQ6. Content Analysis allowed emergent themes to be identified through iterative code book development using cross-checks. Because NELP was designed and implemented with the purpose of scaling, particularly the Model Library component, the team used specific scaling models to assess NELP performance data, specifically for EQ #4 and EQ #5. These scaling models, outlined below, provided analytical constructs with defined criteria to guide analysis, refine Conclusions and produce more actionable Recommendations. NELP did not use either of these models in its design or implementation, so a summative assessment of its performance against these models would not be appropriate. This is partly why the models are more helpful in developing Recommendations for future decision-making. These models include: • The Good Practices for Scaling Innovation1- This model was used to help chart the progress of NELP innovations, specifically the two components of the Library Model, along the different phases of its scale-up pathway. Each phase has its performance criteria against which NELP was assessed. • The Education Scalability Checklist and User Guide2- This education-specific model provides guiding questions meant to be used in pre-planning education interventions to inform the development of a performance management plan for the interventions scale-up pathway. The model was used to guide data collection and assess NELP performance against the criteria for each phase of the scale-up pathway. The quantitative analysis consisted of standard descriptive statistical calculations on NELP performance indicators. COLLABORATION, LEARNING, AND ADAPTION The evaluation was improved using CLA methods which we term Collaborative Data Collection. Collaborative Data Collection: The Centre for microFinance (CmF) accompanied the evaluation team (ET) on field data collection visits, providing the opportunity for formal and informal discussions immediately after data was collected. This collaboration gave the ET a greater contextual understanding of preliminary results and directly contributed to data collection and analysis improvements. For example, the initial evaluation design did not consider scaling as an analytical lens. However, during data collection, these Pause and Reflect sessions made it clear to the ET that NELP introduced innovations with clear evidence of progress on the scale-up pathway. 1 https://www.idiainnovation.org/resources/good-practices-for-scaling-innovation 2 https://www.brookings.edu/blog/education-plus-development/2021/02/24/planning-for-scale-the-education￾scalability-checklist/ 6 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV LIMITATIONS NELP worked in 467 schools, and the ET collected data in 17 (0.25% of the total) where there has been little to no turnover in teachers and school personnel. Given the sample size, this limited sample means the evaluation will not be able to answer questions on the sustainability of NELP objectives where there has been a turnover in school personnel or schools outside of Rajasthan and Karnataka. The evaluation can speak to NELP performance in all 467 schools (using program management and progress report data), but any evidence of outcomes will be limited to the 17 schools where the ET collected primary data and the inclusion of third-party statistical studies. NELP was implemented during COVID, the disruption of which is hard to understate. Partly due to this roadblock, NELP was not able to collect data on its primary outcome measure ES.1-1(Percent of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with USG assistance). Hence all outcome data is qualitative outside the two third-party studies outlined below. THIRD-PARTY STUDY REFERENCES This evaluation used two third-party studies, commissioned by CmF and Room to Read (RtR) included in the Document Review. Footnotes three and four contain hyperlink references to the studies for additional review. Both studies have limitations highlighted below, which should be considered when their results are referenced in this report (when appropriate, the report caveats these studies when their results are used). But even with these limitations, their results add valuable points of triangulation to provide a more holistic story of NELP performance. We include numerous graphics from these studies labeled as such and copied mainly from their original reports to avoid misrepresenting their work. These studies help address some of the limitations of this ex-post evaluation, which is largely qualitative and only includes one data collection round. These studies included more quantitative data using multiple rounds of data collection on key learning and behavior change outcomes in a way this ex-post evaluation could not. In short, including evidence from these two studies help answer the evaluation questions more thoroughly and in a way that provides USAID with better actionable data for its decision-making. • EGRA Study3: In 2015, Room to Read commissioned a two-year study of students enrolled in Chhattisgarh, Uttarakhand, and Rajasthan, including the Sirohi district. In July-August 2015, baseline data on reading skills were collected from 100 students starting Grade 1 across eight project schools and 111 students across eight comparison schools in Sirohi. The midline was collected after three years using the same sampling. The study used a version of the Early Grade Reading Assessment (EGRA) adapted to Hindi by local experts. • Limitations: Less documented statical sampling calculations are present than are needed to call this a valid statistical impact study. The standard limitations for non-randomized impact studies which rely on regression analysis are applicable. Given this, the ET considers this a comparison study and not an impact evaluation and caveats its results as such when referenced. 3 https://www.roomtoread.org/media/p3nhkbhi/2018-india-litearcy-impact-eval-of-hindi-schools-under-seri￾partnership-approach-endline-report.pdf USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 7 • Anganwadi (AW) Study4: CmF commissioned this study was commissioned in 2019 as an assessment of the Preschool intervention program implemented by the Bodh Shiksha Samiti in clusters of AWC in the Abu Road block of Rajasthan. The scope of this study was limited to assessing the impact of practices and methods implemented by Bodh Shiksha Samiti at their AWC to prepare children for primary school. The study focused on qualitative and observational evidence of school readiness and literacy learning in Grade- 1 children and their teachers’ understanding and expectations of the impact of inputs received at NELP AWCs. • Limitation: This study focused only on AWCs in the Abu Road block and used AWCs in a neighboring state to compare (matched on key demographic variables). The study analyzed the two regions on critical variables, but the actual statistical calculations are not included in the final report. Hence while the comparison group might be valid, for this evaluation, the ET does not consider these statistical results to be in line with a statistical impact study and instead refer to them as a comparison study. Additionally, the AWC study was largely based on observational measurements from trained enumerators (with experience in ECE pedagogy, etc.) and so cannot be considered summative but can be used as a point of triangulation. All graphics from these studies have been labeled as EGRA Study and AWC Study for easy reference. 4 https://firebasestorage.googleapis.com/v0/b/cmfwebsite-42d13.appspot.com/o/story%2FFinal%20Report- %20ECE%20Impact%20Study%20by%20CmF%20(NELP).pdf1679755395704?alt=media&token=db368474-6002- 4789-844d-1e9d546e15b0 8 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EQ1. TO WHAT EXTENT AND HOW DID THE ACTIVITIES ACCOMPLISH THEIR OBJECTIVES DURING IMPLEMENTATION? FINDINGS NELP implementation can be considered across two phases, pre- and post-March 2020 when CmF became the sole NELP implementer due to the COVID pandemic. Originally, NELP comprised four implementers (besides CmF) responsible for NELP activities in their geographic areas, as outlined in Table 1 below. The COVID outbreak delayed most field-level activities, impacting the project's progress. TABLE 1: NELP IMPLEMENTATION PARTNERS The evaluation assessed NELP performance across four component areas, 1) Capacity Development, 2) Improved Student Learning, 3) Resource Development, and 4) Community Engagement.5 Capacity Development- NELP built the capacity of government schoolteachers, AWC Workers/Community workers, and Administrators and officials at the state/ district/ block level. 5 The performance numbers for all four component areas are found in CmF Progress Report: February 2020. Centre for microFinance (CmF)– (prime recipient and sole implementer post-March 2020) Kalike, Bangalore Yadgir (Yadgir), Karnataka Pragat Shikshan Sanstha (PSS), Phaltan (Satara), Maharashtra Room to Read India, Pindwara & Sirohi (Sirohi), Rajasthan Bodh Shiksha Samiti, Abu Road (Sirohi), Rajasthan NELP Capacity Development Success: • Trained 11,400+ Teachers, Headteachers and PEEOs on Literacy, Library and Numeracy (target- 3,460 teachers)* • Trained 1500+ AW Workers, AW Helpers and AW Supervisors (target- 1,034 AWWs & AWHs) *note that these numbers are not the same numbers and definitions as used in KPI ES1.6 USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 9 Improve Student Learning- NELP worked in 5 intervention blocks by developing a solid foundation for literacy, including (a) explicit literacy instruction, (b) remedial teaching for at-risk pupils, (c) structured print-rich environment, and (d) pilot- tablet-based reading promotion. Resource Development- This component focused on developing contextual and relevant teaching-learning materials for students and teachers. Community Engagement- NELP’s focus on community engagement was meant to facilitate the enabling factors necessary for improved performance across all the components. The COVID pandemic is the primary reason for performance gaps in the NELP Key Performance Indicators (KPIs) outlined in Table 2. However, these KPIs may not be reflective of NELP performance. NELP commissioned a third-party evaluation to assess its contribution to learning outcomes that identified positive results (as will be discussed in the section below for EQ#4). The results of this third-party evaluation could not be used to inform the KPIs in Table 2 (due to differences in indicator definitions and the design of the third-party evaluation). Still, it does provide a fuller picture of NELP's performance. TABLE 2: NELP FINAL INDICATOR REPORTING6 6 NELP Final Quarterly Performance Monitoring Report (April to June 2021) NELP FINAL INDICATOR REPORTING INDICATOR REFERENCE INDICATOR CUMULATIVE TARGET ES. 1-1 Percent of learners who demonstrate reading fluency and comprehension of grade-level text at the end of grade 2 with USG assistance NA ES.1-5 Number of learners reached in reading programs at the primary level 330,000 ES.1-5a Number of boys 160,000 NELP Resource Development Success (there were no targets for these activities): • 60 multi-lingual resources (multilingual stories, storybooks, conversation charts etc.) • 13 Supplementary readers (PSS, Maharashtra) • 25 songs & rhymes under ECE (Kalike) • 10 Animated story videos with SLS (eLibrary) • TLM kits of ECE and Literacy • Instruction manuals, handbooks and training modules NELP Student Learning Success: • 1,47,000+ unique children reached (overall 3,25,000+ children; target- 93,000) • 46% project children achieved reading and comprehension against 19.8% in control (target- 60%) NELP Community Engagement Success (there were no targets for these activities): • 450+ SMCs/SDMCs strengthened (4,000+ SMC members) • 35,000+ Parents & Caregivers reached through meetings, home visits and community events 10 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV NELP FINAL INDICATOR REPORTING INDICATOR REFERENCE INDICATOR CUMULATIVE TARGET ES.1-5b Number of girls 170,000 ES.1-6 Number of primary or secondary educators who complete professional development activities with USG assistance (Phase – 1) 3,300 ES.1-6a Number of men 1,500 ES.1-6b Number of women 1,800 ES.1-10 Number of primary or secondary textbooks and other teaching and learning materials (TLM) that are inclusively representative provided with USG assistance 350,000 ES.1-12 Number of education administrators and officials who complete professional development activities with USG assistance. 1,700 ES.1-5a Number of boys 160,000 Outside of COVID, ECE performance was hampered by three main barriers: 1. Frequent transfers of teachers posed a significant risk to maintaining a consistent program impact (especially in Rajasthan and Maharashtra) 2. Irregularity of students due to seasonal migration and diseases; average attendance was 55- 65% across all locations/components. 3. Language of instruction and children’s languages created difficulties in classroom transactions, especially in Rajasthan state. Garasiya and Maarvari are the two dominant regional dialects in Sirohi, Rajasthan, even though Hindi is a subject and medium of instruction in the schools. The language has a social-cultural alignment as well. In this region of Rajasthan, the higher caste people advocate for the Maarvari language, and the State Government supports Hindi. Garasiya is looked down on as spoken by marginalized groups. It is impossible to provide a statistical analysis of these barriers to gauge their influence on overall NELP outcomes. But there is qualitative evidence of their effects relevant to several EQs and will be included in those specific sections later in this report. Model Library: NELP’s Model Library sought to improve literacy by exposing younger students to crucial learning materials (Model Libraries, E-Library tablets, and Children’s Books). However, NELP realized that improving access to these materials was insufficient to achieve envisioned outcomes. Improved access and use of literacy materials, created in the Resource Development component, required 1) an evidence-driven pedagogy that facilitated their use by students, 2) cultivating partnerships with champions at individual schools or within government offices, and 3) communicating demonstrated success to build momentum across schools. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 11 e-Library: The ET visited three schools in Rajasthan where NELP introduced tablets (out of 10 who got tablets). We found at least 80% of tablets at all schools still in use, an informative finding given the tablets were up to five years old. This could be considered a proxy measure for the value the schools, and students, place on the tablets. The tablets that were no longer working had battery issues, not issues due to mishandling by students or school staff. The tablets are used daily, and the students are responsible for much of their care (charging, etc.). On a qualitative level, many school staff noted the importance of tablets as an initial exposure of many students to digital technology (a small percentage of students do not have any cellular technology or smartphones in the home). Pictures/pictorial books greatly help beginners learn the language, as reading is the process of meaning-making, comprehension, and interaction between the reader and the text/pictures. Staff feel the interactive nature of the technology (where students engage with the learning software) takes advantage of best practices in Early Grade Reading pedagogy, allowing students to use repetitive learning in a way and at a pace that is not possible without the tablets (i.e., where a teacher is engaging in traditional classroom instruction only). Many staff observed they could see students’ confidence in using digital technologies improving and saw this positively influencing their ability to engage with technology in the future. New government education policy recognizes the need for improved technology in the classroom but, given limited resources, has opted to emphasize the use of a limited number of desktop computer terminals in schools as opposed to tablets, something staff in the surveyed schools essentially felt was less optimal than tablets. The implications of this policy, combined with the lack of future funding for similar CmF activities, means that the e-library NELP innovation could be the only intervention that introduced tablets into classrooms for EGR outcomes in India. FIGURE 1: NELP INTERVENTION AREAS “the e-library is a critical intervention from a learning perspective. Children use it twice weekly as dedicated periods have been given for library.”- school principal in Rajasthan during a KII (2/20/23) 12 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV ECE: Through USAID and Tata Trust support, Bodh introduced Preschool fellows in AWCs. This introduction increased enrolment at AWCs, and people started perceiving AWCs as a foundation for school. Bodh demonstrated the integrated model of AWC and school under this intervention during 2015-2018. Two years later, the Central Government made a policy of integrating AWCs with schools for foundational learning under NEP-2020. During the EGRA study, country teams assessed language curriculums and classroom instruction to ensure that all five core elements necessary in a comprehensive language curriculum are included. These elements, which are best addressed through a combination of listening, speaking, reading, and writing activities and lessons, include: • Phonological awareness • Phonics • Vocabulary • Fluency • Comprehension These findings are encouraging since these elements are critical to the NELP theory of change, and envisioned learning outcomes cannot be obtained without fidelity to these elements. During the evaluation field visits in Rajasthan and Karnataka, most teachers and AWFs trained under the project practiced activity-based pedagogy using children’s literature, TLMs provided by the implementing partners, and some locally procured materials. Even three to five years after NELP completion, most intervention schools used printed materials and pedagogy to equip students with literacy skills. For example, in Rajasthan, Room to Read and CMF developed storybooks in the local language (Garasiya), which helped children gradually migrate from the spoken language to the formal school language (Hindi). These same materials were still being used and in good condition in all the relevant schools visited by the evaluation team. FIGURE 2: SCHOOLS AND ANGANWADIS REACHED BY NELP PER YEAR (EGRA STUDY) The AWCs and schoolteachers in Abu Road, Rajasthan, speak highly of NELPs training and TLMs and cards developed for language and mathematical skills are still being used at all 12 AWCs visited during the evaluation. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 13 CONCLUSIONS • NELP created an enabling environment for literacy by introducing age-appropriate learning materials into a library environment managed by an evidence-driven pedagogy. The ET found that 34 RSLPP model libraries contributed to children's improved reading in the Hindi language two years after the program intervention. A wide array of printed and non-printed learning materials in RSLPPs ensures that students are provided with various options that cater to their learning needs, interest, motivation, and styles. Among the components of RSLPP, e-tablets were very effective in targeting learning outcomes and improving children's ICT skills. As a result, RSLPPs contributed to the overachievement in the number of students who can read with comprehension compared to its target. • NELP facilitated a culture shift that supports literacy by using model libraries to create an enabling ecosystem demonstrating sustainable progress after the completion of the NELP program. This includes creating the necessary Knowledge, Attitudes, and Practices of ecosystem actors like student parents, community leaders, school administrators, teachers, and government officials to all do their part in improving literacy. The model library seems to have a very high return on investment based on the qualitative data available. The library movement for younger children motivated numerous non-NELP schools in Rajasthan, Karnataka, and Maharashtra to establish their libraries with increased community support. In Yadgir, Karnataka, around a hundred students created small libraries at home. This initiative was a significant cultural change in the region as NELP built an ecosystem conducive to literacy by influencing the attitudes of parents, communities, and school administration officials (and their district government counterparts) to support literacy, whether it is by parents reading with children, communities developing their libraries, or governments enacting policies for library administration. The training offered to the 100 schoolteachers on library management in Rajasthan demonstrates a clear commitment by the government of Rajasthan, which is taking up the intervention in more than 3300 schools while CMF is working as a technical support agency. The module on library management developed under the project is being modified with the insights gained through NELP performance management and will be used across the state. Exposure to e-library (tablets) resources by children opens up alternative learning approaches in which students can learn at their own pace and time without depending on teachers. This exposure to digital learning presents various possibilities for customized learning modes with the help of technologies. • There is overachievement in the capacity development of Teachers, Headteachers, PEEOs AW Workers, AW Helpers, and AW Supervisors. This achievement relates to multiple- modes of quality training in in￾person, virtual, modular, mentoring, and online coaching approaches. NELP’s overachievement was partly accomplished due to its positive spillover effects where other schools and communities adopted NELP approaches through direct or indirect collaboration with a NELP IP. But even without these positive spillovers, NELP outperformed expectations due to highly capable IPs and robust performance management. RECOMMENDATIONS The fact that NELP was able to transform the culture towards literacy in various ways, given its resources and mandate, is significant. Because NELP management was high performing, many of the 14 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV issues and problem areas this evaluation could have used as a basis for Recommendations have already been addressed through NELP remedial actions and adaptations. 1. USAID Convening Authority: USAID could use its convening abilities to discuss NELP successes with Indian government stakeholders to identify entry points for continued support. There is a substantial overlap between the Indian New Education Policy -2020 and the new USAID Progress Beyond Programs Policy in building on programs with evidenced success and taking them to scale. A facilitated discussion with NELP IPs to identify specific points of entry for USAID convening actions would be helpful. For example, engaging specific government offices to scale the Model Library beyond Rajasthan or expanding the field testing of the tablet pilot. This collaboration could be a low-cost option for USAID to facilitate NELP’s scale-up pathway as a successful demonstration of Progress Beyond Programs. 2. GOI and USAID Actions to Mitigate Negative COVID Effects: The degree of disruption in ECE caused by COVID is hard to understate. There could be more opportunities to mitigate the adverse effects on literacy from COVID, including finding resources for NELP activities that were halted due to COVID, especially now that NELP produced substantial learning from its internal performance management. 3. GOI Reforms for Teacher Transition: Frequent schoolteacher transfers were a barrier to sustaining the impact of NELP. Introducing capacity development activities for teachers in their new schools was not part of NELP. Still, it could be included in future programming to extend the exposure to literacy activities and cultivate champions in new schools. EQ2. HOW HAS THE PROJECT IMPROVED THE STUDENTS’ LEARNING IN TERMS OF BASIC READING SKILLS? FINDINGS The Model Library is meant to create an enabling environment for literacy outcomes through three activity categories that are documented in a training and instructional manual for school administrators to include: • Reading-related activities- To lead the reading of books so children can join in the reading themselves. They may read with friends or teachers, or the teacher may read with them so they can develop an interest in reading. • Writing-related activities- To provide opportunities for children to read and express their thoughts creatively. • Motivational activities- To attract children towards the books kept in the library. The Model Library has a target of replicating library practices in another 3300 schools across Rajasthan, and the evaluation found extensive support for it amongst school administrators, teachers, communities, and government offices. The Model Library has six factors that are critical to the Library Model: 1. People (have a trained librarian in charge) USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 15 2. Collection (library must have 100+ curated book collections) 3. Space (dedicated space for the library) 4. Engagement (time allocation in school timetable) 5. Administration (HM/Principle attends training offered by CMF) 6. Activities (Teachers demonstrate activities like Book Talk, and Read Aloud) These factors are actively monitored and used to update training and guidance and further build the institutionalization of the model. The dedicated physical space for the libraries is divided into various sections adapted depending on the students' needs and the available space. These sections can include book repair, reading corners, chat/discussion corners, etc. The evaluation team noted teachers' highly adaptable approach to maximizing their space and innovating using outside and classroom areas to increase access to library resources, including using string and rope in classrooms to hang books on the walls, as seen in Image 1. Model library outcomes (i.e., their contributions to improved literacy) are greatly enhanced through dedicated Library time during regular school hours. NELP successfully advocated with district offices to enable policy changes for standard Library Hours at all schools. Photo 1: Book Storage "The furniture, drawing sheets, story cards, remedial classes and the multi￾grade and multilevel teaching method using TLMs are the unique feature of the project. We are still getting the benefit of the remaining resources and our training. “-Government Upper Primary School Teacher at Yadgir, Karnataka Training on library model. The teachers’ library training included 1) Book Posters, 2) Library Organization, 3) Read Aloud, and 4) Curriculum Pair Reading. The modules integrate lessons from storytelling-focused training that became popular with the State Resource Group, which trained other teachers in their district. Higher-order skills were added to the library training, like Book Reviews and Book Circulation, which helps teachers and students use the library more effectively. Provisions for home use: While some library resources stay in the library, others can be checked out by students and taken home. This practice helps develop parental awareness and understanding of the importance of reading for early childhood development. The students are responsible for maintaining and repairing the books. School visits by the evaluation team consistently found all books in excellent condition even when repaired, as the repairs were done by students using repair tape and other materials. 16 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV Community Participation: NELP successfully integrated communities into their model library activities by training school staff to engage parents on reading at home and the overall value of practicing reading. However, the model library component was able to influence positive community outcomes in ways it never intended. The library interventions advanced the AWCs and schools' learning environment and contributed to changing the parents' perceptions. Now more children come to AWCs and stay longer as meaningful learning activities are being done at the AWCs in both states. For example, in Yadgir, the model library intervention was successfully implemented in 100 schools. More than fifty students established their own libraries at home with parental support to buy books beyond required textbooks. During Covid-19, Gram Panchayat (GP) (village collective) community libraries were opened using many model library attributes. Several Kalike schools provided books to GPs while schools were closed. Craft activities were also added, and ID cards were given to the children to provide access to literacy resources. Given the success of the libraries during COVID, the local government is emphasizing scaling up the GP libraries. Now more than 6,000 GP libraries are functioning in the state, and some Kalike schools are assisting in developing a training module for GPs due to begin in March 2023. In the Sirohi district of Rajasthan, the administration has been actively working on establishing five public libraries at the block level and 172 at the Gram Panchayat level. ECE: For children to effectively learn, the classroom environment is a crucial predictor for learning outcomes. A safe, clean, well-lit, and well-ventilated environment is an enabling factor for learning, in addition to classroom displays, ready access to books, and availability of writing and arts and craft materials. The AWC Study assessed the physical environment in selected Anganwadis to assess these enabling factors in preparing children for primary education. The training design for AWFs consisted of cognitive, creative, early/emergent literacy-numeracy, and physical development aspects. AWC learning corners were also organized based on cognitive, social, and creative developments. The training design for AWFs consisted of cognitive, creative, early/emergent literacy-numeracy, and physical development aspects. AWC learning corners were also organized based on cognitive, social, and creative developments. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 17 FIGURE 3: AWC PHYSICAL ENVIRONMENT (AW STUDY) CmF conducted its EGRA study in March-April 2019 across all its project locations. The endline showed significant improvement in children's learning levels. The below figure shows the percentage of learners who demonstrated reading fluency and comprehension of grade-level text at the end of grade 2. The evaluation used observational checklists assessing these dimensions at each school and Anganwadi site and found them very satisfactory in all locations. 18 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV FIGURE 4: COMPREHENSION RESULTS (EGRA STUDY) Baseline results indicated that NELP students entered Grade 1 with the same reading skills as children from the comparison group. Students from both groups could read one letter per minute approximately. Reading skills were tested at midline from the same cohort of students (84 children across eight project schools and 106 children across eight comparison schools) assessed at baseline to determine the program's impact after two academic years. By the end of Grade 2, students from NELP schools could read approximately 55 letters per minute. By contrast, Grade 2 children from comparison schools could, on average, read only 18 letters per minute. Students from NELP schools could, on average, read 25 words per minute, while children from comparison schools could read three words per minute. Both boys and girls from NELP schools performed better and experienced more significant two-year gains than those in comparison schools with no measurable gender differences. FIGURE 5: READING FLUENCY (EGRA STUDY) CmF’s EGRA study found that reading fluency and reading comprehension (on average, 1.8 questions were answered correctly) levels of treated school children were lower than expected. Additionally, 12 percent of intervention school students were essentially non-readers. A possible reason for this could have been difficulties encountered by students transitioning from their home language to the USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 19 standard language of instruction (Hindi). An improved oral language component was added to the curriculum to mitigate this challenge. While no subsequent EGRA testing was done after this modification, schoolteachers were instructed to assess observational evidence that the language barriers were being addressed. CONCLUSIONS Triangulated data from qualitative observations, assessments, and third-party EGRA studies indicate that NELP improved pre-primary and primary student cognitive, behavioral, and literacy skills. USAID’s standard method of measuring learning outcomes is the EGRA tool using multiple rounds of data collection and statistical methods.7 This, combined with the observational and qualitative data collected on the behavior changes needed to achieve these outcomes collected as part of this ex-post evaluation, presents a strong case that NELP was very successful in achieving improved learning outcomes in its treated schools and creates opportunities for similar success in those schools that are adopting NELP practices. RECOMMENDATIONS GOI and USAID Support for Listening, Observation, and Community Outreach: Listening and Observation activities that capitalize on the students' improved access to reading materials could be included in future programing and policies. Likewise, given the success of the children’s reading clubs, this low-cost effort has much potential to engage the community in recognizing the importance of literacy and cultivate them as stakeholders in improving students reading. EQ3. TO WHAT EXTENT HAS THE PROMOTED BEHAVIOR AT SCHOOL AND SYSTEM LEVEL CONTRIBUTED TO THE EARLY LITERACY COMPETENCIES AMONG THE TARGET GROUPS? IF SO, HOW? IF NOT, WHAT ARE THE FACTORS THAT HAVE HINDERED THE PRACTICE TO CONTINUE? This section is organized by RSLPP and ECE activities with subsections for school and system-level behavior changes envisioned by those activities per the evaluation question. FINDINGS NELP was partly meant to improve access to and usage of reading materials while building sound literacy practices. However, the behavior change necessary to accomplish this included students and an entire ecosystem of teachers, principals, administrators, community members, and government officials. NELP focused on behavior change at all levels of operation (from students to administrators, etc.) and used numerous tools to achieve the desired behavior (from capacity development to policy reform). The school and systems-based behavior changes outlined below are not comprehensive, given NELP is still seeing progress with some of its components being scaled by different actors and/or influencing improved policies and practices. However, given the success NELP had in promoting envisioned Pre-Primary behavior change, Angandwai-level behavior is given a specific focus. 7 Early Grade Reading Assessment (EGRA) Toolkit | Education Links (edu-links.org) 20 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV AWC/School Level Behavior Change: Conventional reading is a complex phenomenon. To master it, children must develop an interest in the written language and engage freely. Creating such opportunities is the most crucial strategy for moving young children from oral to written language for meaningful communication. Children begin by showing some literate behaviors before decoding and encoding the print conventionally. This change includes (but is not limited to): • Showing interest in print and understanding that symbols are meaningful. • Being able to construct and share meaningful stories by looking at pictures. • Reading from left to right and from top to the bottom of a page. NELP also developed and used storybooks in a local language (Garashiya ) for a smooth transition from the community’s language to the school language (Hindi). Under NELP, the teachers received orientation to establish and manage libraries with the support of the children’s committee. The behavior-focused training (see Image 2) offered to the schoolteachers on library management in Rajasthan has gained traction since the end of NELP as the government of Rajasthan is taking up the intervention in 6500 schools (discussed at greater length under Evaluation Question # 4). The module developed under the project is being modified with the insights gained through NELP and will be used across the state. FIGURE 6: BEHAVIOR-FOCUSED LIBRARY TRAINING8 Transitioning from home or pre-primary (AWCs) to primary school is a major behavioral step in the lives of young children. They need to learn to look after themselves and their belongings, make new friends, get the necessary information on their new environment, talk to the teachers, work in a group, stay on task, and independently complete the given tasks on time, etc. Therefore, focusing exclusively on academic preparation for primary schools in AWCs is not very useful to young children. Their social and emotional preparedness is far more critical as this gives them greater confidence to function well in the new environment and reduces undue stress. School Level Behavior Change: Through NELP, AWC’s promoted the behaviors associated with Emergent Literacy, including: • Handling books. 8 CmF Progress Report: February 2020. “The teachers who attended the training on early grade pedagogy got transformed. They use teaching materials such as pictorial books, colour books, printed materials, Flashcards, and we will never go back to how it was”. - HM of a government Upper Primary School in Sirohi. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 21 • Reading left to right. • Recognizing printed images and words and relating with them. • Understanding that letters make words. • Interest in listening and reading stories. • Comprehending meaning from the pictures and stories. • These behaviors prepare the child for more foundational literacy behaviors that NELP emphasized in primary education, including: • Reading with comprehension. • Writing with understanding. • Speaking with coherent thought. Young children often respond to stimuli reflexively. To gain self-regulation is to gain control over their thoughts and emotions so that they can move from reflexive to deliberate behaviors. The AWC Study highlighted a gap between social-emotional development in the children in NELP and Comparison schools (which is supported by evidence from teacher questionnaires) that parallels learning outcomes in both groups. FIGURE 7: SOCIAL ENVIRONMENT (AW STUDY) 22 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV FIGURE 8: SELF-REGULATION (AW STUDY) FIGURE 9: ORAL LANGUAGE (AW STUDY) Oral language development provides the foundation for children’s reading and writing skills. This is a crucial area of development in the early years. Children learn to speak their home language naturally. However, they must learn a new language when they come to school. Even if their home language is the same as their school language, the communication patterns at home and school are quite different. Listening is as essential for effective communication as is clear expression. Learning to listen USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 23 is a deliberate and complex phenomenon. One must correctly hear what is being said, make sense of what one hears and respond accordingly. Becoming a skilled listener takes time and practice. Listening to and responding to instructions appropriately is crucial in a learning environment. Good listening skills are necessary for making sense of what is being taught and even more basic learning as deciphering the sounds of the language. FIGURE 10: LISTENING SKILLS (AW STUDY) 24 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV FIGURE 11: READING (AW STUDY) System Level Behavior Change: Through NELP, the library component for primary grades got space in the academic calendar of the state of Rajasthan for the first time (which has since been expanded as outlined in EQ5). Additionally, the Rajasthan Government has included the library component in the primary school ranking indicators. The government in Rajasthan itself has developed a library manual with the technical backing of CmF and Room to Read, which will be sent to 6500 schools in the state. CONCLUSIONS NELP’s behavior-focused approach had measurable positive effects in preparing children in AWCs to become students in early primary education, primary education students to excel across school subjects, community members to engage in literacy promotion, teachers to use best practices in doing so, and policymakers to create optimal enabling conditions. NELP successfully designed and implemented activities that addressed key barriers and built on key enablers for desired behavior change at the System and School (including AWC) levels. This is partly due to a combination of expertise in early-grade literacy (best practices on EGR pedagogies, etc.) and exceptional knowledge of local conditions by NELP IPs. This local knowledge and relationships cultivated ownership from the teachers, HMs/Principals, AWWs, and their supervisors bought into the behavior change objectives through learning opportunities offered by NELP. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 25 RECOMMENDATIONS USAID Support for Expanding the Enabling Environment: Capacity building of block-level education officials, cross-pollination/dissemination of learnings, and adaptation of best practices among the consortium members and schools could improve performance and additional scaling opportunities. GOI Emphasis on Inclusivity: Audiobooks, brail books, and other materials for differently abled students could be introduced to improve inclusivity and help marginalized students to develop the same behaviors as their peers. EQ4. HAVE THE IMPLEMENTING PARTNERS/STAKEHOLDERS STRENGTHENED BY USAID FUNDING CONTINUED TO IMPLEMENT THE ACTION? HOW ARE THE SCHOOLS, COMMUNITIES AND OFFICIALS CONTINUING TO MAINTAIN THE EARLY LITERACY SYSTEM IMPROVED BY USAID FUNDING? WHAT IS THE BENEFICIARIES’ PERSPECTIVE / FEEDBACK ABOUT THE IMPACT OF THE PROJECT? FINDINGS NELP IPs have continued to scale different NELP components beyond the initial NELP schools by working with different government offices, communities, and schools to adopt NELP practices. Given the breadth of this continued progress, we have organized some of these activities by the Indian states in which they are occurring. There has also been feedback from beneficiaries in the form of awards and requests for additional assistance in continuing relevant NELP practices. Karnataka: Before the creation of the National Education Policy (NEP)-2020, after seeing the work by Kalike (sub-partner in Karnataka) during 2018-19, CmF and Kalike partnered with district administration under the Aspirational District Programme (ADP) to expand the NELP ECE work to 13300+ AWCs of Kalyan Karnataka region (consisting of seven districts of north Karnataka). Kalike was asked by Karnataka district officials, as part of ADP, to provide technical support to the department and to carry out perspective-building exercises with Dy. Directors, Child Development Project Officer CDPOs, and Assistant Child Development Project Officer ACDPOs. The departmental officials agreed to train AWWs, Helpers, and Supervisors while Kalike provided resource persons for the training, and the Women and Child Department handled all the expenses. Kalike’s ECE work was also recognized by the Kalyana Karnataka Regional Development Board (KKRDB), earlier known as Hyderabad Karnataka Regional Development Board (HKRDB), which requested that Kalike expand the ECE program in 7 districts of North-East Karnataka. In response, Kalike has scaled up the program through technical support from HKRDB. Maharashtra: Positive feedback for ECE activities in Maharashtra came in the form of an award by the Chief Executive Officer of Satara Zilla Parishad for the quality work in 150 project schools. The implementing partner in Maharashtra, PSS, received a request letter from CEO (Satara district) to scale-up the early literacy interventions in the district which is currently being designed. The Model Library intervention motivated other schools (non-project) in Rajasthan and Maharashtra as they established the library with community support. In the Sirohi district, the administration established a community library in 2023 and aspires to set up libraries in five blocks and 172 Gram Panchayats (Village level). 26 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV Rajasthan: An MoU has been signed between CmF and Government of Rajasthan to scale up NELP activities, especially those related to reading promotion through the school library, in which CmF provides technical and monitoring support to RSCERT and DIETs. There are three components to the MOU 1) Model Library Set-up, 2) Professional Development, and 3) Support and Monitoring. Each with correlated sub-components: 1. Model Libraries Set-up a. Model Libraries at Districts and RSCERT b. Infrastructure Support c. Resource Support 2. Professional Development a. Identification of State Resource Groups (SRGs) b. 12-Week Course for SRGs c. Training Module & Manual d. Library Teachers Trained e. PEEOs / Principals Training 3. Support and Monitoring a. Training Support to KRPs/SRGs b. Online Library Resource Platform c. Library Indicators/Checklist for Integrated Shala Darpan Portal d. Periodic Support & Monitoring through Visits / Meetings FIGURE 12: RAJASTHAN MOU KPI TARGETS CmF is engaged through the MOU to provide technical support at RSCERT-level on: • State Curriculum Framework (SCF) development o Library Indicator development and Library Teacher Manual development workshops USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 27 o Multi-Lingual Education (MLE) resources and training module development o ECCE KRP training module development • Support at DIETs (District Institute of Education and Training) o Workshops with library teachers on active school libraries o KRP, Mentor teachers, AW Workers training for ECCE (Early Childhood Care and Education 28 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV Picture 4: Library Training Manual developed through MOU and CfM lead Teacher Training The MOU has standard KPIs that are regularly collected on and reported as well. TABLE 3: MOU PROJECT -RSLPP PROGRESS UPDATES: PHASE-I (1/2) MOU PROJECT -RSLPP PROGRESS UPDATES: PHASE-I (1/2) KEY ACTIVITY OVERALL TARGET ACTUAL Training of Library Point Teachers – 3300 Teachers [100 teachers per district] 12 days [4+4+4] 4 days [2+2] Training of PEEOs (Panchayat Elementary Education Officers) and Head Teachers [100 schools in each of 33 districts] 4 days [2+1+1] 2 days [1+1] Setting up 34 District-level Model Libraries [1 in each of 33 districts and 1 at RSCERT Udaipur] 34 35 Training of State Resource Group (SRG) [2-3 from each of the 33 districts] – 66 SRGs 66 81 USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 29 MOU PROJECT -RSLPP PROGRESS UPDATES: PHASE-I (1/2) KEY ACTIVITY OVERALL TARGET ACTUAL District Project Steering Committees (DPSC) – quarterly review Quarterly 2 Library books for 3300 school libraries 100 books 100 books Library manuals for trained teachers 2 1 The EGRA study collected data on functional libraries that is reflective of the MOU performance. Given the differences in timelines in MOU implementation and the EGRA study data collection (in addition to other study design limitations), the graph below should be considered indicative of MOU performance but not conclusive. FIGURE 13: MOU PROJECT - OUTCOMES (EGRA STUDY) 30 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV CONCLUSIONS The Rajasthan MOU is a strong demonstration of scaled progress after program completion. The evidence indicates that the curation of the model through effective performance management by the IPs created trust in the model from government officials and school administrators, teachers, etc. This trust was built through effective communication and collaboration with these actors to demonstrate the model's effectiveness and the feasibility of its adoption. RECOMMENDATIONS CLA Driven Planning for Scale in Future USAID Programs: NELP never intended to scale in the way and to the degree it has. The scaling is mainly due to the innovations and commitment of CmF and other NELP implementers. The positive effects of this commitment and innovations could be enhanced with better preparation for scaling during co-creation processes in future programs. Accounting for scaling objectives during program design in a collaborative way that invites participation from those actors (government, school administrators, etc.) critical to scaling success is a best practice in achieving progress after programs. EQ 5. WHAT INTERVENTIONS WERE MORE SUCCESSFUL AND/OR HAD A GREATER CONTRIBUTION TO ACCOMPLISH THE OBJECTIVES? FINDINGS When assessed against their objectives, NELP performed well against their KPIs, considering COVID disruptions. Part of the success of NELP is its design and, perhaps more importantly, its performance management that optimized its resources to address as many of these barriers and enablers as possible. For this section, we outline those NELP components that had more success and the factors for that success without overlapping with the content for Evaluation Question #6. RSLPP: The evaluation found the Model Library component to be highly successful, with higher probabilities of successful scaling when compared to similar early education literacy interventions9. The Model Library is more likely to scale up given the specific enabling conditions described below; however, the E-Library also has a substantial opportunity for scaling, assuming certain factors. FIGURE 14: SCALE UP PATHWAY10 As outlined in the section for Evaluation Question #4, efforts are underway to institutionalize the model by creating an enabling system that focuses on policy reform to create library hours, teacher training to build capacity, and performance management to improve the model. The model has achieved a “Transition to Scale” phase in Rajasthan and could quickly move to “Sustainable Scale” given additional resources and enabling policies. 9 https://www.cgdev.org/blog/it-possible-improve-learning-scale-reflections-process-identifying-large-scale￾successful 10Insights on Scaling Innovation — International Development Innovation Alliance (idiainnovation.org) USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 31 This success is partly due to the ability of NELP to identify and overcome barriers to scale. For example, teachers were not completing the training as they were transferred to new schools before completion. NELP adapted by altering training timetables and curriculum to optimize the number of teachers trained. Training is not the only disruption caused by teacher transition. As teachers leave schools, the model libraries often diminish due to the absence of the “champion.” The importance of the role of the champion in innovations going to scale is well evidenced. But there is a currently untested assumption that as teachers transition, they will take their knowledge and role of the champion to the new school, resulting in improved library access and usage at that school for grades 1-5. Collecting data to test this assumption is extremely difficult. Still, the fact that NELP has 1) identified the assumption and 2) is proactive in looking for any data to test it is indicative of effective performance management. Initially, the model library training was only for teachers and not school officials, which places the teachers as the sole champions and managers of the library. Given the lack of dedicated librarians at the schools, the entire management and student use of the library, including instruction during the dedicated library hour, is the responsibility of the teachers. This heavier workload on teachers is coupled with varying levels of lower institutional support from school officials, given their lack of training on the library's importance, function, operations, and maintenance. NELP adapted again in advocating for school officials to participate in the training and conducting needs assessments to identify the optimal approach. This approach was designed to cultivate champions as part of a larger enabling ecosystem to take the model library to scale. E-Library: The E-Library was a small pilot implemented in 10 schools in Rajasthan, where each school was given 12 tablets. The E-library combines an evidence-driven pedagogy with technology and removes physical access barriers to the library. This approach provides instant access to the content and helps students listen if they cannot read while giving them independence and reducing pressure on the teachers. The tablet pilot seems to be a single occurrence that will not scale due to a lack of donor and government support. However, the pilot's success is significant (as outlined in Evaluation Question 32 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV # 1). Children demonstrated various behaviors (confidence, responsibility, engagement) and improved literacy performance while learning to work with digital technology. However, it does not seem likely that the pilot will scale, given the government’s focus on individual desktop computers in schools (over tablets) and the end of donor funding. CONCLUSIONS NELP cultivated a culture shift that improved literacy outcomes by advocating for using libraries and Activity Based Learning effectively. NELP helped establish libraries by training teachers on library management, developing a module on library management, and supplemental activities like the formation of WhatsApp discussion groups for schools having or wanting libraries that had significant positive effects in shifting the culture towards literacy. NELP had several best practices emerge during its implementation, including: • developing storybooks in the local language (Garashiya) and Hindi • e-books with tablets in Rajasthan • reading cards and village libraries (in Yadgir) Creating an enabling environment and opportunities for reading and writing is crucial. Poor readers cannot be good writers, and if they do not get timely support in ECE, they may leave school early if they do not feel confident or engaged (both behaviors saw improvement under NELP). These behavioral measures signal that NELP had favorable Treatment on Treated (ToT) effects and positive spillover and residual effects that speak more to a shift in culture that has had substantial cascading effects after the completion of NELP. NELP demonstrated practical teacher training in Rajasthan and Karnataka, where training helped to move from the "Chalk-talk" method to small group/Multi-grade-multi level teaching. This was facilitated by the TLMs provided to schools and AWCs that were extensively used and generated interest among the children, parents, teachers, and government officials. Through intervention at AWCs, a learning environment was created there that was perceived earlier as only a food center but learning for kids. RECOMMENDATIONS GOI and USAID Support to Build on Scaled Success: The Model Library has demonstrated success and lessons learned that could be very cost-effective to scale through reasonable funding, communications, and other support by donors and Indian government officials. The critical data for informing future scaling resides in the institutional knowledge of the NELP IPs who could be the formative factor in informing future scaling efforts. GOI Consideration and USAID Support of Further Testing of E-Tablets: The E-tablet pilot was substantially successful for its size. The end of donor funding and a government emphasis on desktop computers in schools has created barriers to further testing this innovation. Given the pilot's success, coordinated efforts to further test and iterate the pilot could provide the success in literacy outcomes seen in NELP and provide the knowledge necessary to determine how best to use tablets in schools for learning outcomes in general. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 33 EQ 6. WHAT ARE THE FACTORS THAT CONTRIBUTED TO OR IMPAIRED THE LONG￾TERM SUSTAINABILITY OF THE ACTIVITIES’ OUTCOMES AND OUTPUTS? When considering the factors that contributed to or impaired outcomes, the evaluation found the same factors relevant for RSLPP and ECE. This section will not disaggregate findings but will look at NELP performance holistically. FINDINGS NELP had three contributing reasons for its success and sustainability: 1. Alignment with government policy, specifically on the transition from rote learning to meaningful engagement. 2. Skilled implementing partners who know the local context, actors, and culture, which facilitates strong performance management. 3. Evidence-driven approaches and focusing on enabling environments. Alignment with policy: NELP was aligned with the NEP-2020 as the policy emphasizes reading, writing, and numeracy as the foundational skills. Concentrated efforts have been made at the national and state levels to equip teachers to provide students with language and mathematical skills at an early stage. The ‘Padhe Bharat, Bhade Bharat’ initiative by the Government of India under Samgra Shiksha Abhiyan has also reiterated the vision for early reading, writing, and arithmetic. The New Education Policy makes foundational literacy and numeracy critical prerequisites for schooling and education and asks for immediate attention. NEP-2020 has focused on preschool education and made it an integral part of the education system, which was less emphasized previously. NEP-2020 aspires to make learners competent in meaningful reading and writing by the time they enter grade 3. They wish to develop the fundamental understanding and competencies of numeracy and its related concepts among young learners by grade 2. Skilled implementing partners: The four implementing partners, headed by CmF, are all well￾known and respected local providers in the states in which they operate. Their high level of contextual understanding allowed them easier access to the actors they needed to engage with, an ability that is hard to overstate. These partners' technical knowledge and capacity, combined with their local contextual abilities, enabled a highly effective performance management system that allowed NELP to find creative solutions to complex problems on a routine basis that went beyond the disruptions caused by COVID. CONCLUSIONS The success, including post-program progress, of NELP primarily resulted from having the right partners (IPs) in the right places at the right time. Out of these three factors, having the correct IPs is possibly the most significant contributor to success, given that, in many ways, NELP IPs used their capacity to maximize the potential of being in the right place at the right time. The CmF lead consortium identified and capitalized on opportunities to improve NELP performance. Some of these opportunities resulted from alignment with national education priorities outlined in NEP-20 or pre-existing relationships with key stakeholders within the geographic areas of NELP 34 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV implementation. Much of NELP’s success can be traced back to high-performing NELP IPs maximizing these opportunities, creating their opportunities, and adapting. • NELP’s work in the AWCs transformed how children spend their time in these centers. Before, these centers were seen as childcare and meal service facilities. Now, they prepare children to become effective students by cultivating their formative skills to excel in and out of school. This success motivated government officials to support the initiative through increased resources and enabling policies. NELP demonstrated AWC opportunities within the National Education Policy of India, 2020. For the first time, the integration of AWC and ECE was demonstrated under this program in Rajasthan from 2015 to 2019. The success of NELP contributed to the central government creating a policy of linking AWC with schools for foundational learning for ages 3 to 8 years. The project showed how AWCs and Schools could be linked through the emergent literacy￾numeracy learning pathway. AWCs and Schools belong to two different departments (Women and Child Development and Education), and the project worked with both institutions on literacy￾numeracy in a continuum. Training AWWs and their Supervisors worked well on the ground, and in Karnataka, Kalike received recognition as a specialized agency for the state to implement foundational learning. In Rajasthan, training AWWs and placing preschool fellows at AWCs looks like a winning strategy. Enrollment and quality of learning both increased at the AWCs. Bodh (an IP) had contributed significantly to revising and creating the capacity to implement the new preschool curriculum of Rajasthan. RECOMMENDATIONS Recommendations for this section are organized by actor. FOR DONORS: • Projects that emphasize shifting from rote learning to learning with comprehension/meaning￾making could have longer periods of performance for the 3 to 9 years age group. Longer periods of performance allow for projects to "treat" the same students as they progress through their most formative years of learning that establishes skills for the rest of their life. • Given the success of NELP and its alignment and contribution to national education policy, USAID could use its convening ability to disseminate NELP's lessons learned and success through wider publications, events, etc., to ensure that its success is built on in India and beyond. FOR IMPLEMENTING PARTNERS: • District and block-level officials could be more included in the training/capacity-building plans of the project. One of NELP's successes was integrating these actors after the award (post-design) in a very effective CLA-driven performance management. But because there was no design co￾creation, this performance management had an opportunity cost due to the defined parameters of the design that could not be changed. • Training components could also be co-created/designed with teachers and officials to increase ownership and effectiveness. One training approach (like activity-based learning) may not suit all the participating teachers, and some may prefer reflective sessions or Edtech-based sessions. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 35 • The exchange of ideas and adaptation should be promoted among the consortium partners. NELP performed well in this cross-pollination of lessons between IPs but was constrained by COVID and prescriptive performance schedules mandated by contracts that did not allow many resources for learning. • For language promotion activities, listening and speaking could also be emphasized along with reading and writing to advance the cycle of language learning. These activities could also be extended to grade three. FOR POLICYMAKERS • AWCs could be resourced and managed as a ‘preschool’ with forward linkages to school as opposed to a daycare service with meal options, and could include a uniform report card could be introduced at the AWCs. • Mother’s education is equally critical to achieving holistic development of the children, including mothers in literacy promotion activities could improve literacy-friendly behaviors in the household. 36 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK Ex-post Evaluation of Nurturing Early Literacy Project in India A. Evaluation Purpose Projects may meet their objectives by improving economic, health, or social conditions while they are operating, but genuine success is achieved only through sustained change that does not depend on continued external resources. The overall purpose of this ex-post evaluation is to assess sustainability, and to know how the outcomes and impacts evolved after completing the education activity. Additionally, the evaluation findings will be used to guide the future programs in the education sector. B. Evaluation Questions The contractor must, at a minimum, address the following evaluation questions. 1. To what extent and how did the activities accomplish their objectives during implementation? 2. How has the project improved the students’ learning in terms of basic numeracy and reading skills? 3. To what extent the promoted behavior at school and system level has contributed to the early literacy competencies among the target groups? If so, how? If not, what are the factors that have hindered the practice to continue? 4. Have the implementing partners/stakeholders strengthened by USAID funding continued to implement the action? How are the schools, communities and officials continuing to maintain the early literacy system improved by USAID funding? What is the beneficiaries’ perspective / feedback about the impact of the project? 5. What interventions were more successful and/or had a greater contribution to accomplish the objectives? 6. What are the factors that contributed to or impaired the long-term sustainability of the activities’ outcomes and outputs? C. Introduction and Background: Despite good student enrollment rates in India (93 percent, according to UNESCO, 2011), the quality of education remains low and reading gains in the early grades are not yet at acceptable levels. Pratham’s Annual Status of Education Report reveals that less than 50 percent of grade five students are able to read a grade two level text, and related statistics from this annual measure make it clear that too many Indian primary school students are not acquiring reading skills at the appropriate rate. As a result, they drop out of school early, or enter adulthood either illiterate or with extremely poor reading abilities. Teacher education is not rigorous enough to effectively USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 37 prepare teachers for the classroom and includes little or no training in early grade reading pedagogy. Similarly, principals and other school administrators receive inadequate training, and school administration and the quality of education suffer as a direct result. Additionally, the schools are not equipped in handling such huge diversity in the student population. The problem is compounded as many children have no prior experience of being in preschools or in any literate environment. Thus, they enter schools without knowing the language in which classroom transactions are taking place, hence impacting their learning. Children who do not make good initial progress in learning to read find it increasingly difficult to master other skills. They also have problems in reading and comprehension at higher levels. Principals and other school administrators receive inadequate training, and school administration and the quality of education suffer as a direct result. Other factors that contribute to poor reading outcomes, and weak learning outcomes in general, include high absenteeism, lack of substitute teachers, sub-contracting of teacher jobs to untrained workers, and periodic removal of teachers from their classrooms by government authorities for activities not related to schooling (e.g., elections, census work). Additional issues that inhibit school performance and ability to achieve grade level appropriate reading ability include poor health and nutrition, a lack of educational materials in the home, and the absence of opportunities to attend a quality pre-school, among others. With a view to addressing the early education challenges, the United States Agency for International Development (USAID) partnered with Centre for Microfinance (CMF) on activity namely, Nurturing Early Literacy Project (NELP). The brief description of the activity is as follows. C.1.1: Description of ‘Nurturing Early Literacy Project’: The ‘Nurturing Early Literacy Project’ is a co-funded project by USAID/India and Tata Trusts. This is a program approved under USAID’s India Partnership Program (IPP), which focused on co￾funded interventions to promote early grade reading. Centre for Microfinance is overall responsible for implementation of project, whereas field level support was provided by Kalike, Bangalore, another Associate Organization of the Tata Trusts and three non-profit partners namely, Pragat Shikshan Sanstha (PSS), Phaltan, Maharashtra, Room to Read India Trust, New Delhi and Bodh Shiksha Samiti, Jaipur. The overall goal of this five-year activity with 9 months extension (October 01, 2015, to June 29, 2021) was to build a strong foundation of emergent and early literacy competencies in more than 100,000 students in selected blocks of Rajasthan, Maharashtra, and Karnataka. The project aimed to achieve the following broad objectives: SPECIFIC OBJECTIVES: 1. Equip teachers with knowledge and skills to improve literacy instruction in classrooms by moving away from rote learning to meaningful engagement with print. 2. Ensure access to age-appropriate, quality children’s literature through classroom/community libraries. 3. Demonstrate a viable model to improve foundational literacy skills in elementary school children and emergent literacy skills in 5+ pre-school children. CORE COMPONENTS OF THE ACTIVITY: 38 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV The core components of the program include the following: 1. Building capacities of: • Government school-teachers - AW facilitators, primary, upper primary school teachers. • Community workers and other staff members supporting intervention school sites. • Administrators and officials at state/district/block level. 2. Improve student learning in five intervention blocks by developing a strong foundation for literacy (including basic numeracy skills) and reading development over 4 years. Inputs in terms of strategies to be adopted under this component will include: • Explicit literacy instruction to build foundational literacy skills in school languages. • Remedial teaching for at-risk pupil studying in primary and upper primary grades. • Establishing structured print-rich environment; and • Implementing first of its kind pilot on tablet-based access to children's literature in multiple formats for reading promotion. 3. Development of contextual and relevant teaching-learning materials for students and teachers • Set up well-functioning libraries in schools and villages • Strengthening of school governance structures through community mobilization • Document and disseminate learning from the project D. Evaluation Design / Methods The evaluation will use both quantitative and qualitative methods for data collection and analysis. The design will include a sample survey to assess the program impact on the beneficiaries and Key Information Interviews will also be conducted to gain insights of perceived triggers and barriers and get insight of the impact of the project on beneficiaries. Prior to conducting field visits for primary data collection, the evaluation team will conduct an extensive review of documents, including, annual work plans, project reports (annual reports and quarterly progress reports), activity monitoring and evaluation plan, and other related documents. The contractor conducting this assessment will gather a wide range of background information from USAID and the Implementing Partner to ensure that the findings and recommendations are based on an accurate understanding of the program, Key Informants’, and beneficiaries’ feedback. The contractor must describe the evaluation design and methodology in detail. E. Deliverables and Timeline The key deliverables of the evaluation are as follows. The contractor must adhere to the timeline of deliverables. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 39 a. Inception report: The inception report must describe the conceptual framework the evaluator will use to undertake the evaluation and the justification for selecting this approach. It must detail the evaluation methodology (i.e., how each question will be answered by way of data collection methods, data sources and sampling). b. Detailed Assessment Methodology: The contractor must describe the methodological approach in detail. The description of the proposed methodology must detail the methods of data collection, sampling strategy / plan, and data analysis plan. c. Debriefing Meeting: The Contractor must debrief USAID on the preliminary findings of the assessment. This meeting must provide a summary of any analytical results; and discuss challenges, successes, and way forward. The Contractor must deliver an oral presentation of the findings. d. Final Assessment Report: The contractor must submit a final report that is based on analyzed facts and evidence and fully addresses all the assessment questions. e. Composition of the Evaluation Team: The contractor must propose a diverse evaluation team expert in different key components of the evaluation. As a means of building local capacity to undertake evaluations, the team composition should have at least one local expert as a key member of the evaluation team. f. Scheduling and Logistics: The Contractor will be responsible for all logistics including coordinating all travel throughout field work, data analysis and report submission to USAID. g. Budget: The contractor must propose the detailed budget aligned with each technical component such as evaluation design, sampling plan, data collection and analysis, evaluation team composing etc. Note: USAID/India will share the necessary documents and reports with the contractor required to conduct the ex-post evaluation. End of document 40 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV ANNEX II: EVALUATION INCEPTION REPORT INTRODUCTION This Inception Report documents the design for the mixed method ex post performance evaluation of the Nurturing Early Literacy (NELP) implemented by the Centre for microFinance (CmF) with funding from USAID and Tata Trusts. This evaluation was designed in compliance with ADS 201, industry best practices, and will provide cost-effective answers to the evaluation questions. This report also includes a description of the methodology the evaluation team will deploy, including a complete sample plan, descriptions of data collection and analysis methodologies, the limitations and potential biases of the methodology, and risk management. It also lays out the evaluation team composition and the timeframe for implementing the evaluation. OVERVIEW EVALUATION PURPOSE This evaluation is guided by its Statement of Work (SOW) which identified the Evaluation Questions (EQs), NELP background, and other requirements. The overall purpose of this ex-post evaluation is to assess sustainability, and to know how the outcomes and impacts evolved after completing the education activity. Additionally, the evaluation findings will provide insights into what worked well and contribute to further learning in the education programming.. EVALUATION QUESTIONS This evaluation will answer the six EQs under the design documented in this Inception Report. Table 1: Evaluation Questions EVALUATION QUESTIONS 1. To what extent and how did the activities accomplish their objectives during implementation? 2. How has the project improved the students’ reading fluency and comprehension during implementation? 3. To what extent the promoted behavior at school and system level has contributed to the early literacy competencies among the target groups? If so, how? If not, what are the factors that have hindered the practice to continue? 4. Have the implementing partners/stakeholders strengthened by USAID funding continued to implement the action? How are the schools, communities and officials continuing to maintain the early literacy system improved by USAID funding? What is the beneficiaries’ perspective / feedback about the impact of the project? 5. What interventions were more successful and/or had a greater contribution to accomplish the objectives? 6. What are the factors that contributed to or impaired the long-term sustainability of the activities’ outcomes and outputs? USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 41 PROJECT BACKGROUND NELP was approved under USAID’s India Partnership Program (IPP), which focuses on co-funded interventions to promote early-grade reading. Centre for Microfinance is overall responsible for the implementation of the project, whereas field level support was provided by Kalike, Bangalore, another Associate Organization of the Tata Trusts and three non-profit partners namely, Pragat Shikshan Sanstha (PSS), Phaltan, Maharashtra, Room to Read India Trust, New Delhi and Bodh Shiksha Samiti, Jaipur. NELP had an original period of performance from October 1, 2015, to September 30th, 2019. USAID granted a nine-month extension due to delays introduced by the COVID pandemic that extended NELP to June 29, 2021. However, the 9 months extension period only focused/worked on the scale-up Library component with the partnership of the Rajasthan State Govt and was implemented by CmF. All Sub-partner activities outside of the Model Library activity in Rajasthan closed in March 2020. NELP aimed to build foundations of emergent and early literacy competencies in more than 93,000 students in selected blocks of Rajasthan, Maharashtra, and Karnataka. Three general objectives contributed to this goal: 1. Equip teachers with knowledge and skills to improve literacy instruction in classrooms by moving away from rote learning to meaningful engagement with print. 2. Ensure access to age-appropriate, quality children’s literature through classroom/community libraries. 3. Demonstrate a viable model to improve foundational literacy skills in elementary school children and emergent literacy skills in 5+ preschoolers. These objectives were themselves contributed to by activities in six core areas. 1. Capacity Development- Building the capacity of key stakeholders to include: a. Government schoolteachers, primary, upper primary school teachers b. Anganwadi workers/ facilitators c. Community workers and other staff members supporting intervention school sites d. Administrators and officials at state/district/block level 2. Literacy Foundation- Improve student learning by developing a solid foundation for literacy and reading development over four years through: a. Explicit literacy instruction to build foundational literacy skills in school languages. b. Remedial teaching for at-risk pupils studying in primary & upper primary grades. c. Establishing a structured print-rich environment. 3. Teaching-Learning Materials- Development of contextual and relevant teaching￾learning materials for students and teachers by: 42 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV a. Implementing the first of its kind pilot on tablet-based access to children's literature in multiple formats for reading promotion. b. Teacher training manuals to assist the transition from rote learning. 4. School Libraries and Village Libraries- Setting up well-functioning libraries in schools and villages by: a. Developing student library committees b. Organizing community libraries with School Management Committee (SMC) input 5. School Governance- Strengthening of school governance structures through: a. Community mobilization around learning outcomes and community libraries in specific blocks. b. Working with SMCs on monitoring progress and barriers towards learning outcomes and the transition from rote to meaning making. 6. Learning- Document and disseminate learnings from the project by: a. Engaging communities around barriers and progress towards key learning objectives. b. Sharing evidence on what works through teacher and school management workshops and other learning events. LITERATURE REVIEW The evaluation team conducted a literature review of relevant studies, research, and expert opinions on transitioning from rote learning to meaning-making and the importance of early-grade literacy outcomes as a foundation for higher-level learning. The literature review highlighted several key constructs that have been integrated into initial evaluation data collection instruments and analysis tools (specifically qualitative codes). These key constructs include: • The prioritized use of oral assessments to measure children’s reading development as a key enabling factor for learning outcomes. • Phonological awareness (alphabet knowledge, phonics, and spelling) as a key predictor for eventual learning outcomes. METHODOLOGY The design of this evaluation builds on evidence that materialized during the implementation of the project as documented in various program reporting and other materials. The design includes factoring in the effects of the COVID pandemic on NELP performance until its final close out in June 2021. While the delay in this ex-post evaluation creates several barriers to producing representative and rigorous results, it does offer the opportunity to focus on those components of NELP that were high performing and have an early indication of sustainment in Rajasthan and Karnataka (the geographic areas that the evaluation will focus on). We will outline the full reasoning and implications of this in subsequent sections of this report. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 43 This evaluation design is also informed by relevant pre-existing evidence on similar theories of change (specifically similar capacity development and learning outcomes) as identified and assessed through a literature review (source material is identified in Annex 3). This evidence is beneficial in the design of the data collection and analysis methods that will be used (for example, using pre￾existing evidence to create the initial qualitative code book). The selection of an appropriate methodology had two other primary inputs, USAID requirements for the evaluation as outlined in the SOW and an internal evaluability assessment. The SOW requires a mixed method performance evaluation, and the evaluability assessment identified optimal sampling, data collection, and analysis approaches, including: • Purposeful Sampling: Given NELP ended approximately 18 months ago for CmF activities and almost 34 months for the remaining sub-partner activities, there are significant barriers in identifying NELP recipients (teachers, SMC members, etc.) who can be respondents for data collection. This attrition is compounded by significant recall bias risks (further elaborated in the Limitations section below). Because of these two risks (attrition and recall bias), after consultations with USAID, the evaluation team decided to 1.) focus primary data collection on Rajasthan and Karnataka and 2.) collect no primary data from students. Focusing data collection in Rajasthan and Karnataka allows for easier identification of respondents as various blocks there are continuing specific NELP components, possibly an opportunity to identify the enabling factors for the sustainability of NELP objectives. The evaluation team and USAID decided to remove students from data collection, given the difficulty in identifying student recipients and the high potential for recall bias given their age. • Best fit qualitative data collection: The evaluation team designed the evaluations’ semi-structured KII’s and FGDs with a mix of closed and open-ended questions. This approach allows for more flexibility in probing for more direct and/or accurate responses (as mitigation for recall bias) and exploring emerging issues (especially in schools where NELP components continue). 44 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV EVALUATION DESIGN MATRIX Table 2 Evaluation Question Data Source/Collection Method Method of Analysis To what extent and how did the activities accomplish their objectives during implementation? ● Project reports (document review) ● NGOs- office bearers (KII’s) ● Implementing Partners (KIIs) ● Early Primary, Upper Primary Teachers (FGDs) ● School Committees (FGD) ● Anganwadi Workers (KII/FGD) Qualitative Content Analysis (Document Review, FGDs, KIIs, Site Observations) Triangulation Indicator Analysis (Descriptive Statistic Analysis) How has the project improved the students’ reading fluency and comprehension during implementation? ● Pre-School, Early Primary, Upper Primary Teachers (KII/FGD) ● Anganwadi Workers (KII/FGD) ● CMF and other Sub-partners reports Qualitative Content Analysis (Document Review, FGDs, KIIs, Site Observations) Triangulation To what extent the promoted behavior at school and system level has contributed to the early literacy competencies among the target groups? If so, how? If not, what are the factors that have hindered the practice to continue? ● Pre-School, Early Primary, Upper Primary Teachers (FGDs and KIIs) ● Implementing Partners (KII/FGD) ● School Committees (FGD) ● School Site Observations Qualitative Content Analysis (Document Review, FGDs, KIIs, Site Observations) Triangulation USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 45 Evaluation Question Data Source/Collection Method Method of Analysis Have the implementing partners/stakeholders strengthened by USAID funding continued to implement the action? How are the schools, communities and officials continuing to maintain the early literacy system improved by USAID funding? What is the beneficiaries’ perspective / feedback about the impact of the project? ● Pre-School, Early Primary, Upper Primary Teachers (FGDs and KIIs) ● Implementing Partners (KII/FGD) ● School Committees (FGD)11 ● Anganwadi Workers (KII/FGD) ● School Site Observations Qualitative Content Analysis (Document Review, FGDs, KIIs, Site Observations) Triangulation What interventions were more successful and/or had a greater contribution to accomplish the objectives? ● Pre-School, Early Primary, Upper Primary Teachers (FGDs and KIIs) ● Implementing Partners (KII/FGD) Qualitative Content Analysis (Document Review, FGDs and KIIs) Triangulation What are the factors that contributed to or impaired the long-term sustainability of the activities’ outcomes and outputs? ● Pre-School, Early Primary, Upper Primary Teachers (FGDs and KIIs) ● Implementing Partners (KII/FGD) ● School Committees (FGD) ● Anganwadi Workers (KII/FGD) ● School Site Observations Qualitative Content Analysis (Document Review, FGDs, KIIs, Site Observations) Triangulation 11 SMC’s tenure is two years from its inception and has to be renewed every two years so there is little chance to enumerate the same SMC members that were involved with NELP. Additionally, SMC activities were only included in the Abu Road Block in Sirochi. Due to this, the evaluation team will assess the ability of current SMC members to speak to NELP activities and any possible effects. 46 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV SAMPLING PLAN NELP implemented activities beyond Rajasthan and Karnataka, but discussions with USAID and CmF have revealed that relevant NELP blocks12 in these two states have continued to implement specific NELP components after the project ended. Hence these two states offer two benefits to the evaluation: 1.) they can offer evidence for enabling factors leading to sustainability, and 2.) they provide more reliable access to NELP recipients as evaluation respondents. Focusing collection on these two states means that the evaluation will not be representative, which is a calculated tradeoff given the resource constraints and timing of the evaluation. Table 3: NELP Sample Frame (Blocks and Anganwadis by State) State Block IP # of Schools # of Anganwadis Karnataka Yadgir Kalike 147 100 Rajasthan Abu Road Bodh 60 60 Rajasthan Pindwara RtR 60 Na Rajasthan Sirochi RtR 40 Na Rajasthan Abu Road CmF 10 Na Total 317 160 12 “Blocks” are units of clustered schools that received NELP assistance within each state. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 47 Table 4: Data Collection Targets Respondents Region Rajasthan Karnataka TOTAL Schools Visited/Site Observations 4 3 7 Primary, upper primary school teachers 12 KII’s (3 per school) 9 KIIs 21 KIIs Anganwadi workers/facilitators 3 KIIs (Abu Road) 6 KIIs 9 KII’s Community workers and other staff members supporting intervention school sites 3 FGD (1 per block) with the community members (see footnote #1) 1 FGD with the community members 6 FGDs Administrators and officials at state/district/block level 3 KIIs 2 KIIs 5 KIIs Implementing Sub Partner Program Managers 1 FGD 1 FGD 2 FGDs TOTAL 30 KIIs/8 FGDs These are the minimum data collection targets as the ET hopes to be able to include additional counts for each method/source of collection if possible. The schools within each block will be randomly selected from the list provided by CmF and the sub-partner IPs. We will select backup schools if the initially selected school is not a feasible option. Specific respondents for each collection method in each school/block will be randomly selected from the lists provided by the sub-partner IP for that specific block in cases with more than three teachers per school. The evaluation team will attempt to contact the respondents through the relvant sub-partner for that block to schedule data collection. If these attempts are unsuccessful, then we will attempt to contact the next candidate on the list. As a last resort, the evaluation team will use referral/snowballing methods to identify additional respondents as needed during field collection. The final result report will contain the full listing of all enumerated schools and their respective blocks, sub-partner IPs, and final data collection counts, accounting for the final sampling methods used. DATA COLLECTION METHODS This evaluation will be a mixed-method performance evaluation comprised of: • Document Review • Key Informant Interviews (KII’s) 48 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV • Focus Group Discussions (FGDs) • School Site Observations • Pause and Reflect Sessions Piloting Plan: The evaluation team will assess all data collection instruments during a peer review process and will pilot them with initial respondents within each cohort for each relevant instrument. After the initial data collection for each instrument, the evaluation team will assess the instrument's performance against standard criteria and make necessary adjustments. Flexible Data Collection: Data collection will allow for findings from previous data collection to inform subsequent collection (referred to as Sequential Data Collection). This enables emerging issues that are identified in earlier data collection to be probed in later data collection. Relevant issues will be identified during routine check ins and spot checks between enumerators during collection. Document Review We will organize the document review into two categories, NELP implementation documentation, and materials collected from headmasters at the school sites. This approach allows the evaluation to better understand NELP performance through triangulating data sources that contribute to answering multiple EQs. Envisioned materials for the document review include: • NELP Implementation Documentation o Annual Work Plans o Project Reporting (Annual Reports and Quarterly Progress Reports) o Activity Monitoring and Evaluation Plan o Other materials as identified during the evaluation that could represent NELP performance in remaining blocks • Documents to be collected from Headmasters o SMC Meeting Minutes (Maximum- two years) (to identify discussion points on early literacy/library etc. o Copy of teacher training schedule and manual o Enrolment Data (Maximum Last two years) USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 49 Key Informant Interviews and Focus Groups The team will perform KIIs and FGDs using the instruments in Annex 2. When possible, a note taker will be used to support the enumerator for both KIIs and FGDs with a priority being given to the presence of a note taker for the FGDs. For each collection instance, the evaluation team will produce notes and summary sheets from KIIs and FGDs. Respondent names will not be recorded, and there will be no audio recording of KIIs or FGDs. While audio recording is optimal, it is the opinion of the evaluation team that it would create a level of apprehension amongst respondents that could affect responses. Given the scope of the KIIs and FGDs, we do not view the lack of an audio recording as a barrier to the integrity of the evaluation. To 1.) advance the participatory nature of the evaluation, 2.) utilize the expertise of the respondents and 3.) improve the quality of data results; enumerators will share summary notes and reflections with respondents immediately after collection to ensure data was captured correctly and to aid in the interpretation of results. The evaluation team will conduct KIIs with teachers and Anganwadi workers using a semi-structured “evolving subject-driven” approach. In this case, “semi-structured " means that the enumerator will use a pre-existing data collection instrument (Annex 2) to integrate evidence that emerges during data collection for deeper dives into issues and quickly identify points of saturation. Enumerators will mitigate recall bias by asking questions specific to NELP activities but then probing to ensure that responses are relevant to NELP. FGDs will follow the same protocol as KIIs for informed consent, semi-structured collection, and data entry. No FGDs will have over five people, and all attempts will be made to ensure respondents are comfortable freely expressing their opinions. This approach means we will ensure gender and cultural norms are adhered to and avoid supervisors being in the same FGD as their supervisees, etc. When an envisioned FGD does not have at least three people a Focus Group Interview (FGI) will be used instead. Site Visits and Observations For selected schools, the Evaluation Specialist will observe classroom operations, as appropriate, to ascertain how NELP influenced teacher practices in transitioning from rote learning to meaning￾making. Observations will also provide evidence on the status of school libraries and the quality, accessibility, and use of learning materials. We will not collect data from students during these site visits and observations. Pause and Reflect Sessions We consider the expertise of USAID, CmF and sub-partner IPs in the various blocks to be valuable inputs for the evaluation. We will conduct Pause and Reflect sessions with these partners for each evaluation deliverable (inception Report, Final Results Report) as part of the dissemination/briefing activities. During these sessions, the evaluation team will solicit input on the adequacy of the evaluation design to answer the evaluation questions (for the Inception Report), interpretation of results, and feasibility of recommendations (for the Final Results Report). After these sessions, the evaluation team will request formal stakeholder feedback that will be documented and included as an annex in the final deliverable per USAID policy (ADS201). 50 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV DATA ANALYSIS METHODS Triangulation – Triangulation will enable the evaluation team to cross-verify and cross-validate findings from distinct data sources to identify correlations between findings related to the six evaluation questions. Methodological triangulation will also enable the evaluation team to strengthen potential linkages and data accuracy in cases where results obtained through one method are less conclusive than another. Indicator Analysis – The only quantitative analysis in this evaluation will be simple descriptive calculations on NELP Key Performance Indicators (KPIs). Quantitative analysis will enable the evaluation team to further examine NELP progress toward targets over time to identify how specific exogenous and endogenous events contributed to these results. Content Analysis– Content analysis will entail the evaluation team’s intensive review of qualitative data to identify and highlight notable examples of NELP’s successes and challenges that contributed to or hindered progress. The evaluation will use standard qualitative analysis methods to include the possible use of software like NVivo. The evaluation team will develop specific qualitative code books for each cohort and type of qualitative collection with standard co-coding, cross-checks and inter-reliability testing practices used. All code books will be made available to USAID upon request. Some data analysis will be ongoing during data collection using check-ins and spot checks to identify emerging themes, gaps, and other issues warranting integration into the remaining data collection. GUIDING PRINCIPLES The evaluation team will operate under the following guiding principles as laid out in the evaluation SOW: • Participation - to ensure that those affected by the project can voice their expectations, experience, learning points, and insights; • Ownership - to ensure that USAID and other key stakeholders own the evaluation process; • Teamwork - to ensure a diversity of approaches and to seek consensus on the fundamental issues; and, • Learning - to provide actionable evidence and integrate the interpretive analysis of USAID, CmF, and other stakeholders to facilitate its uptake into future decision-making. LIMITATIONS AND POTENTIAL BIASES This evaluation has several technical limitations critical in framing how stakeholders could use results. Contribution: No questions of attribution can be answered in this evaluation. At most triangulated mixed method evidence outlining possible NELP contribution to outcomes will be the content of the final results report. Recall bias: This project ended in June 2021, meaning that all data collection will probably encounter recall bias. The evaluation team will attempt various techniques to mitigate this risk (including probing responses for validation, asking the same question differently, and triangulation of USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 51 sources). Still, there is little possibility of completely mitigating this limitation. Additionally, to ensure that NELP activities are being evaluated, respondents may be asked to identify specific activities that they feel contributed to the outcomes they report. Selection bias: Because field data collection is taking place in Rajasthan and Karnataka, where NELP had more success, results will not be representative of overall NELP performance. Outcome Measurement: NELP had one outcome indicator measuring improved student literacy (EG 1.1) collected annually with reported data for two years (2017 and 2018) due to COVID disruptions. This is the primary outcome indicator for NELP. The lack of data, combined with the other limitations outlined above, means the evaluation will not be able to give a summative assessment of outcomes achieved by NELP. At most, the evaluation will contribute to answering the EQs using data triangulated from 1.) qualitative collection in schools where NELP had higher success and 2.) program materials. RISK MANAGEMENT The evaluation will take various steps to mitigate the above limitations and other identified risks to evaluation implementation and quality of results. Availability of Respondents: Given the risk of finding respondents that were recipients of NELP, the evaluation team will over-sample to provide redundant respondent candidates if envisioned respondents are unavailable. We are also factoring in time buffers to allow for any possible snowball sampling that may be required. Fidelity to Evaluation Design: We will closely monitor evaluation tasks per the work plan and methods outlined in this Inception Report. We will document all deviations from this if they occur and provide a justification. If needed, we will document updates to the evaluation design in an Annex before uploading to the DEC. Recall Bias Mitigation: By using more open-ended qualitative instruments that are semi￾structured with the calculated use of probing, enumerators can mitigate some of the effects of recall bias. However, we anticipate that final results will contain some caveats given the length of time since the NELP closeout. Positive Sampling Bias: Because data collection focuses on Rajasthan and Karnataka and USAID and CmF feel that respondents will be easier to access because of NELP success there (given their continuation of various NELP components), final results may not be reflective of overall NELP performance. Mitigating this bias will require careful communication (specifically of limitations) of final results and the use of these results by USAID or other stakeholders. The evaluation team will ensure proper caveats are included around specific results in the final report to facilitate this. ETHICAL PROTECTION AND DATA MANAGEMENT PLAN We have designed our data management plan to comply with ADS 579 and USAID Open Data Policy. Our informed consent protocols are included in all data collection instruments in Annex 2. We will complete all data management from the point of respondent identification, collection, entry, analysis, dissemination, and deletion per relevant USAID requirements as found in ADS 109 and ADS 200. In addition, this Inception Report (including all data collection instruments) was reviewed and approved by Panagora Group’s internal Ethical Review Board under its Human Research Policy. 52 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV • Naming Conventions: No PII will be collected or used in the naming conventions for data files, and data file titles will be formatted as the data source (block/school/respondent category (CMO, etc.), date). • Privacy/Security Protocols on Access: All raw data will be stored in a Google drive, with only the Evaluation Team Lead and Evaluation Specialist having access. Data will be scrubbed and anonymized (including qualitative summary sheets) before being uploaded to the drive. All relevant data will be uploaded to the DDL per the Dissemination section below, at which time all original data will be deleted and the drive erased. USAID can request all data sets at any time and the evaluation team will respond within two business days. • Roles and Responsibilities: The Evaluation Team Lead and Evaluation Specialist will be responsible for respondent identification and data collection. The Team Lead is primarily responsible for ensuring that the data management plan is adhered to for the entire data lifecycle for all collected data. WORK PLAN TEAM COMPOSITION The evaluation team is comprised of two members: Evaluation Team Lead: Michael Cooper will serve as the Evaluation Team Lead and will be responsible for the management and technical oversight of the evaluation. He will lead specific data collection and analysis activities and oversee the technical quality of all deliverables to ensure compliance and best practice while ensuring the work plan is adhered to. He will serve as the primary point of contact with Panagora and USAID. Evaluation Specialist: Dr. Sanjeev Rai is the EGR and Early Education Expert on the evaluation team and will provide input into the evaluation design, lead specific data collection tasks and make inputs into the final report drafting and reporting. EVALUATION TIMEFRAME Table 5: Evaluation Timeline EVALUATION TIMELINE – DELIVERABLE DATES DELIVERABLE/TASK DATE (2023) Draft Inception Report: The Draft Inception Report will be developed in compliance with USAID ADS requirements 1/13 Presentation and Reflection: The Draft Inception Report can be presented to USAID and other stakeholders as part of a Pause and Reflect 1/20 Stakeholder Feedback: Stakeholders will have the opportunity to provide written feedback in addition to the Pause and Reflect 1/23 USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 53 EVALUATION TIMELINE – DELIVERABLE DATES DELIVERABLE/TASK DATE (2023) Presentation and Reflection (if requested): Panagora Group will facilitate another Pause and Reflect to give an overview of how stakeholder feedback was used. All feedback will be documented in Annex 3 of the final Inception Report 1/25 Final Inception Report: The Final Inception Report will document the evaluation design to be implemented 1/25 Draft Final Report: The Draft Final Report will follow the outline in Annex 3 3/10 Presentation and Reflection: Panagora will facilitate a Pause and Reflect on Findings, Conclusions and Summaries with USAID and relevant stakeholders 3/12 Stakeholder Feedback Final Report: USAID and other stakeholders can provide feedback on the Final Report which will be documented in an Annex to the Final Report 3/14 Presentation and Reflection (if requested): If requested, Panagora will facilitate a Pause and Reflect to review Stakeholder Feedback 3/15 Meta-Data, KII and FGD Summary Sheets, Raw Data Sets: Panagora will upload required data sets to the DDL in addition to providing raw data sets and qualitative summary sheets in line with ADS requirements 3/20 54 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV Table 6: Evaluation Timeline Gantt Chart USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 55 EVALUATION STATEMENT OF WORK Scope of Work Ex-post Evaluation of Nurturing Early Literacy Project in India A. Evaluation Purpose Projects may meet their objectives by improving economic, health, or social conditions while they are operating, but genuine success is achieved only through sustained change that does not depend on continued external resources. The overall purpose of this ex-post evaluation is to assess sustainability, and to know how the outcomes and impacts evolved after completing the education activity. Additionally, the evaluation findings will be used to guide the future programs in the education sector. B. Evaluation Questions The contractor must, at a minimum, address the following evaluation questions. 1. To what extent and how did the activities accomplish their objectives during implementation? 2. How has the project improved the students’ learning in terms of basic reading skills? 3. To what extent the promoted behavior at school and system level has contributed to the early literacy competencies among the target groups? If so, how? If not, what are the factors that have hindered the practice to continue? 4. Have the implementing partners/stakeholders strengthened by USAID funding continued to implement the action? How are the schools, communities and officials continuing to maintain the early literacy system improved by USAID funding? What is the beneficiaries’ perspective / feedback about the impact of the project? 5. What interventions were more successful and/or had a greater contribution to accomplish the objectives? 6. What are the factors that contributed to or impaired the long-term sustainability of the activities’ outcomes and outputs? C. Introduction and Background: Despite good student enrollment rates in India (93 percent, according to UNESCO, 2011), the quality of education remains low and reading gains in the early grades are not yet at acceptable levels. Pratham’s Annual Status of Education Report reveals that less than 50 percent of grade five students are able to read a grade two level text, and related statistics from this annual measure make it clear that too many Indian primary school students are not acquiring reading skills at the appropriate rate. As a result, they drop out of school early, or enter adulthood either illiterate or with extremely poor reading abilities. Teacher education is not rigorous enough to effectively prepare teachers for the classroom and includes little or no training in early grade reading pedagogy. Similarly, principals and other school administrators receive inadequate training, and school administration and the quality of education suffer as a direct result. 56 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV Additionally, the schools are not equipped in handling such huge diversity in the student population. The problem is compounded as many children have no prior experience of being in preschools or in any literate environment. Thus, they enter schools without knowing the language in which classroom transactions are taking place, hence impacting their learning. Children who do not make good initial progress in learning to read find it increasingly difficult to master other skills. They also have problems in reading and comprehension at higher levels. Principals and other school administrators receive inadequate training, and school administration and the quality of education suffer as a direct result. Other factors that contribute to poor reading outcomes, and weak learning outcomes in general, include high absenteeism, lack of substitute teachers, sub-contracting of teacher jobs to untrained workers, and periodic removal of teachers from their classrooms by government authorities for activities not related to schooling (e.g., elections, census work). Additional issues that inhibit school performance and ability to achieve grade level appropriate reading ability include poor health and nutrition, a lack of educational materials in the home, and the absence of opportunities to attend a quality pre-school, among others. With a view to addressing the early education challenges, the United States Agency for International Development (USAID) partnered with Centre for Microfinance (CMF) on activity namely, Nurturing Early Literacy Project (NELP). The brief description of the activity is as follows. C.1.1: Description of ‘Nurturing Early Literacy Project’: The ‘Nurturing Early Literacy Project’ is a co-funded project by USAID/India and Tata Trusts. This is a program approved under USAID’s India Partnership Program (IPP), which focused on co-funded interventions to promote early grade reading. Centre for Microfinance is overall responsible for implementation of project, whereas field level support was provided by Kalike, Bangalore, another Associate Organization of the Tata Trusts and three non-profit partners namely, Pragat Shikshan Sanstha (PSS), Phaltan, Maharashtra, Room to Read India Trust, New Delhi and Bodh Shiksha Samiti, Jaipur. The overall goal of this five-year activity with 9 months extension (October 01, 2015, to June 29, 2021) was to build a strong foundation of emergent and early literacy competencies in more than 100,000 students in selected blocks of Rajasthan, Maharashtra, and Karnataka. The project aimed to achieve the following broad objectives: Specific Objectives: 1. Equip teachers with knowledge and skills to improve literacy instruction in classrooms by moving away from rote learning to meaningful engagement with print. 2. Ensure access to age-appropriate, quality children’s literature through classroom/community libraries. 3. Demonstrate a viable model to improve foundational literacy skills in elementary school children and emergent literacy skills in 5+ pre-school children. Core Components of the activity: The core components of the program include the following: USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 57 1. Building capacities of: • Government school-teachers - AW facilitators, primary, upper primary school teachers. • Community workers and other staff members supporting intervention school sites. • Administrators and officials at state/district/block level. 2. Improve student learning in five intervention blocks by developing a strong foundation for literacy and reading development over 4 years. Inputs in terms of strategies to be adopted under this component will include: • Explicit literacy instruction to build foundational literacy skills in school languages. • Remedial teaching for at-risk pupil studying in primary and upper primary grades. • Establishing structured print-rich environment; and • Implementing first of its kind pilot on tablet-based access to children's literature in multiple formats for reading promotion. 3. Development of contextual and relevant teaching-learning materials for students and teachers • Set up well-functioning libraries in schools and villages. • Strengthening of school governance structures through community mobilization. • Document and disseminate learning from the project. D. Evaluation Design / Methods The evaluation will use both quantitative and qualitative methods for data collection and analysis. The design will include a sample survey to assess the program impact on the beneficiaries and Key Information Interviews will also be conducted to gain insights of perceived triggers and barriers and get insight of the impact of the project on beneficiaries. Prior to conducting field visits for primary data collection, the evaluation team will conduct an extensive review of documents, including, annual work plans, project reports (annual reports and quarterly progress reports), activity monitoring and evaluation plan, and other related documents. The contractor conducting this assessment will gather a wide range of background information from USAID and the Implementing Partner to ensure that the findings and recommendations are based on an accurate understanding of the program, Key Informants’, and beneficiaries’ feedback. The contractor must describe the evaluation design and methodology in detail. E. Deliverables and Timeline The key deliverables of the evaluation are as follows. The contractor must adhere to the timeline of deliverables. a) Inception report: The inception report must describe the conceptual framework the evaluator will use to undertake the evaluation and the justification for selecting this 58 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV approach. It must detail the evaluation methodology (i.e., how each question will be answered by way of data collection methods, data sources and sampling). b) Detailed Assessment Methodology: The contractor must describe the methodological approach in detail. The description of the proposed methodology must detail the methods of data collection, sampling strategy / plan, and data analysis plan. c) Debriefing Meeting: The Contractor must debrief USAID on the preliminary findings of the assessment. This meeting must provide a summary of any analytical results; and discuss challenges, successes, and way forward. The Contractor must deliver an oral presentation of the findings. d) Final Assessment Report: The contractor must submit a final report that is based on analyzed facts and evidence and fully addresses all the assessment questions. F. Composition of the Evaluation Team The contractor must propose a diverse evaluation team expert in different key components of the evaluation. As a means of building local capacity to undertake evaluations, the team composition should have at least one local expert as a key member of the evaluation team. G. Scheduling and Logistics The Contractor will be responsible for all logistics including coordinating all travel throughout fieldwork, data analysis and report submission to USAID. H. Budget The contractor must propose the detailed budget aligned with each technical component such as evaluation design, sampling plan, data collection and analysis, evaluation team composing etc. Note: USAID/India will share the necessary documents and reports with the contractor required to conduct the ex-post evaluation. USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 59 ANNEX III: SOURCES OF INFORMATION Data source/collection method Date Government Sr. Sec. School- Chandrawati • Classroom observation • Early primary teacher KII • Library observation 2/2/23 AWC Moongthala 1st • Classroom observation (pre-primary) • AWC observation • AWW KII MGGS Meghvalvas • Classroom observation (e-library and SRG School-RSLPP) • Teacher KII Block Leader CBEO,Pindwara,Rajasthan(Ajay Mathur) CBEO,Abu Road(Mr.Gopal Purohit) APC-1, Sirohi Government Primary School- Bageri • Classroom observation (primary and e-library) • AWW KII • SMC FGD 2/3/23 Khadrafali AWC • Classroom observation (pre-primary) • AWW KII MGGS- Girvar • Classroom observation (Primary) • Teacher KII • SMC FGD Block Leader Room to Read FGD 2/6/23 Govt. Primary School Kharafali, Aamli 60 | USAID/INDIA EX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV Data source/collection method Date • Observation of Language Class (G-1/2) • Observation of Classroom Library • Teacher/ Head Teacher KII (Language and & Library Point Teachers) 2/7/23 AWC Bharja 1st (Scale-up of ECCE by CmF) • Observation of ongoing activities at AWC • KII with AWW & AWH Govt. Upper Primary School Naagpura • Observation of Language Class (G-1/2) • Observation of Classroom Library • KII with Teachers and Head Teacher (Language and & Library • Point Teachers) FGD with Pindwara CDPO/LS and CBEO GUPS– Meghwal Basti Kalandri • Observation of Language Class (G-1/2) • Observation of Library • KII with Teachers and Head Teacher (Language and & Library • Point Teachers) 2/8/23 Govt. Primary School – Tanwari Ward No. 3 • Observation of Language Class (G-1/2) • Observation of Library KII with Teachers and Head Teacher (Language and & Library Point Teachers) HM-Shri Raghu Nath Paliwal Model District Library,Udaipur/ Govt Senior Secondery School,Sector-14, Hiran Magri ,Udaipur 01/02/2023 State Resource Group Members on Library at Rajasthan State Council for Education and Research,Udaipur 01/02/2023 SMC members at village Bageri,Abu Road,Rajasthan 03/02/2023 Mr.Gopal Purohit CBEO-Chief Block Education Officer,Abu Road, Rajasthan 03/02/2023 CDPO-Child Development Project Officer,Abu Road, Rajasthan 03/02/2023 Mr. Ajay Mathur,CBEO, Pindwara,Rajasthan 07/02/2023 Mr. Saurabh Banerjee, CD-RtR via a call 06/02/2023 Mr.Kanti Lal,Academic Project Coordinator(APC),at DPC office, Baggikhana, Sirohi, Rajasthan 08/02/2023 Mr.Yogendra ji, CEO-Bodh Shikshan Samiti, Jaipur, Rajasthan 10/02/2023 USAID/INDIAEX-POST PERFORMANCE EVALUATION: USAID NURTURING EARLY LITERACY PROJECT USAID.GOV | 61 Data source/collection method Date Mr. Lekh Raj Mittal,Dy.Director, Rajasthan Education Initiative, Jaipur, Rajasthan 10/02/2023 Ms. Mallika, Chief Excutive Director, CMF, Jaipur, Rajasthan 10/02/2023 Call with PSS Team 15/02/2023 Members of School Development Management Committee, Upper primar School,Alipur Yadgir, Karnatka 21/02/2023 Mr. Kishore,Director Programme, Kalike at yadgir,Karnatka 21/02/2023 Mr. Shant Gauda Patil, Deputy Director of Public Instruction (DDPI) Yadgir,karnatka 22/02/2023 Mr. Mallikarjun Pujari, Block Resource Centre Coordinator, Yadgir, Karnatka 22/02/2023 Mr.Shiv Kumar, CEO, Kalike, Bengaluru ,Karnatka 24/02/2023 ANNEX IV: DISCLOUSRE OF ANY CONFLICTS OF INTEREST The Panagora Group attests that itself and no member of the evaluation team had any conflict of interest that would prevent them from fulfilling this evaluation in compliance with the USAID Evaluation Policy and ADS 201. ANNEX V: EVALUATION TEAM MEMBERS The evaluation team is comprised of two members: Evaluation Team Lead: Michael Cooper will serve as the Evaluation Team Lead and will be responsible for the management and technical oversight of the evaluation. He will lead specific data collection and analysis activities and oversee the technical quality of all deliverables to ensure compliance and best practice while ensuring the work plan is adhered to. He will serve as the primary point of contact with Panagora and USAID. Evaluation Specialist: Dr. Sanjeev Rai is the EGR and Early Education Expert on the evaluation team and will provide input into the evaluation design, lead specific data collection tasks and make inputs into the final report drafting and reporting. United States Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523