This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the Monitoring, Evaluation and Learning program, DevTech Systems, Inc. NORTHERN EDUCATION INITIATIVE PLUS (NEI+) END LINE PERFORMANCE EVALUATION REPORT FEBRUARY 2021 ACKNOWLEDGEMENTS This document was produced by Fiona Rowand, Team Leader; Dr. Mohammed Ibrahim, Curriculum Expert; Dr. Benedicta C. Agusiobo, Basic Education Expert; Dr. Ganya Adamu, Education Sector Expert; Dr. Arowolo Ayoola Abimbola, Data Analyst Expert; Comfort Kaliyad Boman, Local Evaluation Assistant; and Hadiza Shettima, Local Evaluation Assistant. A Technical Committee composed of USAID/Nigeria’s Education Office (Denise O'Toole, Education Office Director, Elice Elegbe, Deputy Education Office Director, Nura Ibrahim, COR of NEI+ Activity and Olawale Samuel, Alternate COR of NEI+ Activity); Nurudeen Lawal, Chief of Party, and Lydia Ezihe Onuoha, Deputy Chief of Party, of the Northern Education Initiative Plus (NEI+) activity; and Paul Delucco MEL Chief of Party, Salima Mutima, Deputy Chief of Party, Elizabeth Obaba, Senior Education Specialist, Samuel Gyang, Senior M&E Specialist for Analysis, Zuri Linetsky, Glorianna Flanagan, and Jennifer Kennery of DevTech provided support for the discussion and validation during the design, development, and report writing of the evaluation. The evaluation team also thanks NEI+ stakeholders at the Federal, State, LGEA, and community levels for their contributions to this evaluation. We are appreciative of the numerous parents who gave time to offer their views, opinions, and recommendations. The team acknowledges the many classroom teachers and NFLC/AGLC learning facilitators who are at the closest interface with the focal point of this activity: children in schools and non-formal learning centers who are learning to read. Dr. Ganya Adamu and Hadiza Shettima handled photography during the January 2021 fieldwork. NORTHERN EDUCATION INITIATIVE (NEI+): END LINE PERFORMANCE EVALUATION REPORT Draft Report Submitted: February 10, 2021 Second Draft Submitted: February 22, 2021 Final Report to Be Submitted: February 26, 2021 Contract No: AID-602-C-15-0002 Submitted to: Dr. Chidimma Anyanwu, Contracting Officer Representative, MEL Activity, USAID/Nigeria Submitted by: Salima Mutima Acting Chief of Party, MEL Activity DevTech Systems, Inc. Cover Photo: Education continues in focus states despite the COVID-19 Crisis. Credit: Hadiza Shettima, Evaluation Team Member DISCLAIMER The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development TABLE OF CONTENTS ACRONYMS 2 EXECUTIVE SUMMARY 3 INTRODUCTION 8 EVALUATION PURPOSE & QUESTIONS 11 EVALUATION METHODOLOGY 12 FINDINGS AND CONCLUSIONS 15 EVALUATION QUESTION 1 15 FINDINGS 15 CONCLUSIONS 23 RECOMMENDATIONS 24 EVALUATION QUESTION 2 24 FINDINGS 24 CONCLUSIONS 29 RECOMMENDATIONS 29 EVALUATION QUESTION 3 29 FINDINGS 29 CONCLUSIONS 34 RECOMMENDATIONS 34 EVALUATION QUESTION 4 34 FINDINGS 34 CONCLUSIONS 39 RECOMMENDATIONS 39 EVALUATION QUESTION 5 39 FINDINGS 40 CONCLUSIONS 44 RECOMMENDATION 44 ANNEXES 45 ANNEX A: LETTERS CONFIRMING RELEASE OF FUNDING BY STATE GOVERNMENTS 45 ANNEX B: EVALUATION SCOPE OF WORK 50 ANNEX C: EVALUATION WORKPLAN AND DATA COLLECTION TOOLS 66 USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 2 ACRONYMS AGLC Adolescent Girls Learning Center BASAME Bauchi State Agency for Mass Education. BESDA Best Education Service Delivery and Access for All CAI Creative Associates International CBMC Community-Based Management Committee CA Continuous Assessment COE College of Education COP Chief of Party CPD Continuous Professional Development CSACEFA Civil Society Action Coalition on Education for All CSO Civil Society Organization DCOP Deputy Chief of Party EDC Education Development Center EGR Early Grade Reading EGRA Early Grade Reading Assessment EMIS Education Management Information System FME Federal Ministry of Education FMIS Facilitator Management and Information System FOMWAN Federation of Muslim Women’s Associations in Nigeria ICT Information Communications Technology IVR Interactive Voice Responses LEAP Literacy Enhancement Assistance Project LGEA Local Government Education Authority LOI Language of Instruction MTSS Medium-Term Sector Strategy NCCE National Commission for Colleges of Education NEDS National Education Data Survey NEI+ Northern Education Initiative Plus NERDC Nigerian Educational Research and Development Council NFLC Non-Formal Learning Center OOSC Out-of-School Children OSC Overseas Strategic Consulting P (1,2,3) Primary (1,2,3) PTA Parents/Teachers Association RARA Reading and Research Access Activity SAME State Agency for Mass Education SBMC School Based Management Committee SMoE/SMoBSE State Ministry of Education/ State Ministry of Basic & Secondary Education SOW Scope of Work SSO School Support Officer STWG State Technical Working Group SUBEB State Universal Basic Education Board TLM Teaching and Learning Material TMIS Teacher Management Information System TWG Technical Working Group UBEC Universal Basic Education Commission UNICEF United Nations International Children's Emergency Fund USAID United States Agency for International Development 3 EXECUTIVE SUMMARY INTRODUCTION Mu Karanta! Let’s Read! is the foundational Hausa and English curriculum for USAID/NIGERIA’s Northern Education Initiative Plus (NEI+) project funded by the United States Agency for International Development (USAID). The five-year NEI+ project is strengthening the ability of Bauchi and Sokoto states to provide access to quality education—especially for girls, orphans, and children enrolled in non-traditional schools—and to improve children’s reading skills for more than two million school-aged children and youth. DevTech Systems, Inc. (DevTech) with a 12-person evaluation team, conducted the end line Evaluation presented in this Final Report. Creative Associates International, Inc (CAI) is the lead implementing partner for the NEI+ Activity. This report will be used to make recommendations for future programming on USAID early grade reading (EGR) activities. NEI+ is in implementation Year 6, due to end in May 2021, and is expected to demonstrate significant improvement in EGR for approximately 1.6 million children in first to third grades and for more than 500,000 out-of-school children (OOSC) and youth attending some 11,000 Non-Formal Learning Centers (NFLCs), and Adolescent Girls Learning Centers (AGLCs). The COVID-19 pandemic affected education in 2020 and 2021 throughout the two target states. The closure of education centers and the shift to a virtual working environment for many stakeholders meant NEI+ had to act quickly to find creative solutions to safeguard the gains already made by the project and keep activities functioning as well as possible in an environment of uncertainty. EVALUATION METHODOLOGY The evaluation team conducted background desk research using NEI+ project documents and reports from partners and stakeholders. Data collection tools were designed to gather the required information to adequately answer the five evaluation questions presented in the Scope of Work. Data collection tools were key informant interviews and two classroom observation checklists, one for formal schools and one for non-formal learning centers, which were completed by the evaluators while in class and uploaded on Kobo. Preliminary data collection interviews with 233 respondents included 23 interviews (13 males and 10 females) in Abuja, and subsequent virtual field work for Bauchi and Sokoto resulted in 210 key stakeholder interviews (165 males and 45 females) in eight Local Government Education Areas (LGEAs). Thirty-one formal government schools were visited for observations of Primary 1 (P1), Primary 2 (P2), and Primary 3 (P3) classrooms, along with 18 NFLCs. DATA ANALYSIS Qualitative data from the interviews was coded and quantitative data from the classroom observations compiled, and both were analyzed to capture both statistics on NEI+ methodologies being implemented by educators and opinions of multiple stakeholders on the achievements and remaining issues of the Initiative. These were used to produce findings, draw conclusions, and make recommendations for future initiatives. USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 4 METHODOLOGICAL LIMITATIONS ● The changing situation, due to COVID-19 restrictions, shifted the evaluation timeline, and resulted in the selection of schools and centers not being as widespread as initially planned. ● There was not an end line Early Grade Reading Assessment (EGRA) for formal schools, so although respondents gave opinions on current reading levels, this is not backed up by quantitative data. Year 4 EGRA data for NFLCs and AGLCs has been included as a sample of learning gains. ● The evaluation team was not able to access specific financial data from the stakeholders. The team requested documentation to show evidence of funds released and cost-sharing but this was not provided. This means the evaluation contains qualitative views on the sustainability of activities but not specific numbers from the government stakeholders. NEI+ did provide information on numbers of textbooks procured by the states. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: To what extent did the dialect of Hausa, chosen as the Language of Instruction (LOI) in Bauchi and Sokoto target schools, affect learning outcomes in Initiative-affiliated schools? How did the Initiative plan for dialectical variations within target schools? How were any inequities mitigated? The introduction of Hausa as LOI is a popular and effective decision. NEI+ trained formal and non-formal teachers and facilitators in internationally recognized teaching methodologies, in order to build capacity to deliver lively, student-centered lessons. The Initiative produced locally relevant and interesting teaching and learning materials (TLMs), including text books (Mu Karanta! for Hausa, and Let’s Read! for English), teachers’ guides, and teaching and learning aids. NEI+ strategies are very evident in the classrooms in both formal and non-formal centers. Dialectical variations are not a major issue, with teachers using techniques they learn in training to mitigate problems. In general, this area of the project was very successful, with clear evidence of improvement seen in the classrooms. A major issue, which is political and therefore difficult for NEI+ to navigate, is the frequent transfer of teachers, facilitators, and government staff members. CONCLUSIONS Reading standards have improved, with all of the classes where grades were available passing at the level set by the government, and an EGRA conducted by NEI+ for P4 showing gains from the baseline, as a result of NEI+ implemented teacher and facilitator training, and the introduction of strategies and TLMs in Hausa. Dialectical variations do not cause major problems or have an impact on learning outcomes. More support is needed at the government level to ensure teachers, facilitators, and staff members remain in their positions for a minimum of three years after training. RECOMMENDATIONS ● Continuous professional development (CPD) and refresher teacher training courses, based on the NEI+ training programs will ensure that newly employed teachers and facilitators have the same standard of skills as those already trained. 5 ● Close monitoring, mentoring, and coaching to ensure trained teachers are implementing skills learnt from training in classroom delivery will motivate teachers to continue implementing the activities and methodology they learned in their training and ensure lessons remain student centered. ● Work with the federal and state governments to implement policies that ensure teachers, facilitators, and government staff members stay in their positions for a certain period of time. ● Look at ways to make TLMs more durable and replace those that are spoiled or lost. This will make sure that all students have their own textbook and teachers have guides to work from. EVALUATION QUESTION 2: To what extent did “Fidelity of Implementation” factor into programmatic decisions, outcomes, and the overall impact of the Initiative? Fidelity of implementation was factored into the project design but was still an issue at the start of the project. NEI+ worked on this in the second half of the project and included sections in their reports with updates on progress. NEI+ trained, coached, and mentored stakeholders at all levels using systematic, transparent procedures. Community involvement played an important role in encouraging stakeholders to support the initiatives and make sure they were facilitated in accordance with NEI+ and government policies. NEI+ maintained solid communication with all stakeholders throughout the life of the project, adapting it to continue during the COVID-19 crisis. Activities and TLMs are adapted to the local context and are appreciated by the students and community members, and schools and centers are comfortable environments for girls. CONCLUSIONS NEI+ was successful in introducing and maintaining fidelity of implementation in six areas: 1) prioritizing country-focus and ownership; 2) focusing investments on measurably and sustainably improving outcomes; 3) Strengthening systems and developing capacity in local institutions; 4) working in partnership and leveraging resources; 5) driving decision-making and investments using evidence and data; and 6) promoting equity and inclusion. RECOMMENDATIONS ● Continue to build capacity for stakeholders by introducing systems and tools to improve policy through data-driven decision making. ● Improve gender balance of teachers in formal schools by offering training to all teachers and hopefully encourage females to apply as they will be more confident of their abilities and see a structured career path with professional development and mentoring. ● Design and facilitate job- and context-specific soft skills courses for state government officials to improve their capacity, for example: time management skills and planning. EVALUATION QUESTION 3: How well did the Initiative influence factors affecting access to quality education for out-of-school children in target communities? What were the challenges to mainstreaming and retention, and how were they mitigated? NEI+ improved TLMs and trained teachers and facilitators to give interesting lessons designed to make children keen to join their peers in the classroom. The effects are clearly visible and appreciated, with community mobilization and advocacy playing a major role in encouraging parents to value their children’s education. Awareness-building campaigns on the importance of education for girls have been successful USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 6 and, despite ongoing poverty, it is now understood by many parents to have a positive long-term effect economically vis-à-vis than girls staying out of school to sell goods on the street. There are still, however, issues with some parents not being able to afford school uniforms and equipment, and an inadequate number of qualified teachers. CONCLUSIONS NEI+ is viewed as extremely effective in its initiatives for enrollment and retention of out-of-school children in non-formal learning centers and then mainstreaming them into formal schools. RECOMMENDATIONS ● Consider an extension of NEI+ or a project with similar activities related to the attendance, retention, and mainstreaming of out-of-school children. ● Continue and expand advocacy activities to inform the community on the importance of education. ● Continue teacher training programs to keep lessons interesting, which will encourage students to attend. ● Work with the government to hire more qualified teachers. ● State governments should open more NFLCs and AGLCs so that there are more places available where OOSC can study. ● State governments should work with communities, philanthropies, and local businesses on fundraising to supply children with the uniforms and supplies they need to attend learning centers. EVALUATION QUESTION 4: In what ways did the Initiative achieve institutionalization of efficiencies for more effective educational management by state and local education authorities? Where gaps exist, what factors limited progress? To what extent are these activities and activities across all Initiative Result Areas sustainable without USAID investment? Why? NEI+ improved education management systems through developing a policy framework for EGR instruction, improving coordination of education activities, and supporting strategic planning. For financial capacity building, NEI+ was successful in introducing budget planning tools and activities, and training financial officers at state level. Strengthened management of Information and Communications Technology (ICT) was achieved by establishing an Education Management Information System (EMIS) at the state level, and a Facilitator Management and Information System (FMIS) to manage data coming from the non-formal learning sector. Staff were trained by experts to use these systems provided by NEI+. The chances of project sustainability are seen as positive by stakeholders especially in the areas of teacher training, and keeping systems that NEI+ put in place functioning well, through the people who have been trained to use them. CONCLUSIONS NEI+ was effective in strengthening personnel development, through training and mentoring and coaching in school supervision, and teacher support. Systems for finance and ICT/EMIS were strengthened. The capacity building exercises will help ensure sustainability as staff using the new systems are confident in their abilities. 7 RECOMMENDATIONS ● State government education authorities should continue to implement activities and mechanisms, and to provide follow-up on personnel training for those who will continue to manage and sustain the gains recorded in system strengthening so that there is no sudden relapse. ● Continue to maintain healthy relationships with the implementing states to provide further support in education sector budgeting in order to ensure a steady flow of resources to fund activities that will promote EGR instructions. EVALUATION QUESTION 5: To what extent has the Initiative integrated a conflict￾sensitive approach that addressed or reduced the safety concerns of communities and beneficiaries (e.g., school-related violence, gender-based violence, etc.), including but not limited to learners, parents, school-based management committees, etc.? Although NEI+ implemented training programs on safety and security in schools and learning centers, training in peace and avoiding violence, and the introduction of TLMs relating to this topic, the Initiative did not fully address issues of school-related and gender-based violence. Although some respondents mentioned positive gains in this area, the majority of respondents had not heard of any activities related to this and could not pinpoint visible effects in the communities. Issues still to be improved on are more robust community awareness and involvement, youth productivity to keep them away from “bad influences,” and strengthened collaboration with the government to sensitize communities about the causes and effects of violence. CONCLUSIONS NEI+ did facilitate training on reducing violence to community-based organizations, but this did not seem to be reaching to the parents of school children, or the children themselves. More needs to be done to advocate in the community. RECOMMENDATION ● Develop methods to make sure that the NEI+ trainings on gender-based and school-based violence are better known in the community. Ensure follow on to establish how many community members have been influenced by the training and subsequent awareness building campaign. USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 8 INTRODUCTION BACKGROUND OF LOCAL CONTEXT Nigeria currently has the highest number of out-of-school children (OOSC) of primary age in the world, although data on the exact number varies across development agencies. The World Bank reports that Nigeria has nearly 8.7 million OOSC (as of September 2020),1 and UNICEF reports 10.5 million.2 This reporting variance indicates a fundamental weakness: data gaps and inaccuracies that inhibit strategic planning for future school generations and limit the ability for stakeholders to determine where scarce resources should be channeled. Those children attending school are likewise inadequately learning. According to World Bank data, 81 percent of young adults with only a primary education are unable to read.3 Poor quality teaching and learning environments directly correlate with these low achievement levels. Crumbling physical infrastructure, a scarcity of teaching and learning materials (TLMs), and inadequate teacher training and professional development opportunities are drivers of educational failures. Regional disparities exist too, with northern Nigeria states performing well below the national average across education indicators such as attendance and completion rates. The 2015 National Education Data Survey (NEDS), facilitated by the National Population Council, reveals that the average net attendance ratio of the combined North West and North East Regions was 46.6 percent, meaning that approximately 53.4 percent of the total number of primary school-aged children in the north were not attending school. For example, 56.1 percent of children in Bauchi and 68.7 percent in Sokoto had never been to school, compared to the national average of 24 percent. This stark data posed serious developmental challenges, which necessitated the USAID Northern Education Initiatives to address to key important problems of access to education and provision of qualitative literacy education at the primary level. The NEI+ project has built on previous education sector investments by USAID in Nigeria that have sought to assure equitable access as well as improved quality and efficiency of learning. USAID’s Literacy Enhancement Assistance Project (LEAP) 2001-2004, awakened community demand for better education in Nigeria’s primary schools by training and working with parent-teacher associations (PTAs). The Nigeria Reading and Research Access Activity (RARA) was a task order in 2014-2015 designed to carry out research in the areas of early grade reading and access to education in Nigeria. ACTIVITY OVERVIEW NEI+ is a five-year activity funded by USAID/Nigeria, with Creative Associates International as the main implementing partner, to strengthen the ability of Bauchi and Sokoto states to provide access to quality education and significantly improve the reading skills for more than 1 million school-aged children and youth, especially girls, orphans, and children enrolled in non-traditional schools. The Initiative employs a 1 Children Out of School, Primary, World Bank, (September 2020). Available at: https://data.worldbank.org/indicator/SE.PRM.UNER 2 Education, UNICEF Nigeria. Available at: https://data.worldbank.org/indicator/SE.PRM.UNER 3 Learning-Adjusted Years of Schooling (LAYS). World Bank, (2018). Available at: https://documents.worldbank.org/en/publication/documents-reports/documentdetail/243261538075151093/learning￾adjusted-years-of-schooling-lays-defining-a-new-macro-measure-of-education 9 holistic approach, addressing a broad range of critical factors that affect learning, teaching, systems management, parental participation, and community engagement across two core intermediate results. The Initiative focuses on three main areas: 1) strengthening technical and administrative capacity; 2) commitment, and accountability of federal, state, and local government education authorities (LGEAs) to provide effective English and mother-tongue-based EGR teaching and learning to pupils; and 3) deliver literacy, numeracy, and life-skills to OOSC and youths. Whenever possible, interventions strive to mainstream these youth into formal schools for onward learning and social mobility. Regarding capacity strengthening, LGEAs are supported to plan, budget, supervise teachers, deliver high-quality TLMs, and mobilize community involvement for increased enrollment and improved reading outcomes. Colleges of education are also empowered to integrate best practice EGR concepts, techniques, and materials into their curriculum, while playing a key role within the education sector to promote EGR policy and research. Specific to strengthening efficiencies related to teacher payments, the Initiative also integrates the technology of mobile money to regulate payments and reduce absenteeism related to extended long-distance travel of individuals to collect salaries. The Initiative received two technical contract modifications over the life of the project. In March 2018, while maintaining the two core strategic objectives of the Initiative, Modification 4 reduced the number of NEI+ implementation states from three to two and enhanced the focus on government systems strengthening, policy reform, and LGEA accountability and transparency. These modifications were intended to ensure increased sustainability of USAID investments in the two target states. In May 2019, Modification 7 expanded the scope of NEI+ to include development of English language materials for Primary One (P1). The current NEI+ activity focuses on improved reading outcomes and increased access to learning, by strengthening both the administrative and financial management functions of the state entities that oversee and provide education. USAID, through NEI+, is concerned with the capacity of governmental and non￾governmental systems to manage themselves, both administratively and financially. NEI+ is designed to create sustainable education policies, strengthen planning, management, and assessment systems, and build capacity in the two states to deliver and support education systems, with the end goal of increasing access to education and improving EGR outcomes among the learners. The five-year activity strengthens the states’ ability to provide quality education—especially for girls, orphans, and children enrolled in nontraditional schools—and improve children’s reading skills. Partnering with NEI+ are three U.S.-based international organizations—Education Development Center (EDC), Florida State University, and Overseas Strategic Consulting (OSC)—and four local organizations—Value Minds, Association for Education Development Options, Civil Society Action Coalition on Education for All (CSACEFA), and the Federation of Muslim Women’s Associations in Nigeria (FOMWAN). USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 10 The activity works in LGEAs in each target state, with half (10) of the LGEAs in each of the two states supported directly by NEI+ and the other half supported by the state. Criteria for the selection of current target LGEAs include gross enrollment ratio, net enrollment ratio, literacy rates, number of OOSC, number of non-formal education learning centers, percentage of qualified teachers in the state disaggregated by gender and local government, and the number of facilitators in the non-formal education centers. NEI+ and its partners focus on building programmatic ownership among federal, state, and LGEAs, as well as increase their commitment to quality early grade reading instruction and increased access. The NEI+ activity aims to strengthen LGEA’s ability to better train and manage teachers, along with improving their capability to plan, budget, and deliver on professional development, ensure school governance, mobilize community support for learning, and distribute TLMs in a timely manner. KEY PARTNERS AND ROLES Table 1 lists the key partner institutions who participated in this evaluation and their roles. In addition, other key stakeholders and partners include the USAID/Nigeria Education Office, the implementing partner Chemonics International, the state governments of Bauchi and Sokoto, religious and community leaders, village elders, School Support Officers (SSOs), civil society organizations (CSOs), school-based management committees (SBMCs), community-based management committees (CBMCs), and parents. TABLE 1: PARTNER INSTITUTIONS AND ROLES OF EVALUATION PARTICIPANTS STAKEHOLDER ROLE & RESPONSIBILITY Local Government Education Area (LGEA) Implements all programming in both reading and access. Collaborates with NEI+ activity personnel and CSOs to achieve improvements in reading and access. Formal Schools Head Teacher is the school administrator. Class Teacher delivers instruction. SBMC serves as liaison between the community/parents and school. Adolescent Girls Learning Center (AGLC) Non-Formal Learning Center (NFLC) Grantees are CSOs that coordinate a number of centers. Facilitators deliver instruction, and share student attendance and progress with the state education authorities. State Universal Basic Education Boards (SUBEB) Coordinates the development of statewide policies, standards, frameworks, and monitoring tools for both reading and access. Provides financing and logistical support to LGEAs for materials procurement, teacher training, monitoring, and testing. State Ministry of Education (SMOE) Approves statewide policies, standards, frameworks, and monitoring tools for reading and access. Leads the development and approval of appropriate materials for formal schools and non-formal learning centers. Nigerian Educational Research and Development Council (NERDC) Ensures conformity of teaching and learning materials with national primary school curriculum. Universal Basic Education Commission (UBEC) Ensures the allocation of funds for basic education. Acquisition and distribution of textbooks. 11 National Commission for Colleges of Education (NCCE) Develops and approves pre-service coursework designed to prepare new primary teachers. Colleges of Education (COE) Pre-service training and implementer of recently developed pre-service reading instructional program. Federal Ministry of Education (FME) Endorses decentralized efforts to improve reading and access. Incorporates best practices into national-level plans. Approves national policies on education. Best Education Service Delivery and Access (BESDA) Development partners and donors with programs in Northern Nigeria to deliver quality education in the implementing states. Teacher Registration Council of Nigeria Provides registration to professional teachers in Nigeria. EVALUATION PURPOSE & QUESTIONS The purpose of this activity was to conduct an end line performance evaluation of the Northern Education Initiative Plus (NEI+), also referred to by stakeholders as the Initiative. NEI+ is a five-year activity funded by USAID/Nigeria. The period of performance for NEI+ is October 26, 2015 to October 25, 2020, extended to May 25, 2021. The assumption underpinning NEI+ is that if state and local education actors can make education more child-friendly, relevant, flexible, and evidence-based, then greater numbers of children will have access to instruction that improves their reading and life skills. Specifically, the end line performance evaluation: ● Assessed the extent to which the activity met its key objectives. ● Provided a better understanding of the progress made by each component of NEI+ on issues such as relevance, sustainability, and adaptation to local context. ● Identified implementation challenges, unmet needs, and unintended consequences or impacts of NEI+. The end line Performance Evaluation achieved these goals by responding to the following five key evaluation questions at the level of the two NEI+ intermediate results mentioned above. EVALUATION QUESTIONS 1. To what extent did the dialect of Hausa, chosen as the language of instruction (LOI) in Bauchi and Sokoto target schools, affect learning outcomes in Initiative-affiliated schools? How did the Initiative plan for dialectical variations within target schools? How were any inequities mitigated? 2. To what extent did “Fidelity of Implementation” factor into programmatic decisions, outcomes, and the overall impact of the Initiative? 3. How well did the Initiative influence factors affecting access to quality education for out-of-school children in target communities? What were the challenges to mainstreaming and retention, and how were they mitigated? USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 12 4. In what ways did the Initiative achieve institutionalization of efficiencies for more effective educational management by state and local education authorities? Where gaps exist, what factors limited progress? To what extent are these activities and activities across all Initiative Result Areas sustainable without USAID investment? Why? 5. To what extent has the Initiative integrated a conflict-sensitive approach that addressed or reduced the safety concerns of communities and beneficiaries (e.g., school-related violence, gender￾based violence, etc.), including but not limited to learners, parents, school-based management committees, etc.? The audience of the evaluation report will be USAID/Nigeria, specifically the Education Office, implementing partners, key officials from the national, state, local government, and community levels, and other development partners. These documents will synthesize findings and recommendations for future programming. The Government of Nigeria, USAID, and its implementing partners will use the evaluation findings to make decisions for possible follow-on activities. EVALUATION METHODOLOGY DESIGN The evaluation team was comprised of 12 members, including the Team Leader (international), three local Education Sector Experts, a Data Analysis Expert, and two Evaluation Assistants, in addition to support provided by two full-time DevTech staff based in Nigeria, who arranged visits, coordinated logistics, and conducted interviews and class observations during fieldwork, and three full-time DevTech staff based in the United States. The team read and reviewed NEI+ program documents and reports from stakeholders. A list of key stakeholders who could provide relevant information that addressed specific key evaluation questions was developed. Data collection tools were developed for 1) high level policymakers in Abuja and the states, 2) technical staff and representatives (a. Desk Officers, b. School Support Officers, c. Quality Assurance Officers, d. Access and Fragility Officers, e. Reading Specialists, and f. BESDA), 3) ICT/EMIS, 4) logistics, 5) educators, 6) community members, 7) community mobilization, 8) civil society organizations/grantees, 9) women’s groups, 10) NEI+ (Abuja and states), 11) finance and budgeting, 12) technical interview for teachers and facilitators, 13) classroom observation (formal), and 14) classroom observation (NFLCs and AGLCs). DATA COLLECTION Due to the COVID-19 crisis, the evaluators mostly conducted interviews remotely by phone or video communications applications. The team conducted in-person classroom observations in both Sokoto and Bauchi. A mixed-methods approach was used to collect data in Abuja and the two states. The classroom observation tools collected quantitative and qualitative data, whereas structured interviews gathered 13 qualitative data using the tools in the Workplan (Annex C). All tools were submitted to USAID for review prior to fieldwork. In selecting sites for data collection, a key consideration was a variety of school settings: rural and urban schools, formal and non-formal. The implementing partner supplied the team with a list of all schools in which NEI+ is being implemented in the two states. From that list, a random selection was conducted to reduce the list to schools in each state accessible within the constricted timeline. TABLE 2: NUMBERS OF INTERVIEWS AND OBSERVATIONS INTERVIEWS MALE FEMALE TOTAL Abuja 13 10 23 Bauchi 77 28 105 Sokoto 88 17 105 Total 178 55 233 CLASSROOM OBSERVATIONS FORMAL SCHOOLS NFLCS/AGLCS TOTAL Bauchi 13 8 21 Sokoto 18 10 28 Total 31 18 49 DATA ANALYSIS The interviews were recorded, and transcriptions written by the evaluators with the aid of an online transcriber, Otter.ai to ensure accuracy. The audio recordings were used to clean and verify accuracy of transcripts. Quantitative data collected via the classroom observations was analyzed using Microsoft Excel to provide descriptive statistics and evidence of change in key indicators where possible. Qualitative data from the interviews was coded thematically using NVivo, a platform application for qualitative and quantitative analysis, to address each evaluation question. Prior to analysis, the team created a codebook internally to ensure relevance of the coding structure to the data and consistent code application across analysts. EVALUATION LIMITATIONS ● Classroom observations were initially cancelled due to COVID-19 and replaced with a technical interview with teachers and facilitators. On review, it seemed that some schools were still operating and so the classroom observation protocols were re-added to the evaluation. DevTech USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 14 agreed with USAID to add two fieldwork days to capture this data. This meant that time was limited, and the selection of schools and centers was not as widespread as initially planned. ● The latest EGRA data from Sokoto and Bauchi is from the 2018 EGRA conducted by Creative Associates International. To date, there is not a full end line EGRA. This means that, although respondents gave opinions on current reading levels, this is not backed up by quantitative data. The classroom observations gave a snapshot of the activities used, and the atmosphere of the lessons at the time the evaluators were there, but they did not produce data on reading levels. ● The evaluation team was not able to access specific financial data from the stakeholders. The team requested documentation to show evidence of funds released and cost-sharing, but this was not provided. This means the evaluation contains qualitative views from stakeholders on the sustainability of activities but not specific numbers. 15 FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 To what extent did the dialect of Hausa chosen as the language of instruction (LOI) in Bauchi and Sokoto target schools affect learning outcomes in affiliated schools of the initiative? How did the Initiative plan for dialectical variations within target schools? How were any inequities mitigated? FINDINGS IMPACT OF USING HAUSA AS THE LANGUAGE OF INSTRUCTION According to the NEI+ Y4 NFLC report, the Initiative conducted an EGRA to determine Y4 learning gains and final performance levels in Hausa basic reading skills in NFLCs and AGLCs in both Bauchi and Sokoto. Data collection for the pre-test was conducted from July 9-12, 2019 in Bauchi and July 16-19, 2019 in Sokoto. For the post-test, data collection was conducted from March 10-13, 2020 in Bauchi and March 19-22, 2020 in Sokoto. For the pre-test, 1,599 students were tested, (799 in Bauchi, and 800 in Sokoto), and for the post-test, 1,590 were tested, (794 in Bauchi, and 796 in Sokoto). The assessment tested five sub-tasks, (Letter Dictation (letters written correctly), Reading Comprehension (questions correct), Oral Reading Fluency (words correct per minute), and Syllable Identification (syllables correct per minute)). The table below shows the mean scores for the five sub-tasks in both states, in NFLCs and AFLCs combined, disaggregated by gender. TABLE 3: MEAN SCORE SUBTASK RESULTS FOR ALL CENTER TYPES AT PRETEST AND POST-TEST, BY GENDER SUBTASK BAUCHI SOKOTO PRE-TEST POST-TEST PRE-TEST POST-TEST Male Female Male Female Male Female Male Female Syllable Identification 2.1 3.4 15.5 15.9 2.2 1.7 15.1 13.6 Familiar Word Reading 1.3 2.2 10.1 10.9 1.4 1.1 9.2 9.0 Oral Reading Fluency 1.6 2.5 11.7 12.9 1.6 1.2 10.5 10.1 Reading Comprehension (out of 5) 0.1 0.2 0.8 0.9 0.1 0.1 0.7 0.6 Letter Dictation (out of 15) 2.3 2.5 6.3 7.0 1.8 1.4 6.3 5.8 At pre-test and post-test, no significant gender gaps were found in Bauchi or Sokoto for any subtask, except for familiar word reading in Bauchi in the pre-test, in which girls scored higher than boys; still, the difference is small. The percentage of learners meeting the Oral Reading Fluency standard increased from 4.7% in Bauchi and 2.7% in Sokoto at pre-test to 24.4% in Bauchi and 18.6% in Sokoto at post-test. Similarly, the percentage of learners meeting the comprehension benchmark increased from 3.8% in Bauchi and 1.9% in Sokoto at USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 16 pre-test to 23.1% in Bauchi and 16.1% in Sokoto at post-test. While this result indicates that 75%- 80% of learners need further progress to attain the expected P2 levels, it is important to note that their achievement as a whole is similar to that of formal school pupils tested at the end of P2. At pre-test, 80.0% of learners in both states were unable to read any of the passages. At post-test, the percentage of zero scores dropped to 49.4% in Bauchi and 69.9% in Sokoto. The evaluation team obtained Continuous Assessment (CA) and Final Exam data from November and December 2020, for 18 formal school classes (6 in Bauchi and 12 in Sokoto). The grades in table 4 are the average percentages from all students in each class combined (disaggregated by males and females). Although this is a small sample, and the school grades are not directly comparable to an EGRA, as the items tested are not identical, the table shows that all of the classes analyzed are passing with at least the 40% standard set by the government, with many classes getting an average higher grade. Taking into account the months lost dues to Covid-19 school closures, this is a good sign that learning outcomes have been positively influenced. The percentages are converted to grades as follows: Percentages for Letter Grade and Classification A: 70% - 100% (Distinction) D: 45% - 49% (Merit) B: 60% - 69% (Upper Credit) E: 40 - 44% (Pass) C: 50% - 59% (Credit) F: 0% - 39% (Fail) TABLE 4: FORMAL SCHOOL GRADES/PERFORMANCE OF STUDENTS IN HAUSA IN SOKOTO AND BAUCHI STATES BAUCHI STATE SCHOOL AND CLASS LEVEL NUMBER OF STUDENTS CONTINUOUS ASSESSMENT - 40% FINAL EXAM - 60% FINAL GRADE - 100% Males Females Males Females Males Females Males Females ALKALERI LGA - BAYAN BANKI PRIMARY SCHOOL P1-A 26 22 24 36 45 46 69 82 P2-A 26 24 26 26 38 31 64 57 ALKALERI LGA - CENTRAL PRIMARY SCHOOL P3-B 15 28 28 24 23 28 51 52 GANJUWA LGA – KWANA LABI PRIMARY SCHOOL P1-A 32 18 16 18 38 34 54 52 GANJUWA LGA - KAFAR BURI PRIMARY SCHOOL P2-A 22 27 30 26 41 37 71 63 P3-B 13 22 22 19 49 41 71 60 SOKOTO STATE SCHOOL AND CLASS LEVEL NUMBER OF STUDENTS CONTINUOUS ASSESSMENT - 40% EXAM - 60% FINAL GRADE - 100% Males Females Males Females Males Females Males Females DANGE LGA - BAKIN KASUWA MODEL PRIMARY SCHOOL P1-A 36 41 23 23 18 28 41 51 P2-A 37 24 20 24 20 28 40 52 SHAGARI LGA - GANGAM PRIMARY SCHOOL P1-A 34 30 18 25 50 24 68 49 P3-A 46 30 24 22 32 19 56 41 SHAGARI LGA - MAGAJI BELLO PRIMARY SCHOOL P1-A 54 36 30 31 29 31 59 62 P3-A 51 43 25 23 29 31 54 54 SOKOTO SOUTH LGA – MANA KARAMA AREA PRIMARY SCHOOL P2-A 22 29 08 12 34 37 42 49 17 SOKOTO SOUTH LGA -MANA 500 HOUSING ESTATE ACADEMY NURSERY AND PRIMARY SCHOOL P2-A 37 37 17 13 40 34 57 47 WAMAKO LGA - SABILILLAHI NURSERY & PRIMARY SCHOOL BADO QUARTERS P1-A 12 20 29 23 14 16 43 49 P2-A 14 12 30 31 54 45 84 76 P3-A 17 24 19 25 33 46 52 71 YABO LGA - SARKIN KEBBI SHEHU PRIMARY SCHOOL P2-A 58 23 22 27 18 21 40 48 MAINSTREAMING NEI+ has been successful in mainstreaming students from NFLCs and AGLCs into formal primary schools. The evaluation team obtained results from four LGAs in Sokoto and four in Bauchi. Tables 4 and 5 below show the numbers of students mainstreamed from cohorts 1-4 in both states. TABLE 5: SOKOTO STATE TOTAL LEARNERS MAINSTREAMED TO NEAREST PRIMARY SCHOOLS FROM COHORT 1 – 4 (2016-2021) IN 5 LGAS LGA LGA TOTAL NUMBERS OF LEARNERS MAINSTREAMED IN THE LGA TO DATE TOTAL NUMBER OF LEARNERS MAINSTREAMED BY CENTER TYPE TOTAL NUMBER OF LEARNERS MAINSTREAMED BY GENDER NFLC AGLC MALE FEMALE 1 Dange Shuni 8,682 6,122 1,315 4,347 4,338 2 Sokoto South 3,835 2,331 1,504 2,241 1,594 3 Shagari 3,112 1,612 1,500 1,561 1,551 4 Wamakko 6,608 2,479 1,1614 1,380 1,117 5 Yabo 6,507 3,985 2,412 2,736 3,771 TABLE 6: BAUCHI STATE TOTAL LEARNERS MAINSTREAMED TO NEAREST PRIMARY SCHOOLS FROM COHORT 1 – 4 (2016-2021) IN 5 LGAS LGA LGA TOTAL NUMBERS OF LEARNERS MAINSTREAMED IN THE LGA TO DATE TOTAL NUMBER OF LEARNERS MAINSTREAMED BY GENDER MALE FEMALE 1 Shira 326 123 203 2 Misau 259 164 95 3 Toro 1,073 678 395 4 Bauchi 60 30 30 5 Itas Gadau 556 234 322 NEI+ schools and centers use Hausa as the language of instruction in P1 – P3. Fifty respondents answered questions about the impact of this (18 teachers and facilitators, ten School Support Officers, eight Quality Assurance Officers, four Access and Fragility Officers, four Reading Specialists, and six CSO representatives). All of them said that introducing Hausa as the language of instruction was a good decision, USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 18 and that it had a positive effect on learning outcomes. The main reason they gave is because it is easier and faster for students to learn to read and write in a language they already know how to speak. A Headteacher from Sokoto said, “They speak Hausa already; all we have to do is teach them to read which they grab quite fast.” An Access and Fragility Officer from Bauchi said, “One of the impacts it has is that when a child is taught in his mother tongue, he easily grasps it. That is also a credit to the project. Teaching them in their mother tongue, it enables them to learn faster and easier.” A headteacher from Bauchi said that using Hausa means students can read much sooner. “The impact Hausa language has had is evident. Before NEI+ came, a child would get to P3 without the ability to read a word. Now we have children reading fluently from P1. So, the progress being made is evident.” Another reason given was that families can help the children read at home. A Headteacher from Sokoto said, “The Hausa books given to the pupils have helped, they are even able to get assistance from their siblings at home. The pupils’ capacity to read has greatly increased because of the use of Hausa language.” A School Support Officer (SSO) from Sokoto said, “The parents are more involved in the learning of their children. Hausa is the mother tongue so learning to read and write becomes easy.” Twelve of the respondents in the categories above also mentioned that learning to read and write in Hausa first will make English easier to learn later. An SSO from Sokoto said, “It is easier to transition to learning English language when they already know how to read in Hausa.” A CSO representative from Bauchi said, “The beauty of the whole thing is that at the end of six months, when many of them have mastered reading Hausa, you will now begin to see that passion to read English because they see English everywhere in the community.” The Hausa reading program is also appreciated by parents, who say they can see the impact. A parent from Bauchi said, “We have seen immense improvements in our children’s reading and writing abilities. The teachers are doing well and the children have learnt a lot.” Another parent, in Sokoto, said, “The community members say that they have never experienced such an intervention. Now children can put letters together to form meaningful words, they can read and write. Education has been brought closer to us without us having to travel.” IMPACT OF TEACHER TRAINING AND TLMS ON LEARNING OUTCOMES Teacher Training Programs According to NEI+ project reports, the Initiative conducted various trainings for teachers and facilitators to build their capacity in student-centered teaching strategies. This, in turn, has an impact on attendance and retention. Students are more likely to attend if they are interested, and the materials in Hausa, which are related to their local environment and traditions, encourage them to learn. Teachers were trained in “We didn’t look for NEI+ but they came looking for us, and found us. They enabled our children to learn and become meaningful in the society. For that I say thank you.” Parent of child in an AGLC Sokoto A class at an NFLC studying outside. 19 how to use the Mu Karanta! textbooks and teachers guides, which contain communicative activities and encourage the use of group and pair work. Out of 45 teachers and facilitators who spoke about their training, all were pleased with the training and said it had positively affected them as teachers and their learners. Teachers said the techniques they learned and are using them in their classrooms include introducing vocabulary using items available in their environment, pair and group work, the gradual release approach (“I do, we do, you do,” where a teacher introduces a new concept to the whole class, then they practice together, and then in pairs or small groups), using songs and games, and reading comprehension activities. The Teacher’s Guide has everything the teacher needs to teach, and is a scripted lesson guide that can be followed step by step. When asked how the techniques have affected teaching, a formal school teacher from Bauchi spoke about the usefulness of the teacher’s guide. “The aspect I enjoy most is the fact that I follow the guide step by step.” Another formal school teacher said, “Actually I have learned a lot. Like peer work. I did not know it before. It helps students to understand the lesson, as they learn in groups, they have a better understanding of the topic.” An NFLC facilitator in Sokoto spoke about a change in how they motivate students through encouragement, not by being strict. “We were trained on how to motivate learners to learn gently, not by coercion but through encouragement that will arouse their interest to learn.” Observations in Formal Schools, NFLCs, and AGLCs The Evaluation Team observed a total of 49 lessons: 31 in formal schools (13 in Bauchi and 18 in Sokoto), and 18 in NFLCs and AGLCs (eight in Bauchi, and ten in Sokoto). Table 7 shows the locations and number of lessons that the evaluation team visited. TABLE 7: SCHOOL/CENTER OBSERVATION LOCATIONS AND NUMBER OF SCHOOL/ CLASSES OBSERVED IN EACH BAUCHI SOKOTO LGEA Formal NFLC AGLC LGEA Formal NFLC AGLC BAUCHI 2 schools 3 classes 1 class 1 class WAMAKKO 1 school 3 classes 2 classes - ALKALERI 2 schools 2 classes 1 class 1 class DANGE SHUNI 1 school 3 classes 3 classes - GANJUWA 2 schools 4 classes 1 class 1 class YABO 2 schools 6 classes 2 classes - TORO 2 schools 4 classes 1 class 1 class SHAGARI 2 schools 6 classes 3 classes - Table 8 shows the gender breakdown of teachers and facilitators in attendance in schools and centers, in comparison to the 2020 government census data from both states conducted by the Department of Planning Research and Statistics. TABLE 8: GENDER OF TEACHERS/FACILITATORS FORMAL SCHOOLS SUPPORTED BY NEI+ NFLCS & AFLCS SUPPORTED BY NEI+ LGEA CENSUS DATA BAUCHI SOKOTO BAUCHI SOKOTO BAUCHI SOKOTO Male Female Male Female Male Female Male Female Male Female Male Female 15% 85% 33% 67% 50% 50% 90% 10% 66% 34% 79% 21% USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 20 Although the observations were of a small sample, if they are representative as a whole, they show that NEI+ has a much higher rate of females employed as teachers in formal schools than the average in government schools. Formal schools supported by NEI+ in both states actually have a gender balance in favor of female teachers. For non-formal centers supported by NEI+, Bauchi has an equal number of male and female facilitators, while Sokoto still has a large imbalance in favor of males. Comparing this to the government statistics for formal schools, which have a much higher percentage of male teachers, could be an indicator that NEI+ has been successful in raising the rates of female teachers and facilitators except for in non-formal centers in Sokoto. Table 9 shows the percentages of students enrolled in P1-P3 in NEI+ supported schools and non-formal learning centers using data from the classroom observations, compared to the government census data. TABLE 9: GENDER OF STUDENTS ENROLLED IN P1-P3 FORMAL SCHOOLS SUPPORTED BY NEI+ NFLCS & AFLCS SUPPORTED BY NEI+ LGEA CENSUS DATA BAUCHI SOKOTO BAUCHI SOKOTO BAUCHI SOKOTO Male Female Male Female Male Female Male Female Male Female Male Female 46% 54% 52% 48% 38% 62% 57% 43% 53% 47% 56% 44% Government schools have a quite good gender balance and this is reflected in the NEI+ supported schools. For non-formal centers, NEI+ has a higher number of females which can be attributed to the opening of female-only AGLCs. Table 10 shows the results of the classroom observations relating to teacher and facilitator manner, student engagement, methodology used in the classes, and TLMs. TABLE 10: RESULTS FROM CLASSROOM OBSERVATIONS THEME FORMAL SCHOOLS NFLCs and AGLCs TEACHERS/FACILITATORS Engaged 78% 94% Encouraging 75% 94% Confident 75% 100% STUDENTS Enthusiastic 81% 100% Following instructions 83% 100% METHODOLOGY NEI+ strategies used 83% 94% Strategies adapted to local context 69% 83% TLMs Mu Karanta! books used to teach lesson 83% 72% Students with their own copy 53% 72% Use of word cards 36% 11% Posters on wall 22% 6% 21 Teachers and facilitators in both states were engaged, encouraging, and confident, and students enthusiastic and following instructions. NEI+ strategies were used in a high percentage of classes. Most lessons were adapted to the local context, more so in Sokoto than Bauchi, with teachers in Sokoto using more teaching aids and examples related to their environment. Mu Karanta! TLMs were used frequently in both states. Not all students had their own copy of textbooks, with just under half in Bauchi having to share. Most of the teachers observed used the TG and followed the step-by-step instructions in the scripted lessons from beginning of the lesson to the end. Use of word cards and posters was limited, perhaps due to lack of materials, or perhaps because teachers and facilitators do not always have their own classrooms and so do not want to leave them in one place. In general, NEI+ classes have a good atmosphere and strategies learned in the trainings are used. PLANS FOR AND MITIGATIONS OF DIALECTICAL VARIATIONS The Initiative planned for dialectical variations when designing the TLMs. Reading Specialists in the two states explained how the process worked. A specialist from Bauchi said, “NEI+ organized a series of workshops between Bauchi and Sokoto, which brought a good number of educationists with track records who sat down to look at dialect issues. They struck a deal so that both sides could benefit.” A specialist from Sokoto said, “I know during the material development we had a composition of people from both states, and that helped a lot in trying to include both the dialects. So actually, there is no issue of dialects when it comes to the content of Mu Karanta!” None of the seven Reading Specialists saw the differences as a problem. As a respondent from Sokoto said, “There was never a complaint either from the pupils or from the teachers or from the parents on the use of the dialect. Everything has been going on smoothly without problem.” The opinion was similar in all locations, including in rural schools. As for how the TLMs work in the classrooms, interview respondents said that although there are differences between dialects in Hausa, and the materials contain elements from standard Hausa that sometimes differs from those used in the states, which is sometimes confusing, it does not have a big impact on learning outcomes. All but one of the 20 teachers and facilitators who answered this question and said they have students with varying dialects in their classes, in both states, said that dialectical variations are not a big challenge, and that they either translate or explain different words to their students. A formal school teacher from Bauchi explained how they deal with this, “There are times where I have to translate or make further explanations of vocabulary that is not clear to the children. I rephrase the sentence to bring out the original meaning of the word.” A formal school teacher in Sokoto said that some students require extra help but it is not a big issue. “I read out the text to the students exactly as it is written in the material, then I explain the differences and tell them they can use our Hausa instead of the one in the materials since the differences are not much.” Three teachers and facilitators said that they make sure to plan ahead so that they are aware of any problems that may come up in the lesson, and if they are unsure, they ask colleagues what a word means. A formal school teacher in Bauchi said, “If there is anything I don’t know before I go to class, I normally USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 22 ask other people who are familiar with the dialect to explain some difficult words to me. I then go to the class and explain to my children.” One formal school teacher in Bauchi said that their students find the differences funny, as some words have different meanings in their dialect. “Yes, there are very hilarious differences that tend to confuse the learners, but we make effort to explain such varying terms in our dialect. As they grow and learn, they discover the differences by themselves.” An NFLC facilitator in Sokoto sees the differences as a learning experience, “These are new words we facilitators and learners have got and are very excited about and want to learn more.” The one formal school teacher who said dialectical variations are problematic said “We have issues with pronunciation and vocabulary. It is a challenge for me and my students.” Four teachers and facilitators said they do not have any students in their classes who speak a different dialect and so there is no issue. Something that is an issue, which is outside the scope of this report, is that many classes have students in them who have Fulani as their first language and so they have problems with Hausa. Some teachers and facilitators said they also speak Fulani so they translate, and those who do not said they explain with gestures. CHALLENGES AND ISSUES NOT ADDRESSED Teachers, Facilitators, and Staff Being Transferred Too Frequently People being transferred frequently is a problem. This was mentioned in 18 interviews with stakeholders and NEI+ staff (SUBEB Permanent Secretary, an LGEA Executive Secretary, an SMBC member, a Reading Coordinator, a School Support Officer, and a Headteacher, and from NEI+, the COP, DCOP, three systems strengthening and EMIS officers, three ICT officers, two State Team Leads, an Access and Fragility Officer, and a Reading Specialist). The main issue this causes is that the project invests time and money in training teachers and then does not get the benefit of their skills in the classroom. An SBM C member in Bauchi said, “Some teachers are transferred to other schools after being trained and that creates a gap within the school.” The LGEA Executive Secretary in Sokoto said, “We faced the problem of rapid transfer of teachers who have the training, teachers within the service of our LGEA. So, after we train the teachers, they go into another employment then that leaves us with a gap.” The problem is not only with teachers but with staff at the various levels of government administration. The NEI+ Access and Fragility Officer in Bauchi said, “We have trained more than 200 staff of BASAME at the community and LGEA level. After the project trains them, there is usually a massive transfer of staff. Then you have to look for another person to fill that role.” NEI+ has tried to mitigate this, with appeals to the government to keep staff members in the same position. The NEI+ DCOP said, “We had discussions with the Commissioners of Education and other key stakeholders in the two states and made a passionate appeal not to transfer people. The states listened, but not 100 percent.” Respondents said that frequent transfers are a political issue. The NEI+ Systems Strengthening Officer in Bauchi said, “It is political interference in the education system. They won’t talk to us, at our level.” The NEI+ COP said, “We need Students using their Mu Karanta! books. 23 to engage with the government. Sometimes they will tell you that it’s not your business, you are here to run a project, forget what goes on in our politics.” Inadequate Number of Qualified Teachers and Large Class Sizes Respondents frequently praised the interesting teaching methodology implemented by NEI+. The improved way of teaching has highlighted the disparity between trained teachers and those who have not attended courses, and so the communities would like to see higher numbers of qualified teachers in the classrooms. A SUBEB representative from Sokoto spoke about the need for more CPD. “Strengthen the training of teachers and then strengthen also monitoring, mentoring, coaching.” There is currently a gender imbalance, with many more qualified male teachers than female in the state system. Table 11 shows data from the 2020 census reports from Bauchi and Sokoto. TABLE 11: NUMBER OF QUALIFIED TEACHERS IN LGEA FORMAL SCHOOLS BAUCHI SOKOTO TOTAL MALE FEMALE MALE FEMALE MALE FEMALE 85% 37% 63% 24% 74% 43.5% A community member from Sokoto said that the government needs to take responsibility for hiring teachers. “The government needs to make more efforts in employing more teachers. You will find a school that has 500 pupils and two teachers. These two teachers cannot handle the children effectively. So, the government must be ready to employ teachers to deal with this issue.” They also mentioned that peer teaching should be encouraged, which would help in some respects, but is not an adequate replacement for teachers versed in methodologies designed to maximize learning results. Deterioration of TLMs Many respondents complained about issues with keeping TLMs in good condition. A community member in Bauchi said, “In terms of reading materials, something has to be done. These materials spoil easily because they have been bound into a single book. The pupils spoil them easily so they need to bind them properly since they are to be used for one whole session.” The books were changed to include two terms rather than one and the bindings don’t last, so the books fall apart. There will be a negative impact on the quality of lessons if the government does not have the ability to replace lost or spoiled books. CONCLUSIONS The decision to implement Hausa as LOI has had a positive impact on learning outcomes, with gains seen in test results and with a high percentage of students mainstreamed into the formal education system. Dialectical variations are not seen as a big issue in the classes where there are students with different dialects. The TLMs were designed by a group of experts from both states and a compromise reached. Teachers and facilitators are comfortable and have techniques to mitigate any problems that arise. The TLMs are locally relevant and interesting, although they deteriorate quickly, and classes are interesting, and student centered. NEI+ strategies are very evident in the classroom in both formal and non-formal centers. USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 24 Transfer of teachers and facilitators and other education staff, after training, causes gaps in the system and an inability to utilize their skills. These decisions are made at a political level and it is difficult to reverse them. RECOMMENDATIONS ● Continuous professional development and refresher teacher training courses, based on the NEI+ training programs will ensure that newly employed teachers and facilitators have the same standard of skills as those already trained. ● Close monitoring, mentoring, and coaching to ensure trained teachers are implementing skills learnt from training in classroom delivery will motivate teachers to continue implementing the activities and methodology they learned in their training and ensure lessons remain student centered. ● Work with the federal and state governments to implement policies that ensure teachers, facilitators, and government staff members stay in their positions for a certain period of time. ● Look at ways to make TLMs more durable and replace those that are spoiled or lost. This will make sure that all students have their own textbook and teachers will have guides to work from. EVALUATION QUESTION 2 To what extent did “Fidelity of Implementation” factor into programmatic decisions, outcomes, and the overall impact of the Initiative? According to the USAID Education Policy Program Cycle Implementation and Operational Guidance (2020),4 fidelity of implementation covers six main areas: 1) prioritize country-focus and ownership; 2) focus investments on measurably and sustainably improving outcomes; 3) strengthen systems and develop capacity in local institutions; 4) work in partnership and leverage resources; 5) drive decision-making and investments using evidence and data; and 6) promote equity and inclusion. This section will discuss NEI+ initiatives in these areas. FINDINGS PRIORITIZE COUNTRY-FOCUS AND OWNERSHIP According to NEI+ project reports, the Initiative collaborated and cooperated with government entities and local organizations in the planning stages and roll out of all activities. This included working with federal and state education authorities, CSOs, CBMCs and SBMCs, women’s groups, SSOs, local religious and traditional leaders, headteachers, teachers and facilitators, and, of course, the students who attend the schools and centers. 4 Education Policy Program Cycle Implementation and Operational Guidance, USAID, April. Available at: https://www.usaid.gov/sites/default/files/documents/1865/USAID_Education_Policy_Program_Cycle_Implementation _and_Operational_Guidance_FINAL.pdf 25 Government organizations at the state level are encouraged to take ownership, through being empowered by training and the introduction of transparent systems. A Desk Officer from NCCE said, “We have been involved in the pre-service program for teachers for the past four years, the pilot of the program, and as of now, we are now taking a step further toward ownership of the program by inputting the EGR into our NCCE minimum standards, the minimum standards. NEI+ is helping us do this.” A BESDA representative spoke about the involvement of government education authorities, saying, “We make sure that the state government has total ownership of the program. That way you can see who is committed to the job. NEI+ has made sure they use systems to track finances. That is to show how much they're spending on education, so they cannot just allow that money to go down the drain.” According to NEI+ reports, states received support in developing their respective workplans with a focus on documenting best practices along with fidelity of implementation, development of implementation guide, and performance monitoring plans. Throughout the project, NEI+ used community mapping activities to consult with partners and stakeholders in local communities in order to implement activities relevant, needed, and wanted by the community. For example, according to NEI+ reports, the Initiative used mobile technology to map and collect data through digital forms for surveys, focus group discussions, and geo￾mapping. This data was used to make evidence-based decisions on where to support schools and open NFLCs and AGLCs. Several respondents in both states mentioned that NEI+ had been successful in finding out what the community felt would work through this exercise. A Community Mobilization Officer in Bauchi said “It has been very workable and very successful because it was well planned out with relevant stakeholders from all works in line within the local communities.” This mapping exercise meant that the Initiative was able to pinpoint those most in need and implement activities in those areas. Parents and those on school committees are involved through meetings designed to get feedback and solve any issues. Of the 35 community members who responded to questions on ownership, 33 of them felt that they were involved in the project, listened to, and had a degree of ownership of initiatives and activities (13 parents, 12 SMBC members, six CBMC members, three women’s leaders). A parent from Bauchi said, “Definitely we were given that opportunity because it was, we, the parents, that mobilized the community.” An SBMC member in Sokoto said, “We attended many trainings. We were taught how to ensure that things are running smoothly in the schools, and also encouraged to take ownership.” Three CSO representatives also spoke about how NEI+ involved the community, saying, “People are taking ownership. Organizations that have been empowered are taking ownership.” The two parents who said they had not been supported felt that NEI+ had not reached out to them enough. All NEI+ activities were well adapted to the local context. The TLMs were designed to include topics that the students can relate to, and advocacy messages were tailored to be sensitive to local culture. An SMBC member in Bauchi said that the activities were well suited to the environment, “I can say they try to adopt our ways through role plays, skits that are in the books, and the other activities they do in school.” There is also sensitivity to the local religion and language. A CBMC member from an AGLC in Sokoto said, “The training at the center is in Hausa and the lessons respect our religion. The Initiative aligns with our culture and tradition.” Teaching written and spoken syllables. USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 26 FOCUS INVESTMENTS ON MEASURABLY AND SUSTAINABLY IMPROVING OUTCOMES NEI+ introduced methods to improve transparency and systematic monitoring and supervision of activities in schools and centers in order to measure gains in the improvement of teaching. According to NEI+ reports, SUBEBs and LGEAs now have a system for coaching at the school and cluster levels. This ensures fidelity of implementation of the reading programs in schools. SSOs developed schedules for classroom observations and coaching carried out within the LGEAs’ Quality Assurance unit. Of the 13 SSOs who mentioned this topic, 11 of them appreciated the new systematic approach and felt it made their supervisory role easier to carry out. An SSO in Bauchi said, “I was given enough training that permits me to go back and support teachers and mentor them, coach them, and this really helped me.” An SSO in Bauchi spoke about how the provision of tablets for record keeping was useful, “They introduced a good system. We use tablets to monitor and coach teachers and we sync everything to NEI+ dashboard. So, they check and respond where needed.” An SSO in Bauchi spoke about how NEI+ training on monitoring has become more streamlined, “In my role as an SSO, I go to schools and observe lessons. I observe how they follow the fidelity of implementation of NEI + activities. All they need to do is follow the teacher’s guide and teach. It has made things much better.” The two SSOs who felt they did not have enough support both mentioned inadequate financial support, with an SSO in Sokoto saying, “They didn’t support us. Our allowance and the expenses we incurred were not taken care of.” Measures that NEI+ have put in place to ensure sustainability and stakeholder opinions on its likelihood are discussed in Evaluation Question Four. STRENGTHEN SYSTEMS AND DEVELOP CAPACITY IN LOCAL INSTITUTIONS NEI+ worked with the government at the state level to strengthen systems and build capacity. These included training and mentoring with representatives from SUBEB, UBEC, LGEAs, CSOs, women’s groups, CBMCs and SBMCs, headteachers, teachers, and facilitators. The teacher training programs were outlined in Evaluation Question One, and Evaluation Question Four contains information on personnel development strengthening, education management system strengthening, financial institution strengthening, ICT/EMIS strengthening, and quality assurance strengthening. WORK IN PARTNERSHIP AND LEVERAGE RESOURCES NEI+ has a good reputation with partners. They are in constant touch and are accessible. Stakeholders in both Bauchi and Sokoto states agreed that they were supported with mentoring and coaching, both in areas of teacher monitoring and implementing activities. An Access and Fragility Officer in Bauchi said that partners working with NEI+ had adopted a similar approach as the project, “Most of the partners, for example from SUBEB and BASAME, have started adopting these measures, especially in monitoring activities.” A Desk Officer in Bauchi spoke about how NEI+ asks for feedback from stakeholders to adjust their procedures. “Our contributions were taken by NEI+. They usually request for the feedback of what we have done. If we go to supervise [a school], they ask for reports, so I feel the NEI+ team is doing their best to monitor what we are doing.” Measures put in place during the COVID-19 crisis meant that meetings moved online, but there was still a high degree of contact, follow up, and communication with stakeholders to keep up the high standards of partnership, and communication seen throughout the duration of the project. The UBEC Desk Officer 27 in Abuja spoke of how accessible the NEI+ team was in handling problems. “We are almost like in the same house. I can call them any time of the day and tell them my problems.” NEI+ staff frequently hold meetings with stakeholders and follow up with emails, text messages, or phone calls. Stakeholders understand how to facilitate activities and feel they can contact NEI+ for support if they have any issues. The state governments in Bauchi and Sokoto are providing resources for textbooks and training of teachers. This is discussed in more detail in evaluation question four. Community members are making in￾kind contributions by helping to maintain schools and centers by cleaning the grounds, in some cases by providing fences to secure the perimeters, providing security guards during term time and in the holidays, provision of spaces for NFLCs (for example: verandas, buildings, mosques, and even their own homes). CBMCs, SBMCs, and PTAs also provide their time and knowledge by attending meetings and following up on children who are not attending. Local philanthropists assist in cash and kind donations, including breakfast for both facilitators and learners, reading materials, chalkboards, and school supplies for the students. DRIVE DECISION-MAKING AND INVESTMENTS USING EVIDENCE AND DATA NEI+ conducts monitoring, research, and assessments to inform decision making. According to project reports, the research areas throughout the life of the project have included baseline assessments and an EGRA to determine reading levels; a Community Education and Conflict Assessment to determine levels and drivers of violence in communities; standardized learner, teacher, and school assessments to evaluate effectiveness and impact; performance progress and institutional capacity monitoring to monitor fidelity of implementation and institutional capacity; community analysis; conflict/gender analysis; continuous monitoring through EMIS, report cards, institutional capacity assessment, and project-based tracking forms; and special research “mini-experiments” or inquiries to assess the effectiveness of specific interventions. The community mapping mentioned above is an example of decisions made based on data collected in a systematic, evidence-based process to inform the project of the communities most in need. According to NEI+ reports, the Initiative proactively deployed a Fidelity of Implementation tool in its bid to improve the monitoring and mentoring activities of its Access Coordinators. This tool produces data to record children who are not enrolled but are attending school, keep track of how many students there are in class in order to implement measures such as lengthening the hours of the NFLCs and AGLCs to run more lessons and reduce crowding, and measure achievements and attendance of girls to guide decisions on how to provide more support. Technical Working Groups (TWGs) also provide data for programmatic decision making. According to NEI+ reports, the members are technical staff from stakeholder organizations, who exhibit a high degree of professionalism, stability, and longevity in their positions. Examples of TWGs set up by NEI+ are the Monitoring, Evaluation, Assessment, and Research, and the Reading Materials TWGs. Stakeholders generally agree that NEI+ implemented activities based on research. An LGEA Reading Officer in Bauchi spoke about the research that was conducted before teacher training, saying, “The capacity needs of the teachers were researched and formally determined before the training was given. Representatives from SUBEB and the LGEAs were brought out for a kind of test to determine what was actually needed. The training was given a lot of thought.” A Desk Officer at the Federal Ministry of Education spoke about the importance of research in order to inform decisions. “Before you make a USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 28 policy, you have to do a kind of research. You must find the gaps and bring something new. Then you pilot activities and make decisions based on the results. NEI+ did this and then the decisions become policies that are binding.” The NEI+ Gender Specialist spoke about their methods. “For most of the steps NEI+ took, there was always research to be sure that we are in line with what is happening. Then we can design activities from a knowledgeable perspective of what the community is expecting, and what the standards are, as compared internationally, but most importantly, in line with the national standard.” PROMOTE EQUITY AND INCLUSION According to the Year 5 NEI+ report, the Initiative has made remarkable progress in ensuring gender equality and empowerment of women. It has worked with male and female partners at the federal, state, and LGEA levels to bring about changes in attitudes, behaviors, roles, and responsibilities in the workplace, learning centers, schools, and communities. The Initiative prioritized gender in key positions recruitment as seen in the recruitment of the DCOP, Operations Manager, HR, ICT Manager, Communications Manager, Community Mobilization Specialist, Grant Manager, Hausa Reading Specialist, Teacher Education Specialist, and English Reading and Standards Specialist. As seen in Evaluation Question One, NEI+ has a higher rate of females employed as teachers in formal schools than the average in government schools. AGLCs were established to provide adolescent girls with basic literacy skills needed to engage in further learning opportunities and for the economic and social development of their homes and communities. There is also a roughly equal number of boys and girls enrolled in the formal schools supported by NEI+. The Permanent Secretary of SUBEB in Sokoto said, “NEI+ strengthened government system by increasing the number of students enrolled through appropriate and relevant educational options, especially for the girls and orphans that are all vulnerable, and out-of-school children.” The project has also introduced guidelines for partners to promote gender balance. A CSO representative in Bauchi said, “The project has guidelines that the CBMC must have at least three female representatives at a minimum. We have communities where it was the first time in the history of that community that men and women have sat together to have a meeting about their children's education.” NEI+ works closely with women’s groups, who are instrumental in advocacy initiatives to encourage parents to send their children (especially girls) to attend school. In 2020, their role was expanded to include sensitizing and raising community awareness in COVID-19 prevention and helping parents to support their children to learn during the lockdown and ensuring all pupils and learners will resume attending schools when they are running again. TLMs were designed to be inclusive. According to the NEI+ DCOP, “Through our materials, we ensured that we avoid anything that could be demeaning to a particular religious group or a particular set of people, we made our materials to be inclusive, and we also are sensitive to gender equity, we ensure that both boys and girls are equally represented.” Girls in an AFLC proudly display their books. 29 Separate facilities are provided for boys and girls in the schools and centers. According to the NEI+ Access and Fragility Specialist in Abuja, “In some centers, we have toilets for girls, we have toilets for boys. No child is discriminated against based on gender.” CONCLUSIONS NEI+ has been successful in improving “Fidelity of Implementation” into the program across all six areas outlined in the USAID Education Policy Program Cycle Implementation and Operational Guidance (2020). The Initiative has a good relationship with stakeholders, who feel listened to and involved. Constant communication, even with the restrictions brought about by the COVID-19 crisis, have meant that NEI+ is able to keep abreast of any issues and keep a finger on the pulse of what is happening in the communities. Stakeholders are clear on the policies and procedures implemented by the project and appreciate the support the NEI+ team gives. Community members are consulted and involved in initiatives. Activities and TLMs are adapted to the local context and create an interest in learning. The project strives to create gender balance, both with their own staff members and those of partners. RECOMMENDATIONS • Continue to build capacity for stakeholders by introducing systems and tools to improve policy through data-driven decision making. • Improve gender balance of teachers in formal schools by offering training to all teachers and hopefully encourage females to apply as they will be more confident of their abilities and see a structured career path with professional development and mentoring. • Design and facilitate job- and context-specific soft skills courses for state government officials to improve their capacity, for example: time management skills and planning. EVALUATION QUESTION 3 How well did the Initiative influence factors affecting access to quality education for out-of￾school children in target communities? What were the challenges to mainstreaming and retention, and how were they mitigated? FINDINGS NEI+ initiatives to encourage OOSC to attend classes included establishing child-friendly NFLCs and AGLCs, raising the levels of literacy for students in these centers in Hausa to the equivalent of Grade 2 in formal education in order to increase mainstreaming and designing and rolling out numeracy, and life skills lessons. According to NEI+ annual reports, there was also a focus on providing evidence-based TLMs, criterion-referenced classroom teacher monitoring and coaching support, and teacher/facilitator professional development activities. The Initiative also supported community mobilization to improve reading outcomes, engaging with PTAs and SBMCs to develop skills in school-level decision making, monitoring school quality, monitoring or participation in school governance, advocacy work, and/or sponsorship or fundraising initiatives for supplemental educational materials. Women’s groups were supported in making house visits to talk to families about the importance of education and literacy. USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 30 As COVID-19 spread in Nigeria, NEI+ worked quickly to introduce and implement emergency measures both to keep education activities running for those children who were in school before the closures and to reach OOSC to try to mitigate long-term ramifications. These included relocating CPD for teachers and facilitators, and creating remote EGR lessons to virtual platforms. The NEI+ Year 5 Annual Report explains that the project developed and disseminated both public health care messages on how to stay safe and learning-from-home tips. These took the form of posters, banners, and town-criers. The messages were designed to encourage continued learning and improve the project relationship with communities and government partners as they witnessed the sustained presence of the project despite the pandemic. Engagement with partners included advocacy visits to encourage the government of Sokoto to initiate Mu Karanta! Let’s Read! lessons by radio and TV to bridge the possible learning gap. A total of 48 lessons each by radio and TV were produced and broadcast between March and September 2020 targeting NFLC and AGLC learners in Sokoto state. In both states, learners had the opportunity to listen to radio lessons produced for formal schools which reached 40,320 households. The figure was generated from the level of the coverage by the BRC (FM) which has a programmed system that can track the level of their listeners on any program aired. Parents were also sent IVR messages on enrollment, the importance of girl-child education, retention, and hygiene. To maintain teacher morale, the Initiative supported teachers and facilitators with digital pedagogical, curricular messages/instruction using Internet platforms. POSITIVE IMPACTS OF NEI+ ACTIVITIES Of 45 community members in both states who answered questions about the effect of NEI+ initiatives on getting OOSC back into education, all of them said that there was a very positive change (21 parents, 13 SMBC members, six CBMC members, and five women’s group members). Interview respondents focused on five main themes when talking about the benefits of the Initiative: improved materials and teaching methods, the visible effects of the project in the community, mainstreaming, community mobilization, and girls’ education. Improved Materials and Teaching Methods Many of the respondents mentioned that there has been a huge improvement in access to TLMs during the period of the Initiative, which has motivated learners to attend classes. A Desk Officer in Abuja said “Having those materials really motivated the children to go to school. They have something to read, they have something to take to school unlike before, when nothing was with them.” A community member from Bauchi spoke about a snowball effect created by the TLMs. “When the books are distributed and some children return home with their copies, it encourages other children to come to school. We also tell the children that if they do not come to school, we would collect the books from them and give them to those pupils who are regular in school.” Another community member from Bauchi corroborated this. “The supply and distribution of the books to the children is one of the major factors that has encouraged children to come to school, because the books are very attractive and the contents are interesting.” A Initially, I had the intention of withdrawing my children completely from school but the coming of this project changed my perception completely. The reason is that before, somebody could finish primary school without knowing how to read or write but now, in my household everybody can read and write courtesy of NEI+. Parent of school children, B hi 31 CSO representative in Sokoto said, “These books have drawn the attention of OOSC children. They want to be enrolled in school because they want to have those books and be able to read them like the kids that have already been enrolled.” The teacher training has improved methodology in the classrooms and made it more attractive to students according to two community members from Bauchi. “The NEI+ style of teaching attracts the children to school. So, when they hear the songs, they are attracted to go school and learn.” Another said “The way they teach them in school in form of singing draws children’s attention to join the school. The stories they tell the children in school draws their attention too.” Visibility of Effects in the Community Many of the stakeholders interviewed praised the obvious effects of the project on the communities. A headteacher from Sokoto said “OOSC and their parents now know the importance of education, so parents encourage their children to enroll into public schools to learn how to read.” An Access and Fragility Officer from Bauchi said that “You’ll see children gathering and discussing activities that have been carried out in our centers, and if you ask people in the community, you’ll hear positive thoughts about what the project have been doing.” Mainstreaming Several respondents mentioned that NEI+ has also been instrumental in affording students from NFLCs and AGLCs admission into formal education schools. According to the Executive Secretary for BASAME, “Now you can see that after children are enrolled in the centers, they are mainstreamed into the formal schools for completion while others transit to the post-literacy program to continue their education.” A CSO representative from Bauchi said, “If you have 50 learners, you find out more than 70 percent of the learners graduate successfully from the center, and have been mainstreamed to formal schools. And when you check those learners, you find out they're able to read or write to a very large extent. I think that's a very good achievement for NEI+.” Community Mobilization Community mobilization for OOSC was praised by interview respondents, particularly the efforts of the women’s groups supported by NEI+. A representative of a CSO in Sokoto spoke about how the women’s groups keep track of children’s attendance. “In order to monitor children not attending school, women’s groups go to visit parents. They mark houses to indicate that they have been visited, and when the program officer comes to monitor at the end of the month, they will call the women’s group to show the houses they visited and record this in his tablet. Then the women’s groups follow up with the parents to inform them about the importance of education so they release their children to come to school.” A Mobilization Officer in Abuja commended the advocacy and public awareness campaigns introduced by the Initiative saying, “Without the community support, I don't think we would be where we are today. Without the awareness, creativity, sensitization, and behavior change, we wouldn't have had the number of children that were enrolled.” Girls’ Education USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 32 One of NEI+’s aims was to focus on enrollment of girls. This was also appreciated by the interview respondents. A Mobilization Officer in Abuja spoke about the awareness building campaigns, and how they relate to the local context and showed parents the benefits for the whole family, sometimes leading to a deterrence of early marriage. An Access and Fragility Officer in Bauchi spoke about the importance of girls’ education for the whole community, “One of the primary goals of every parent in the community now is to see that his or her child gets a quality education, most especially girls. During advocacy, one of the messages is that if a girl child is educated, the whole community is.” The Initiative also enabled girls to attend school instead of selling goods on the street. A Head Teacher from Sokoto said, “The majority of the girls in my school can read and write and have also acquired skills while learning in school instead of hawking.” CHALLENGES AND ISSUES STILL TO BE FACED None of the respondents had any negative comments on the efforts of NEI+ in getting out-of-school children into the classroom. In fact, they emphasized that the continuing problems of getting parents to enroll their children could be mitigated by either extending the project with USAID support or replicating it with government or community initiatives. Respondents expressed that there is worry in the community and even anger that children may not have the opportunities provided by the activities in the future. The four categories of challenges and issues most mentioned by the respondents were: poverty, parental negligence, poor infrastructure, and an inadequate number of qualified teachers. Poverty The majority of the respondents from all categories spoke of economic issues and poverty as a major challenge. Parents cannot afford to provide necessary school supplies and lose the potential income of their children. Many children do not have parents to provide for them, and so they are in an even more difficult situation. A community member in Bauchi said “Some children do not go to school for various reasons such as poverty. Others are orphaned children who do not have caretakers to support them. The NEI+ activities have supported them to the best of their abilities, but there is more to be done. They supported by giving them books, uniforms, and scholastic materials. Prior to this, they didn’t have these things.” An Access and Fragility Officer from Sokoto said that sometimes bureaucratic fees prevent attendance. “We had a case of children sent out of school because of PTA fees of 50 Naira, but then we stopped that and tried to support with the ones we have on ground because driving children from school for the sake of PTA fees is bad for the school.” NEI+ and project stakeholders are committed to solving problems like these, but they still are an issue. A CSO representative from Bauchi spoke about the need for children to be active in their family’s economy. “Mainstreaming and retention is an economic question. We had one community where girls harvest cassava and then process and cook it to sell for profit. The village head told us, ‘Yes, we love Cards used as teaching aids. 33 education. Yes, we want our girls to know how to read and write but this is their life, this is how they make their living.’ So, if you are suggesting anything that would prevent them from doing this, even the village head will not support you.” They went on to praise the life skills initiatives of the project. “I think what needs to be done is as we did during the first and second year, when we had a livelihood intervention program. If these girls are learning skills that they can convert to something economical, the parents will be happy.” Parental Negligence Many of the respondents were concerned about the inadequate importance parents place on education. They commended community members who were making an effort to advocate for children going to school through NEI+ interventions, but said there was still a lot of work to be done by both the local community and the government. A representative of an SMBC in Sokoto said, “We, the parents are responsible to make a move on this. The Government should also ask parents to allow their children to go to school. Sometimes the children engaged in hawking show their interest and stand by classes to watch the children learning through interesting methodologies used by the teachers, but their parents don’t care about this. Some of the children want to go to school but their parents do not support or allow them.” A community member from Sokoto spoke about the importance of advocacy. “The parents of those children who are out of school need to be sensitized on the importance of education. The parents must be willing to release their children to get to school and get educated.” A School Support Officer from Bauchi said, “Maybe if parents support the children to go to school and not stay at home, come to school at the right time, and go through their books after school, it will help. As we have observed, some parents are not committed to the learning needs of their children. There is need to give them more awareness for improvement.” An SMBC representative from Sokoto spoke of the need for NGOs to be involved in advocacy for education. “There should be more enlightenment and education for parents. Non-governmental organizations should be more involved, people listen more to strangers than the government.” Poor Infrastructure Some respondents spoke of the need for more NFLCs and AGLCs so that the out-of-school children have somewhere to study if they decide, or are able, to start or resume their education. They appreciate the centers set up by NEI+ and want to see the initiative expanded. A State Technical Working Group (STWG) representative from Sokoto would like more learning centers throughout the state. “Establish more learning centers and have a wider coverage of the children that are out of school. They should extend it to cover the whole state so that those children that have not benefited before can easily get into centers and then they will be able to access the intervention.” A Head Teacher from Sokoto spoke of the poor learning conditions currently available. “A very great issue is the issue of facilities in schools. Do we even have these classrooms? Do we have the teachers? Are they there? Because if you go to those schools especially in the interior, you will find that the schools are deserted. If you go to a school, then you will find only two teachers manning a whole school. If you go to these schools, the structures are dilapidated, no roof, some of them are outside under a tree, are being taught under a tree.” USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 34 An Access and Fragility Officer in Sokoto said that the community can do a lot to improve education options, but they need assistance. “The community needs to be given support, to establish these non￾formal learning centers and manage them themselves.” CONCLUSIONS All of the interview respondents expressed satisfaction with the NEI+ initiatives for getting out-of-school children into education. They spoke about improved teaching methods and interesting lessons creating a desire in children for studying. A big draw is the TLMs, which in many cases are the first materials the communities have had, and in others a vast improvement on what was there before. The project has generated enthusiasm in the community, with parents and women’s groups mobilizing to advocate for out-of-school children and steadily building on their successes with NEI+ support. The negatives stated by the respondents were all to do with wanting the activities to continue, which may reflect their doubts about sustainability after project close-out. Community members, educators, CSOs, and policymakers spoke about the need for continued support in order to carry on educating parents and children on the importance of going to, and staying in, school. They talked about the need for methods to alleviate the burden on parents to provide uniforms and school supplies, ways to integrate economically beneficial skills into learning, the necessity to improve infrastructure and open new NFLCs and AGLCs, and a continuation of advocacy initiatives to convince parents that the education of children, notably girls, is crucial for the development of the whole community. RECOMMENDATIONS ● Consider an extension of NEI+ or a project with similar activities related to the attendance, retention, and mainstreaming of out-of-school children. ● Continue and expand advocacy activities to inform the community on the importance of education. ● Continue teacher training programs to keep lessons interesting, which will encourage students to attend. ● Work with the government to hire more qualified teachers. ● State governments should open more NFLCs and AGLCs so that there are more places available for OOSC to study. ● State governments should work with communities, philanthropies, and local businesses on fundraising to supply children with the uniforms and supplies they need to attend learning centers. EVALUATION QUESTION 4 In what ways did the Initiative achieve institutionalization of efficiencies for more effective educational management by state and local education authorities? Where gaps exist, what factors limited progress? To what extent are these activities and activities across all Initiative Result Areas sustainable without USAID investment? Why? FINDINGS The Initiative focuses on strengthening the technical and administrative capacity, commitment, and accountability of federal, state, and LGEAs. NEI+ built initiatives into the project to try to ensure sustainability, and although stakeholders are optimistic about the chance of activities continuing, they were 35 unable to provide concrete financial evidence of the funding for this. Four main themes emerged from the data analysis in these areas: personnel, education management system, financial system, ICT/EMIS, and quality assurance strengthening. PERSONNEL DEVELOPMENT STRENGTHENING On personnel support, respondents in Bauchi and Sokoto, and across stakeholders, expressed the view that NEI+ provided adequate training to support the role they played in the implementation of activities. For example, all the teachers/facilitators in formal schools and AGLCs/NFLCs that the Evaluation Team interviewed said that they had attended at least one form of training and that the trainings positively affected their ability to deliver EGR instruction. The Head Teachers also expressed an increased ability to provide support to their teachers due to the training from NEI+. Similarly, SSOs, who are responsible for school supervision of quality control, also confirmed they received adequate training which significantly changed, in a positive way, how they carry out their roles. The training received, they explained, strengthened them in using modern techniques such as deploying ICT in supervising and providing feedback to the appropriate education authorities. They expressed that the trainings increased their ability to deliver coaching and mentorship when they go on school supervision as opposed to the earlier approach of fault finding. In the words of one SSO from Bauchi State, “I was given a training, enough training that will permit me to go back and support teachers and mentor them, coach them…” The mentoring and coaching approach to supervision introduced by NEI+ changed the relationship between SSOs and teachers in the schools for the better because they now see themselves as partners. Other stakeholders like SBMCs/CBMCs, CSOs, and women’s groups, with exception of a few SBMC members who joined the committee recently, also confirmed that they received training to carry out their responsibilities in their various schools. EDUCATION MANAGEMENT SYSTEM STRENGTHENING According to NEI+ annual reports, activities influenced education policy by developing a policy framework for the implementation of EGR instruction and got it approved by SUBEB, Ministries of Education at the state level, and NERDC at the Federal level. This paved the way for adoption of the 5Ts plus C framework for EGR in the two states, which is a system to improve teaching techniques based on 5 key areas, ( Time, Text, Tongue, Teaching, Testing and Curriculum). The Initiative also revived and improved the Teacher Recruitment and Deployment policy in both states, to ensure better teacher management. Also, as part of policy strengthening, the Initiative influenced a modification of the school timetable in the implementing states to allocate more time for EGR instruction. Another area of improvement is in coordination of educational activities. Prior to NEI+, according to respondents, the implementing states operated different steering committees at the state level, which led to a duplication of functions and energy. In the Ministry of Education, respondents explained, the commissioner chaired several different steering committees. The same applied to other parastatals under the ministry. To streamline coordination, NEI+ influenced the collapse of these committees into one, called the Education Steering Committee, under the leadership of the Commissioner of Education. The committee drew membership from the service ministries and parastatals including the Ministry of “NEI+ has introduced systems to check our level of understanding. Training and retraining of the supervisors and facilitators have strengthened the system. These factors are game changers in knowledge transfer.” SAME Scheme Organizer, Sokoto USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 36 Education, the Ministry of Finance, Budget, and Economic Planning, the Ministry of Women’s Affairs, and SUBEB, BASAME, and LGEAs at the local level. The steering committee addressed development of policy, policy constraints, and policy gaps, and these made it easy to take and implement decisions leading to improved education management. In addition, the Initiative influenced strategic planning for the education sector A in the implementing states. With support from NEI+ in capacity building, the ministries developed Strategic Education Sector Plans as a guide for educational development. From the general plan, the education stakeholders developed a Medium-Term Sector Strategy (MTSS) and an Annual Work Plan to map out actions to develop the education sector. NEI+ supported education management in the two states in generating credible data for decision making. To obtain this school data, the Initiative established an annual school census and trained SUBEB and Ministry of Education personnel to carry it out. The Initiative utilized the census to plan and implement activities, especially the production and distribution of TLMs and facilitation of teacher training. In the 2019/20 academic session, Bauchi state conducted an annual school census that was 100 percent sponsored by the government, with NEI+ providing technical support and playing a supervisory role. With accurate data from the school census, education managers can plan and take data-driven decisions. FINANCIAL INSTITUTION STRENGTHENING One of the areas where the financial system has been strengthened is in budgeting. The Initiative helped states develop a Budgeting Planning Tool for the education sector. This tool helped the Ministry of Education and its parastatals to develop a strong budget, because it adopts a practical approach, and shows resource constraints and levels of utilization. According to the NEI+ Systems Strengthening and EMIS Officer in Sokoto, “We developed a budget and planning tool. This eased the development of the medium￾term education sector strategic plan in Sokoto state. Before, it took the state a month to develop, but with the tool we provided it makes it easier. The last plan sector was developed in three days.” The SUBEB ICT and EMIS Director in Sokoto said, “NEI+ developed a Budget Planning Tool. It is a nice application that helps us to develop our medium term basic strategic plan, and it is very useful. Because of the efficacy of the budgeting tool, according to respondents in Bauchi state, the State Planning Commission has adopted the tool and has disseminated it to other ministries for use. In addition, NEI+ built the capacity of financial officers at the state level. Working together with the State Education Account technical group, according to one source, NEI+ built the capacity of the states’ Financial Officers to do analysis to track government expenditure and produce concrete documentation for people to see instead of just relying on hearsay. The NEI+ State Team Lead from Sokoto said this about financial system strengthening: “State members have been strengthened on how to improve the state’s education account, and this has been helping the state. The financial system was surveyed, and a cleansing system was adequately and effectively put in place to strengthen the accuracy of the system.” In terms of financial management, respondents expressed the opinion that, through interacting with NEI+ finance officers, the state finance officers have learned a lot of good practices, particularly transparency in payment, by employing the financial inclusiveness method of making e-payments to beneficiaries as practiced by NEI+ instead of payment going through government offices, which can result in under-paying. The LGEA Executive Secretary in Sokoto said, “Our accountants and auditors were trained on financial 37 management, and it helps to make operations clear. NEI+ is developing our financial institutions in rules and regulations of finance.” The SUBEB Permanent Secretary in Sokoto said, “NEI+ trained our finance team. It was successful. There are proper checks and balances and things are properly done.” However, as the evaluation team was not able to obtain financial data from government officials, who said they did not have it at hand during the interviews, and did not respond to follow ups, it was not possible to verify this statement. ICT/EMIS STRENGTHENING According to NEI+ annual reports. The Initiative established an EMIS at three levels (Ministry of Education, SUBEB, and LGEA) to control the annual census. The NEI+ DCOP explained that the staff that manage the EMIS were adequately trained by experts in Nigeria and the United States and are now able to use the system effectively, “We have been able to train EMIS officers at the state level, to be able to handle and collated data using ICT and to collect data, interpret, analyze data, and all that. We've been able to train these EMIS officers on the use of dashboard and how to set up a dashboard.” The LGEA ICT & EMIS Officer in Sokoto said, “I learned a lot from the training on data management using so many packages like Excel, PowerPoint. Before I didn’t know how to use it, but now I am familiar with it and use it in my job.” The Initiative also developed an FMIS to manage data coming from the non-formal learning sector. The FMIS data bank is located in the adult and non-formal education unit of the local governments. FMIS, the NEI+ Systems Strengthening Officer stated, bridged the gap that had existed in the annual school census, which often under-reported learners from the non-formal sector. As part of system strengthening, NEI+ facilitated the development of a Teacher Management Information System (TMIS). The Evaluation Team gathered that the software related to this has been developed and will be shared with the Ministry of Education, SUBEB, and BASAME. The SUBEB level will directly control all the teachers that are in primary education across the LGEAs, while the Ministry of Education will utilize theirs to manage teachers at the secondary level. Implementation and institutionalization of the TMIS was scheduled for January 2021, but has been shelved due to the COVID-19 pandemic. EMIS respondents across the two states of Bauchi and Sokoto corroborated the existence of the software, and the capacity building training they received to enable them to operate the various systems. QUALITY ASSURANCE STRENGTHENING The Initiative inherited a Quality Assurance Department within SUBEB, but according to the NEI+ State Team Lead in Bauchi, “Functionality was not well defined because they didn’t have tools. They didn't have a process. They didn't even have a database.” NEI+ tried to correct this, by providing a Quality Assurance Handbook to SUBEB, quality assurance evaluators, and school system evaluators to strengthen the department. The Initiative also supported SUBEB staff with training on coaching and mentoring and then provided a coaching guide and tablets to aid supervision. Similarly, it gave officials at SUBEB and BASAME tablets (221 in Bauchi, and 150 in Sokoto), to track SSO mentoring and coaching activities at formal schools and NFLCs as well as providing support where necessary. Respondents informed the Evaluation Team that, in Bauchi, SUBEB purchased an additional 400 tablets in order to support coaching and mentoring while BASAME also received an additional consignment from UNICEF, which they have deployed for mentoring and coaching at NFLCs. The story is the same in Sokoto USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 38 state, where the government purchased and distributed 62 tablets, as outlined in the Ministry of Education action plan. The SSOs upload the information collected during school visits to an Information Dashboard. This enables education decision makers in the states to deal with issues related to the quality of education at the primary level in general and EGR instruction specifically. The system put in place allows authorities in the states to track TLMs, and teacher and student attendance in the various LGEAs. Information is compiled in the Information Dashboard for use in planning. A Quality Assurance Officer from Sokoto Ministry of Education confirmed this. “It was NEI+ that actually initiated e-monitoring, that is monitoring using tablets. So that system has actually considerably improved our monitoring strategy.” SUSTAINABILITY OF ACTIVITIES Respondents across the states are optimistic that NEI+ activities will be sustained when the project lifespan expires. Varying reasons influenced the respondents’ optimism. NEI+ representatives pointed out that a sustainability strategy was embedded into activities right from the inception. The respondents from the relevant organizations explained that NEI+ advocacy effort for states to have a budget line will ensure a continuous flow of funds to sustain activities in the future. According to NEI+ reports, as part of the Bauchi government’s commitment to providing 70% of the TLM needs for the year, SUBEB procured 1,307, 286 textbooks (Mu Karanta! and Let’s Read!) for the 2019- 2020 academic year. SUBEB also procured an additional 800,000 copies of TLMs for non- intervention LGAs. In Sokoto, SUBEB is currently accepting delivery of the early grade reading teaching and learning materials they procured for 10 focus Local Government Areas formerly supported by USAID's Northern Education Initiative (NEI) Plus activity. Valued at more than 413 million naira ($1 million), the books will reach more than 270,000 children in grades 1 to 3 and 3,676 teachers will receive matching guides. In February 2021, SUBEB trained 1,500 English teachers in how to use the TLMs. The State Ministry of Basic and Secondary Education (SMoBSE) in Sokoto is asking for assistance from NEI+ before project close-out, to provide refresher courses for teachers, in how to effectively use the materials they are providing. The letters confirming the provision of funds from and asking for assistance, and photos of the textbooks with the Sokoto state government logo are in Annex A. Implementers of the Initiative cited two key institutions that have made a lasting impression in the states’ programming: the MTSS and annual school census. According to a respondent, MTSS is now institutionalized and sustained because the Initiative staff do not participate as organizers, but as reviewers of the developed plan. The states now lead the MTSS on their own timetable and provide the necessary funds and participants to implement it. On the annual school census for this year, which is being supported by UNICEF and NEI+, Bauchi state did not wait to be supported; they drew up their own timetable. The state made a budget provision for the activity, trained data collectors, and engaged a data analyst and report writers. These are indicators that states can sustain activities. In the area of teacher development, NEI+ worked to entrench EGR training courses in colleges of education curricula as well as empowering the colleges in the implementing states to deliver on their plans. NEI+ is confident there will be a sustainable staffing supply to drive EGR instruction because of the 39 cascading method of training. There will still be LGEA trainers, with the knowledge to conduct training courses for teachers and facilitators, in place after the project closes out. The state governments’ respondents also expressed optimism about the sustainability of NEI+ activities. They explained that they have already mainstreamed activities into a BESDA project, which is sponsored by the World Bank, to provide quality education in some selected states in Nigeria. Bauchi and Sokoto states specifically selected the NEI+ model in implementing the BESDA project. In Bauchi for example, the state government used BESDA funds to expand NEI+ activities to the remaining ten local governments in the state that were not part of the project. BESDA Desk Officer respondents at the federal and state levels confirmed this and said seven states in Nigeria decided to implement the NEI+ model with their funds. Other stakeholders overwhelmingly share optimism of sustainability of NEI+ activities. Teachers and facilitators expressed confidence in continuing to deliver EGR instruction using the new strategies acquired in NEI+ trainings. Similarly, the SSOs are optimistic about continuing to utilize ICT in school supervision and using the techniques of coaching and mentoring in assisting the teachers and facilitators they work with. One respondent in Bauchi did express a reservation about the government’s ability to provide funds for activities after the exit of NEI+, but overall, people believe NEI+ activities are sustainable. CONCLUSIONS The Initiative successfully delivered on its mandate of systems strengthening by introducing innovative ideas in education management, financial, and quality assurance systems in the two states of operation. Based on the level of training provided by the Initiative, the states will be able to sustain education management tools like the Budget Planning Tool, TMIS, FMIS, and the Information Dashboard in decision making to improve the quality of education in the states. RECOMMENDATIONS ● State government education authorities should continue to implement activities and mechanisms and to provide follow-up on personnel training, for those who will continue to manage and sustain the gains recorded in system strengthening so that there is no sudden relapse. ● Continue to maintain healthy relationships with the implementing states to provide further support in education sector budgeting to ensure a steady flow of resources to fund activities that will promote EGR instructions. EVALUATION QUESTION 5 To what extent has the Initiative integrated a conflict-sensitive approach that addressed or reduced the safety concerns of communities and beneficiaries (e.g., school-related violence, gender-based violence, etc.), including but not limited to learners, parents, school-based management committees, etc.? USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 40 FINDINGS According to NEI+ annual reports and interviews with NEI+ staff, initiatives implemented by NEI+ to integrate a conflict-sensitive approach include stories developed in Mu Karanta! textbooks that support peace, tolerance, and respect; training for educators and community members in peace education and conflict sensitivity; integrated themes relating to gender in a radio drama produced for community outreach activities; prioritizing gender in key positions; and ensuring learning centers are child-friendly and comply with safety standards. Although CMBC, SMBC, and women’s group members, and CSO representatives in both states said that NEI+ had introduced training and advocacy against gender-based and school- based violence, the initiatives are not well known in the community. Of 30 parents asked about this, (11 in Bauchi and 19 in Sokoto) only four had heard of any activities in this area. Thirteen parents (six in Bauchi and seven in Sokoto) said that there was no violence in their schools or communities and so it was not an issue. Where people did attribute a decrease in conflict to the project, they stated that education in general had a positive effect on the safety of the community, because busy children and adolescents are focused on learning rather than “making problems.” There were a few respondents who spoke about conflict that occurred in the community where students were the victims of adults, but most respondents emphasized measures that had been, or could be, taken to prevent the students themselves from causing problems. The main points raised focused on physical safety measures provided in schools such as fences, provision of sanitation, and the hiring of security guards. Head Teachers spoke about measures introduced to inform the community on the risks and prevention of Covid-19, including extra hand washing facilities in the schools, banners and posters with guidelines on wearing masks and hygiene, and text messages to parents with updates and public health messages. POSITIVE IMPACTS OF NEI+ ACTIVITIES The themes most mentioned by respondents on the issues of conflict-sensitivity were school enrollment and education, training in safety and avoiding violence, materials addressing peace and conflict, community awareness, safety for women and girls, and general school security. School Enrollment and Education The majority of stakeholders interviewed mentioned that educating children has a positive effect on decreasing conflict and violence, or the potential for it, in general. A community member from Bauchi said, “The fact that children are effectively engaged in school activities will indirectly keep conflict at bay.” An Access and Fragility Officer from Bauchi said, “Young ones that are known to be terrorizing the community are now calm and can read and write.” A parent from Sokoto spoke about the benefits of getting children off the streets and into classrooms, “Instead of having a high number of children engaged in hooliganism and hawking, they are getting educated and cannot be distracted.” Another community member in Sokoto spoke about the protective umbrella education provides. “The schools and centers that were opened by NEI+ have provided an alternative for children. Instead of roaming about and picking up bad habits or exposing themselves to bad vices and opportunities for bad things and risks, the schools have shielded our children and also educated them on the dangers of violence and conflict.” 41 Training in Safety and Avoiding Violence Although parents interviewed were largely unaware of activities to mitigate violence, respondents from CMBCs, SMBCs, and CSOs did speak about trainings facilitated by NEI+ and how the content improved safety and security in communities. An Access and Fragility Officer in Bauchi praised the NEI+ trainings, saying, “We had security and conflict sensitivity and it was cascaded to the CBMCs and it has reduced community conflicts.” A CSO representative in Bauchi spoke about the effect of NEI+ training on adolescents. “We did most of the orientation in the Youth Learning Centers with adolescent boys, because they tend to be easily drawn into acts of violence. The project gave enough training to mitigate against violence in our centers and in the community.” A CMBC member in Sokoto spoke about the importance of raising the self-esteem of boys. “We learned how to educate boys to know their rights, to do the right things and see themselves as important assets in the community.” A SMBC member in Sokoto spoke about the facilitators’ roles in raising students’ awareness after they attended training. “The facilitators have been trained by NEI+ and they help educate the students on the side effects of engaging in violence, conflicts, and other negative vices.” It seems that, although NEI+ did facilitate trainings, the information was not effectively passed down to the communities. Materials Addressing Peace and Conflict Several respondents spoke about the usefulness of the materials and activities provided by NEI+ in addressing issues of conflict. A parent in Sokoto said, “NEI+ provided materials for special lessons to educate children on the dangers and risks associated with spending time on the streets. I have seen dramas being organized in schools highlighting the difference between children that engage in hawking and aimless wandering to children that go to school to get an education.” An SMBC member in Sokoto said, “NEI+ has brought about change in the lives of children since knowledge on how to be better people is part of what they are taught in school. They have been given books on how to build life skills and are also taught security education. These pupils now know the risks involved in engaging in negative social vices.” A CMBC member in Sokoto said, “Most of the books distributed by NEI+ contain vital information on how to live a peaceful and harmonious life in the community.” Community Awareness and Involvement Awareness raising for communities and their involvement in activities was mentioned by some respondents as a crucial element in maintaining safety. Several respondents spoke about the importance of including local leaders in activities. An STWG representative in Sokoto said, “The engagement of traditional and religious leaders and top government officials is one of the measures taken which is commendable and which has yielded fruitful results in curtailing violence and conflicts.” A CBMC member from Sokoto spoke about how NEI+ has changed attitudes toward safety. “NEI+ has enlightened us on the negative effects of conflict, violence, rape, and the like, on the victim and the perpetrator. In the past the information we had was not as much and as such we did not care. But now we are alert and wiser. They even send us messages from time to time to remind us.” This refers to advocacy messages that NEI+ sends to community members via SMS. Safety for Women and Girls USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 42 There were not many respondents, particularly among the parents, who mentioned gender-based violence as an issue, but those who did said that NEI+, with community support, had addressed these issues to a certain extent. An LGEA representative in Sokoto said that, “We keep training the community on how to curtail this issue of gender-based violence. All the invited community leaders were mobilized and trained on how to properly take care, how to properly watch, how to make sure that we keep them safe.” Another LGEA representative spoke about how issues of gender violence are sensitive, “It's something that you cannot touch but, you know, if you ask a community member, maybe one of the trusted members, you will hear things like that from them and then we can act.” A Women’s Group member from Sokoto spoke about their role saying, “We sensitize parents and community members on the risks associated with gender-based violence.” The Director of Planning in Bauchi spoke about how they ensure the safety of girls to encourage them to continue education, “We put an effort in ensuring that girls are taken very good care of. Maybe they have fear of being maltreated and so the government is trying to bring a way that they will be motivated and have interest in coming to school.” A CSO representative in Bauchi spoke of problems at the start of the project that have been improved by insisting on female facilitators for girls. “The NEI+ project has what they called ‘do no harm,’ and that, honestly, was really, really effective. In first year, we had a situation where girls were molested by the facilitators. and then the second year, we now said no, this time around we are not going to have facilitators of the opposite gender in any of the classes at the AGLC level. It’s a policy that all the facilitators must be females.” General School Security Many participants spoke about school security in general, with issues relating to guards on the school premises and the importance of safe facilities. A CSO representative in Bauchi spoke about an action when NEI+ took on an outbreak of violence. “There was one of our communities where there was a little challenge when our learners were preparing for exams. Immediately the directive was that the exam should be suspended. And this is very important; the safety of the children was considered above all.” Some respondents said that their schools and centers were secure as a result of NEI+ initiatives to provide security. An SMBC chairman in Bauchi said, “We ensure safety of the pupils by assigning people to patrol the school premises.” A Head Teacher in Bauchi said, “My school is safe and secure. We have toilets, fences, and water sources. We also have security men within the premises.” NEI+ also implemented training on safety issues. A Head Teacher in Bauchi said, “NEI+ trained us to educate the community and the pupils on many safety measures. For example, our school is situated on the main road, and children have to cross the street to come into the school. So, they encouraged the parents and SBMC to ensure that they support children in crossing the street.” Education continues in Sokoto State, despite the COVID-19 crisis. 43 CHALLENGES AND ISSUES NOT ADDRESSED In interviews where respondents talked about security challenges and issues with violence, there were four main themes: improving the school/center environment, advocacy and sensitization, community involvement, and youth productivity. Improving the School/Center Environment Some of the respondents stated that there were still not enough security measures in place in schools and learning centers. A community member in Bauchi brought up the need for more security, “In the majority of our schools, we do not have enough security officers. What can be done is that security guards should be employed to ensure the safety and security of lives.” The TRCN Desk Officer in Abuja said, “There are schools that don’t have fences; there are schools that are very vulnerable.” Advocacy and Sensitization Respondents spoke of the need for training and meetings to make all levels of the community aware of the issues and possible solutions around safety, conflict, and gender-based violence. The NCCE Desk Officer in Abuja said, “Social and emotional learning should be the basis for orientation in every community, every classroom. Then these children will carry these things out to the community.” An Access and Fragility Officer in Bauchi said, “There should be more training of community people especially the community leaders, and even the children. They should be trained on how to avert conflict or how to implement conflict-sensitive activities.” Some respondents spoke of the need for ongoing advocacy. A CSO representative in Bauchi said, “I think the major issue is just continuous orientation in how to draw our youth and our parents away from tendencies or people that will lure them into extremism, especially with the advent of the Boko Haram and so many things around.” A CSO representative in Bauchi spoke of the need for introducing gender champions. “If we can make every village head and every community influencer a gender champion, some of these issues can be discussed. They can become a kind of custodian of ensuring that nothing happens within their domain.” Community Involvement All the respondents who gave advice on future initiatives spoke of the importance of involving the community at every stage. A CSO representative in Sokoto said that community ownership was the key to successful initiatives. “I believe each stakeholder, the security agencies, the traditional rulers, and also the community members will choose to safeguard their community and to protect their children.” A parent in Sokoto said that elders should be supported. “We give advice based on our experience in stopping community-based violence. We can advise them because we know our people. We can also help in investigating and putting sanctions on any erring person.” Youth Productivity Respondents in all categories emphasized that education and keeping children and youth in school and off the streets is a major deterrent to violence. A parent in Sokoto said, “It would be good if children are USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 44 empowered and taught entrepreneurship skills. With these skills, they will not have to finish school and wait for government jobs that are not available and, in the process, get distracted.” Collaboration with the Government Several respondents spoke about the importance of getting government support for activities. A community member in Sokoto advised that NEI+ should collaborate with the government, “To expand their community engagement and enlightenment activities on the effects on violence, conflicts, and other negative social vices.” An LGEA Executive Secretary in Sokoto said NEI+ should work with the federal government to sensitize the public about violence. “Then I am very sure that it will be reduced. If the public or the stakeholders know what to do and what not to do, they can differentiate between good and bad.” CONCLUSIONS Although NEI+ implemented initiatives to integrate a conflict-sensitive approach, particularly early on in the project, the majority of parents interviewed were unaware of specific activities conducted by the project to meet this goal. In general, over the five years of the project, issues of safety and security in schools were the priority, rather than specific activities to raise awareness on conflict, violence, and gender-based violence. RECOMMENDATION ● Develop methods to make sure the NEI+ trainings on gender-based and school-based violence are better known in the community. ● Ensure follow on to establish how many community members have been influenced by the training and subsequent awareness building campaigns. 45 ANNEXES ANNEX A: LETTERS CONFIRMING RELEASE OF FUNDING BY STATE GOVERNMENTS USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 46 47 USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 48 49 USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 50 ANNEX B: EVALUATION SCOPE OF WORK Scope of Work End line Performance Evaluation Northern Education Initiative Plus (NEI+) 1. PROJECT IDENTIFICATION DATA Development Objective Project Title 2.2. A healthier, more educated population in targeted states Northern Education Initiative Plus (The Initiative) Award Number Award Dates AID-620-C-15-00002 October 26, 2015 - October 25, 2020 Funding Implementing Partner $82,201,163 Creative Associates International. Inc. COR Nura Ibrahim 2. BACKGROUND INFORMATION Nigeria is critically important to the global stage as the largest economy, largest democracy and most populous country on the African continent. Nonetheless, Nigeria faces daunting developmental challenges that threaten its potential as a world leader. Nearly 65 percent of Nigerians live in extreme poverty. Education sector statistics are equally underwhelming with an estimated 10.5 million Nigerian out-of-school children (OOSC) between the ages of 5 and 14, the highest number in the world. Those children attending school are likewise inadequately learning. According to World Bank data, 81 percent of young adults with only a primary education are unable to read5. Poor quality teaching and learning environments directly correlate with these low achievement levels. Crumbling physical infrastructure, a scarcity of teaching and learning materials (TLMs) and inadequate teacher training or professional development opportunities are drivers of educational failures. Regional disparities exist too, with northern Nigeria states performing well below the national average across education indicators such as attendance and completion rates. The 2015 National Education Data Survey (NEDS), facilitated by the National Population Council, reveals that the average Net Attendance Ratio (NAR) of the combined North 5 2018. Learning-Adjusted Years of Schooling (LAYS). World Bank 51 West and North East Regions was 46.6 percent, meaning that approximately 53.4 percent of the total number of children in the north of primary school age were not attending school. For example, 56.1percent of children in Bauchi and 68.7 percent in Sokoto had never been to school, compared to the national average of 24.1 percent. Similarly, primary completion rates in Bauchi and Sokoto are 30.8 percent and 23.7 percent respectively, with the national average at 68.9 percent 6. Outcomes vary by gender also, similar to geography, with NAR at 31.8 percent and 36.5 percent for girls in the north-east and north-west, respectively. In other words, more than half of girls in the northern region are not in school. In 2013, the United States Agency for International Development (USAID) financed an Early Grade Reading Assessment (EGRA) in Hausa and English in grades two and three in Bauchi and Sokoto states. Ninety-two percent of the students could not read a single word in Hausa by the end of second grade. Regretfully, Nigeria holds the sad distinction of achieving one of the highest rates of zero scores the Agency has ever recorded through the EGRA instrument among more than 30 countries with USAID-financed testing. 3. PROJECT DESCRIPTION The Northern Education Initiative Plus (referred henceforth as the Initiative) is a five-year activity funded by USAID/Nigeria to strengthen the ability of Bauchi and Sokoto states to provide access to quality education and significantly improve the reading skills for more than one million school-aged children and youth, especially girls, orphans, and children enrolled in nontraditional schools. The Initiative employs a holistic approach, addressing a broad range of critical factors that affect learning, teaching, systems management, parental participation, and community engagement across two core intermediate results: ● Intermediate Result 1: Government systems strengthened to increase the number of students enrolled in appropriate, relevant, approved educational options, especially girls and out-of-school children in target locations, and ● Intermediate Result 2: Government systems strengthened to improve reading outcomes for primary grade learners in target locations. The Initiative focuses on strengthening technical and administrative capacity, commitment and accountability of federal, state and Local Government Education Authorities (LGEAs) to provide effective English and mother￾tongue-based Early Grade Reading (EGR) teaching and learning to pupils; and to deliver literacy, numeracy, and life-skills to OOSC and youths. Whenever possible, interventions strive to mainstream these youth into formal schools for onward learning and social mobility. Regarding capacity strengthening, LGEAs are supported to plan, budget, supervise teachers, deliver high-quality TLMs and mobilize community involvement for increased enrollment and improved reading outcomes. Colleges of education are also capacitated to integrate best practice EGR concepts, techniques and materials into their curriculum, while playing a key role within the education sector to promote EGR policy and research. Specific to strengthening efficiencies related to teacher payments, the Initiative also integrates the technology of mobile money to regulate payments and reduce absenteeism related to extended long-distance travel of individuals to collect salaries. Overall, the Initiative is expected to demonstrate significant improvement in EGR for roughly one million children in grades one through three in formal schools and 400,000 OOSC attending 11,000 Non-Formal Learning Centers (NFLCs), Adolescent Girls’ Learning Centers and Youth Learning Centers across two states. The Initiative will likewise train nearly 12,000 teachers and learning facilitators who support student learning. By improving learning outcomes across Bauchi and Sokoto states, more Nigerian children will be better equipped 6 https://www.unicef.org/Nigeria/education USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 52 to lift themselves from abject poverty and participate in a more prosperous and democratic society. In so doing, the Initiative’s investments will catalyze measurable educational gains that will contribute to enhancing the stability and productivity of the US-Nigeria bilateral partnership. The Initiative received two technical contract modifications over the life of the project. In March 2018, while maintaining the two core strategic objectives of the activity, Modification 4 reduced the number of NEI Plus implementation states from three to two and enhanced the focus on government systems strengthening, policy reform, and LGEA accountability and transparency. These modifications were intended to ensure increased sustainability of USAID investments in the two target states. In May 2019, Modification 7 expanded the scope of NEI Plus to include development of English language materials for P1. While the Scope of Work includes multiple references and results to be delivered in Islamiyah, Qur’anic and Tsangaya Education (IQTE) schools, early on in activity implementation these components of the activity were dropped. As a result, the end line evaluation will not look at aspects of the program that were intended to be implemented or delivered through IQTE. 4. ACTIVITY DESIGN FRAMEWORK Development Hypothesis The Initiative’s development hypothesis asserts that through the strengthening of local education management systems, in conjunction with extensive community-based outreach activities, education access and quality for all children, including vulnerable children, will be improved and will result in significant improvements in reading outcomes for primary grade learners. Firstly, improved educational management is expected to lead to an increase in the number of students, especially girls and vulnerable children (e.g., physically disabled, Almajiri, etc.), enrolled in appropriate, relevant and approved educational options. Supported opportunities for out-of-school populations would also allow students, especially girls and vulnerable populations, to gain access to informal and formal education. Secondly, because quality is defined as improved reading skills in primary grades, the Initiative is also intended to contribute to improved reading outcomes for primary grade learners, a key skill needed by pupils to support learning in all other academic subjects. Critical Assumptions and Risks: The realization of the Initiative’s monitoring and evaluation plan will be premised on the following critical assumptions: 1. The implementing environment will continue to remain relatively stable: to achieve all project objectives and results, we assume that the level of insecurity and instability will not worsen to a critical point in the target states. We recognize, however, that we are working in an unstable environment, requiring both sensitivity and a flexible approach to prevailing conditions. 2. State/LGEA officials share the goals of The Initiative and are willing to make the requisite systemic, institutional and resource allocation changes. From the outset of the project, we will work with the state and local governments to identify and empower appropriate persons and units to participate and support the design, development, delivery and assessment work to ensure full integration into government operations and systems. 3. State government budgets and releases the budgeted amounts to contribute to the implementation of the designed intervention tailored toward the achievement of the goals. Design and Activity Scope 53 Household and schooling data collected from the NEDS, conducted in both 2010 and 2015, form the analytical basis of the Initiative activity design. In addition, the design was informed by the Focus States Strategy Assessment, the Mid-Term Performance Evaluation of the Leadership, Empowerment, Advocacy and Development (LEAD) Activity and, most importantly, the Northern Education Initiative (NEI) Mid-Term Performance Evaluation (2011). The Initiative focuses on the northern states of Bauchi and Sokoto. The activity operates in 10 LGEAs in each of the two states, making up a total of 20 LGEAs participating in the project. Criteria for the selection of target LGEAs include Gross Enrollment Ratio (GER), Net Enrollment Ratio (NER), literacy rate, number of out-of-school children, number of non-formal education learning centers, the percentage of qualified teachers in the state disaggregated by gender and local government; and the number of facilitators in the non-formal education centers. In addition to the LGEAs and states, the Initiative engages multiple institutional partners throughout the Nigerian educational system. Below, Graphic 1 outlines the detailed Initiative results framework and Table A lists key institutions for collaboration with a brief description of the role of each institution: USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 54 Graphic 1: NEI Plus PROJECT RESULTS FRAMEWORK 55 Table A: Partner Institutions and Roles Local Government Education Area (LGEA) ● Implement all programming in both reading and access ● Plan for and arrive at improved reading and access ● Collaborate with NEI+ activity personnel and CSOs to achieve improvements in reading and access State Universal Basic Education Boards (SUBEB) ● Coordinate the development of statewide policies, standards, frameworks, and monitoring tools for both reading and access ● Provide financing and logistical support to LGEAs for materials procurement, teacher training, monitoring, and testing State ministries of education State Ministry of Education (SMOE) ● Approve the final copies of statewide policies, standards, frameworks, and monitoring tools for both reading and access for use throughout the state ● Report to the Federal Ministry of Education on the progress in reading and access in the State Agencies for Mass Education (or the equivalents) and state ministries of religious affairs ● Lead the development and approval of appropriate materials for public schools and for non-formal learning centers Nigerian Educational Research and Development Council (NERDC) ● Ensure conformity of teaching and learning materials with national primary school curriculum Universal Basic Education Commission (UBEC) ● Ensure availability of basic education allocations to state ministries of finance National Council for Colleges of Education (NCCE) ● Approve introduction of teaching and learning materials for primary grades into pre-service coursework designed to prepare new primary teachers National Council for Colleges of Education (NCCE) ● Approve introduction of teaching and learning materials for primary grades into pre-service coursework designed to prepare new primary teachers Colleges of Education (COE) ● Ensure availability of basic education allocations to state ministries of finance Federal Ministry of Education (FME) ● Endorse decentralized efforts to improve reading and access ● Incorporate best practices into national-level plans USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 56 Expected Outcomes The two fundamental expected outcomes of the Initiative, per award Modification 7, include: (1) all children in target LGEAs access learning (via public school or NFLCs); and, (2) reading performance improves for children in grades one to three for children in all learning environments (public and NFLCs) where the activity operates. To achieve these two outcomes, LGEAs, State Universal Basic Education Board (SUBEB), and State Ministries of Education will work in partnership with the Initiative to ensure community mobilization for reading and access, improved data management targeting the expansion of access and improved reading performance. With the support of the Initiative, states and LGEAs should possess and use sets of validated materials for early grade reading in Hausa for grades one to three, for oral English instruction for grade two, and for English reading instruction for grade three. Furthermore, LGEAs will collaborate with partner CSOs to provide training for teachers in all school contexts on students’ social/emotional wellbeing and conflict resolution. With the support of the Initiative, states and LGEAs will possess and use research-based, criterion-referenced monitoring tools and classroom and student assessments that enable them to measure progress in expanding access, reducing risk, and improving early grade reading instruction. 5. INITIATIVE IMPLEMENTATION METHODS As expected by USAID, the Initiative throughout implementation is gradually applying and scaling-up evidence-based practices and approaches that have been proven effective in contexts similar to northern Nigeria. The Initiative currently plans for and implements the following approaches during project implementation: Strengthening systems through embedded LGEA and state teams: By embedding the Initiative’s staff in focal states and local level education units, the contractor addresses priorities for human and resource management for improved reading outcomes. The contractor has developed and implemented tools and metrics for rigorous monitoring of capacity transfer in improving reading. Reducing risks related to learning: Sexual violence, psychosocial difficulties and physical violence are highly prevalent in Northern Nigeria. All activities both formal and non-formal work with local government and communities to help schools meet risk reduction indicators that address school-based violence, gender bias, ethnic bias, and promote social cohesion, equity and conflict resolution/resolution. Integrating a conflict-sensitive approach: Education can help promote social cohesion, contribute to identity formation, build peace, and bridge the gap between humanitarian assistance and sustainable development. However, education can also undermine these processes. When it is not provided responsibly, education can be exclusionary, oppressive, exploitative and corrupt, and it can propagate extremism. All the Initiative’s activities in both the formal and non-formal sectors are designed to avoid reinforcing stereotypes and exclusion. Applying international, evidence-based best practices: The Initiative applies tested and proven strategies and techniques for improving reading outcomes, and reducing risks associated with education. These include community-based schooling; structured, phonetically-driven, sequenced reading instruction in languages children use and understand; teaching children at their instructional level and providing remediation where needed; ensuring children have a solid foundation in Hausa to facilitate reading acquisition in English; attention to education needs of underserved groups (OOSC and youth, girls, and 57 displaced populations); and do-no-harm provision of services for traumatized and endangered populations. Utilizing rigorous monitoring of progress toward outcomes: The Initiative is required to report on not only outputs (i.e., number of teachers trained), but also outcomes (i.e., percent of children achieving a passing score on an early grade reading assessment). Tools focused on quantifiable outcomes in reading are employed regularly in all the Initiative’s LGEAs. Data collected on increases in equitable access and improvements in learning outcomes are expected to be reported to USAID/Nigeria. 6. EVALUATION PURPOSE The end line evaluation is expected to build on the NEI Plus Midline Evaluation and use appropriate methods in order to analyze progress over the life of the activity. The end line evaluation is intended to deepen USAID’s understanding of what was achieved and how it was achieved. The Government of Nigeria, USAID, and its implementing partners will use the evaluation findings to make decisions for possible follow-on activities. The audience of the evaluation report will be USAID/Nigeria, specifically the Education Office, implementing partners, key officials from the federal, state and local government levels, and other development partners. An Executive Summary narrative with recommendations and a PowerPoint presentation will be provided to USAID/Nigeria in advance of the full report completion. These documents will synthesize findings and recommendations for future programming. The effectiveness of the Education Office’s strategy of investing in systems strengthening at the state and local levels to improve the quality of education should be explored. 7. EVALUATION QUESTIONS Final research questions will be determined in a meeting between USAID/Nigeria and the Contractor during the initial three days of the contract period. Preliminarily, the following categories and illustrative questions outline the range of areas the evaluation is purposed to explore: 1. To what extent did the dialect of Hausa, chosen as the Language of Instruction (LOI) in Bauchi and Sokoto target schools, impact learning outcomes in affiliated schools of the Initiative? How did the Initiative plan for dialectal variations within target schools and mitigate any inequities? 2. To what extent did “Fidelity of Implementation” factor into programmatic decisions, outcomes, and the overall impact of the Initiative? 3. How well did the Initiative influence factors affecting access to quality education for out-of￾school children in target communities? What were the challenges to mainstreaming and retention, and how were they mitigated? 4. To what extent are state and local education authorities able to sustain activities of the Initiative, across all Initiative Result Areas, without additional USAID investment? Why? 5. In what ways did the Initiative achieve institutionalization of efficiencies for more effective educational management by state and local education authorities? Where gaps exist, what factors limited progress? USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 58 6. To what extent has the Initiative integrated a conflict-sensitive approach that addressed or reduced the safety concerns of communities and beneficiaries (e.g., school-related violence, gender-based violence, etc.), including but not limited to learners, parents, school-based management committees, etc.? 8. EVALUATION APPROACH Available Data As earlier noted, results from an Early Grade Reading Assessment conducted in 2013 are available for consideration. The assessment was administered in grades two and three government and Tsangaya (Qur’anic) schools to 1,881 pupils in Bauchi and 1,674 pupils in Sokoto states. Approximately 92 percent of 2nd graders in each of the two states could not read a single word on the test administered in the Hausa language, which a vast majority of the children tested do speak and understand. Additionally, a midline assessment was conducted 2018 under the Initiative in which 1,210 2nd graders and 1,198 3rd graders were administered the EGRA test. Grade two pupils were administered the Hausa EGRA version only whereas the grade three pupils took both Hausa and English versions. Grade two students performed better on average in both states than their counterparts in nonintervention settings as compared with Initiative baseline in five of six subtests. However, despite the notable improvement on subtests and positive reduction in zero scores, in all subtests other than dictation over half of sampled Primary two students in Bauchi and Sokoto were unable to read or respond to a single question-item correctly. Primary three students scored even stronger increases in learning from baseline on average in Hausa. Regretfully however, roughly 65 percent of sampled pupils could not answer a single comprehension question correctly. English results at the grade three level were less favorable. Understanding of English remains a substantial constraint as grade three students prepare for English as the medium of instruction in grade four. Zero scores for English listening comprehension hovered near 80 percent for both states, while zero scores for reading comprehension were in the range of 75 percent and 85 percent in Bauchi and Sokoto respectively. Possible factors limiting the impact of the Initiative are complex and multidimensional. High rates of absenteeism among teachers and students, insufficient time on task, and low capacity and motivation of the educational officials responsible for coaching are among the possible compounding factors driving these results. The Evaluation team will have access to the original NEI+ Contract and all programmatic modifications (Mod 4 and Mod. 7). Otherwise, the evaluator(s) will have access to data captured in the Initiative’s workplans, performance monitoring plan, quarterly and annual reports, and state government education sector reports. In order to answer questions related to cost and financial commitments from government, the evaluator(s) will need to work with the implementing partner and state education authorities to gather relevant financial and cost data. In addition, the evaluators will have access to the NEI+ Midline Evaluation report and data. 59 Evidence of Participatory Approach Key stakeholders in government and the education sector community will be involved in the validation of findings, the development of conclusions, and the development of final recommendations. An evaluation findings validation workshop will be held with key stakeholders where draft reports, summary reports or excerpts will be discussed, validated, and made available for consideration and follow-up. The strategy for identifying and finalizing these actors will be decided in consultation with USAID/Nigeria, most specifically under the leadership of the Education Office. Dissemination Plan All materials will be made available to USAID/Nigeria’s Education Team electronically on a secure USB Drive (e.g., Iron Key encrypted USB) and published on the Development Exchange Clearinghouse (DEC) by the evaluators. See additional guidance on the formatting of the report in the section on Reporting Guidelines below. 9. EVALUATION DESIGN AND METHODOLOGY This is an end line performance evaluation design. The evaluation will use a non-experimental design and employ a range of quantitative and qualitative data collection methods in answering the evaluation questions to be finalized after onboarding of evaluators. Although predominantly qualitative, the evaluator(s) should make recommendations based on a mixed-methods approach involving (1) a desk review of available primary documents; (2) semi-structured key informant interviews; and (3) site visits to schools involving semi-structured key informant interviews, focus groups, and mini-surveys and direct observation methods as appropriate for acquiring data from respondents. Data Collection Methods The contractor will use an appropriate mix of qualitative and quantitative data collection methods to answer the evaluation questions and to understand the accomplishments and achievements of the Initiative. The methodology may include key informant interviews and focus group discussions with a sample of activity stakeholders and beneficiaries. This may include teachers, instructors, administrators, school directors, local officials, communities, parents and/or parent teacher associations, as well as Ministry of Education officials and members of Civil Society Organizations (CSOs) in Bauchi and Sokoto states, complemented by a thorough desk study of relevant documentation and performance indicators. Key approaches that would be used to collect and analyze data for the evaluation would be as follows: review of background materials; project documents relevant to the evaluation for review and analysis including project designs, scopes of work, annual and quarterly reports, annual work plans, technical and training materials. As needed, the evaluator(s) will work with USAID and the Initiative’s CO to obtain approval for contacting schools and other beneficiaries in order to alert them of the impending data collection activities. The contractor would organize a pre-data collection visit to selected schools to: a) gain advance entry into the schools and explain the purpose of the data collection, b) verify the eligibility of the schools for the intervention and data and c) obtain student lists and contact information of parent teacher student association members and principals in order to facilitate USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 60 contacting parents during the field period. Logistics related to data collection must include advance state/LGEA approval for entry. USAID/Nigeria expects the evaluator(s) to propose a detailed data collection plan, including sampling methodology and final site/school selection, in the Evaluation Work Plan. This will be reviewed by the USAID Education Development Unit and Program Unit and approved by the COR. Data Analysis Methods USAID/Nigeria expects the evaluators to propose an outline of the evaluation’s data analysis plans which will be reviewed and approved by the Mission’s Education Development Unit and program teams. It is expected that this plan will directly address each evaluation question with specific methods for collecting and analyzing the data that will be used to generate findings, conclusions and recommendations. Also, it is expected that all data reported in this evaluation, including outcomes and input data that illustrates the degree to which reading and access outcomes are impacted and improved in the focal sites, and as a result of the Initiative’s system strengthening, must be reported on a sex-disaggregated basis. Methodological Strengths and Limitations 10. EVALUATION PRODUCTS Required Deliverabl es Deadlines (contingent upon another prior deliverable or deadline) To Whom Remarks 1 Work Plan including the Methodolog y Plan No later than 6 days of work. (During the Technical Planning Meeting prior to implementation) To be submitted to the COR at USAID/Nigeria for approval A detailed work plan which will include the methodologies to be used in the evaluation 2 Present Preliminary Findings Midway during the data collection process USAID for preliminary comments A PPT with preliminary findings to provide some insights to USAID 3 Debriefing with USAID An out brief for USAID before the evaluation team’s departure from the country (This should provide findings at length for USAID). USAID/Nigeri a Presentation of major findings of the evaluation through a PPT presentation and will include a discussion of achievements and issues as well as any recommendations for possible modifications to project approaches, results, or activities will consider and revise the draft report accordingly, as appropriate. 4 First Draft Evaluation Report USAID/Nigeria who will provide comments within two Draft written report of the findings and recommendations; clearly describing findings, conclusions, and recommendations. 61 weeks of submission. 5 Final Report No later than five days after USAID/Nigeria provides written comments on the ET’s draft evaluation report USAID/Nigeria Report to incorporate the team responses to Mission comments and suggestions. Report format to include: -executive summary table of contents methodology, findings, and recommendations. Report will be disseminated within USAID. A second version of this report excluding any potentially procurement-sensitive information will be submitted (also electronically, in English) for dissemination among implementing partners and stakeholders. Reporting Guidelines USAID’s evaluation policy requires that all evaluation scopes of work (SOW) include USAID’s Criteria to Ensure the Quality of the Evaluation Report (see USAID Evaluation Policy Appendix I). In addition to adhering to the requirement criteria stipulated by the ADS, (appendix 1), it is expected that the format for this evaluation report will be as follows: 1. Executive Summary: concisely state the most salient findings and recommendations (2 pp); 2. Table of Contents (1 pp); 3. Introduction: purpose, audience, and synopsis of task (1 pp); 4. Background: brief overview of the NEI+ project, USAID project strategy and activities implemented in response to the problem, brief description of the implementing partner/s, purpose of the evaluation (2–3 pp); 5. Methodology: describe evaluation methods, including constraints and gaps (1 pp); 6. Findings/Conclusions/Recommendations: for each objective area; and also, include data quality and reporting system that should present verification of spot checks, issues, and outcome (17–20 pp); 7. Issues: provide a list of key technical and/or administrative, if any (1–2 pp); 8. Future Directions (2–3 pp); 9. References (including bibliographical documentation, meetings, interviews and focus group discussions); 10. Annexes: annexes that document the evaluation methods, schedules, interview lists and tables should be succinct, pertinent and readable. USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 62 The final version of the evaluation report will be submitted to USAID/Nigeria in hard copy as well as electronically via an encrypted USB (e.g., IronKey). The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins 1” top/bottom and left/right, formatted for LETTER sized. The report should not exceed 60 pages, excluding references and annexes. Report Delivery: The evaluation Team Leader shall incorporate USAID’s comments and submit the final report to USAID/Nigeria in electronic format (Microsoft Word) on an encrypted USB (e.g., IronKey) as well as printed and bound copies (five copies in English) no later than 15 working days of the receipt of the comments. The evaluation Team Leader shall submit one either electronic or hard copy to the Development Experience Clearinghouse at http://dec.usaid.gov or M/CIO/KM, RRB M01, USAID, Washington DC 20523. 11. TEAM COMPOSITION The intended size of the evaluation team, the roles and responsibilities of team members, and the specific qualifications that team members are expected to possess for this evaluation is yet to be finalized, but notwithstanding, as required by the ADS, at least one member of the evaluation team (possibly the team leader) must be an evaluation specialist and must possess significant experience in designing evaluations and a strong understanding of data collection and analysis methodologies. Also, the team leader is further expected to provide samples of past evaluation reports produced under his/her direction. In addition to the team leader, the evaluation team will consist of 2 - 4 other technical experts who should consist of expatriate and local consultants. A representative from the USAID Nigeria education team, SUBEB/MoE and/or LGEA representative should be asked to participate as well. All attempts should be made for the team to be comprised of an equal number of male and female members. Personnel Qualification Team Leader/Senior Evaluation Expert (International) – S/he should have a master’s degree in education, international development, or an applicable field. S/he should have at least 5 years senior level experience working in improving the quality of early grade reading at basic education projects in a developing country including Sub Saharan Africa. S/he will work together with team of experts comprised of Education/Conflict Expert (international), local expert(s) and local logistics assistant (optional). S/he should have strong technical expertise in monitoring and evaluation methodology, data collection methods, including qualitative and quantitative data; extensive expertise in the area of basic education and significant international experience. S/he will elaborate the methodology for evaluating the NEI+ project and write a Design Plan, including sampling and data collection tools. Strong organizational skills, including assessing priorities, strategic planning, meeting deadlines and attention to detail and quality is required. Education Sector Expert (International) – The person should possess substantial 63 12. LEVEL OF EFFORT Tentative Timeline of Activities Item Number of Days Period of Performance (timeframe) Team Leader/ International Evaluation Expert Local Reading Expert Local Evaluator/ Basic Education Expert Local Evaluator/ Education Sector International travel to Abuja, Nigeria Nov 1 1 0 0 0 Quarantine period while Review background documents/ Desk Review & preparation work. Conduct remote interviews and discussions with activity staff; ensure that activity staff provide initial contact lists of beneficiaries, GoN officials, other donors, and other USAID activity representatives as needed. Most of this work will be done through emails and Skype. The team will work though USAID and Initiative’s COP to set as many meetings and interviews as possible prior to arrival in Nigeria. Preparation and submission of draft workplan & draft data collection tools (protocols) Nov 2-9 6 6 6 6 Workplan submitted to MEL for Review Nov 9-10 0 0 0 0 MEL Reviews Workplan/Protocol with comments Nov 11 1 1 1 1 MEL reviews comments with Evaluators Nov 12 1 1 1 1 Workplan/Protocols submitted to USAID for review Nov 13 0 0 0 0 USAID reviews Workplan/Protocol/ Evaluation team continues with on￾boarding and planning Nov 16-17 2 2 2 2 In-brief/ Team planning meetings with USAID Nov 18 1 1 1 1 In-brief with/ Implementing Partner and education officials in Abuja and making arrangements for field work Nov 19-20 2 2 2 2 Sub-teams travel Team A: Sokoto Team B: Bauchi Nov 21 1 1 1 1 Morning in-brief with State Activity Leads and staff; finalization of local site visit schedule/ site visit Nov 23-Dec 5 12 12 12 12 Sub-team review meetings and initial internal write-ups; coordination with team lead USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 64 Data collection and observations in schools/ NFE centers; visits to LGEA offices Travel to Abuja Dec 6 1 1 1 1 Preparation of the draft evaluation report, preliminary findings/PPT Dec 7-15 7 7 7 7 Submission of presentation to MEL; continuation of report draft Dec 16 1 1 1 1 MEL reviews submitted report of Presentation; evaluators continue report draft Dec 17-18 2 2 2 2 MEL submits presentation to USAID; evaluators continue report draft Dec 21 1 1 1 1 Presentation of preliminary, exit brief and submission of report draft to MEL Dec 22 1 1 1 1 International travel Dec 23 1 0 0 0 Christmas/New Year break Dec 24-Jan 1 0 0 0 0 USAID reviews report and provides comments to draft report Jan 4 0 0 0 0 Incorporation of USAID comments Jan 11-13 3 3 2 2 Submission of draft to MEL Jan 13 0 0 0 0 Submission of draft evaluation to USAID by MEL Jan 18 0 0 0 0 USAID reviews and submits comments on final draft evaluation report Jan 19-25 0 0 0 0 Team addresses USAID comments and finalizes the report Jan 25-29 4 3 1 1 Report to MEL Feb 2 0 0 0 0 Final report due to USAID/Nigeria Feb 6 0 0 0 0 TOTAL LOE 48 44 42 42 65 Scheduling USAID/Nigeria will be responsible for information to the evaluation team on any scheduling restraints that could affect the evaluation. This is in addition to providing (at a later date) the expected period of performing this evaluation. Timeline Task Accomplishment Duration Pre Field-Work Obtain key documents, make key contacts, and plan for interviews and discussions in Nigeria with project staff, beneficiaries, GoN officials, other donors, and other USAID project reps as needed. Most of this work will be done through emails and Skype. The team will work though USAID and Initiative’s COP to set as many meetings and interviews as possible prior to arrival in Nigeria. A part-time local hire can be brought onboard to assist with this process. 11 days Week 1 In-Brief and Planning This period will consist of meetings with USAID, the staff of the Initiative, gathering and reviewing data not already available, and solidifying plans for visits to states and focal LGEAs. In the latter part of this week the interview process with beneficiaries and others will begin including Federal agencies and educator sector stakeholders. 4 days Week 2 Field Work The focus of this entire week will be on interviews and discussions with beneficiaries, donors, government officials, representatives of related USAID projects, and others who work with or have been impacted by the Initiative. Team members will visit and assess activities in at least six LGEAs that have been involved in Initiative activities. Those selected will vary by key qualities like geographical location (rural/urban) and such related. As time allows, the team will begin preparing the first few sections of the final report on the background, setting, and previous evaluative efforts related to the set of the Initiative. 14 days Week 3 Field Interviews, Draft Report, and Debrief Any remaining interviews will be completed. Follow-up meetings to discuss questions arising from the interviews and to clarify and remaining issues will be held with the implementation teams of Initiative, analyze data and findings; draft report for USAID comment and De-brief USAID. 13 days Week 4 Post Field Work Finalize draft final report and submit to USAID/Nigeria no later than two weeks following receipt of final comments from the Mission. 21 days USAID.GOV NORTHERN EDUCATION INITIATIVE (NEI+) END LINE PERFORMANCE EVALUATION REPORT | 66 13. EVALUATION MANAGEMENT Logistics USAID/Nigeria will provide overall direction to the evaluation team, identify key documents, and assist in facilitating the evaluation work plan. USAID/Nigeria will also assist in arranging meetings with key stakeholders identified by the Team in consultation with the Mission prior to the initiation of this field work. The evaluation team is expected to be responsible for arranging other meetings as identified during the course of this evaluation and but advising USAID/Nigeria prior to each of those meetings. The evaluation team will also be responsible for renting vehicles and drivers as per their need for site visits in-country as well as all logistics concerning lodging and field needs. The evaluation team will be responsible for procuring its own work/office space, computers, internet access, printing, and photocopying. Evaluation team members will be required to make their own payments. USAID/Nigeria personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. The evaluation team must emphasize safety and security in all actions related to execution of the assignment. USAID Evaluation Policy - Appendix 1 Criteria to Ensure the Quality of the Evaluation Report ● The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why. ● Evaluation reports shall address all evaluation questions included in the scope of work. ● The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the technical officer. ● Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. ● Evaluation findings will assess outcomes and impact on males and females. ● Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. ● Sources of information need to be properly identified and listed in an annex. ● Recommendations need to be supported by a specific set of findings. ● Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. ANNEX C: EVALUATION WORKPLAN AND DATA COLLECTION TOOLS 67 NORTHERN EDUCATION INITIATIVE PLUS (NEI+) END LINE PERFORMANCE EVALUATION WORK PLAN NEI PLUS STATE OFFICE, BAUCHI 68 NORTHERN EDUCTION INITIATIVE PLUS (NEI+) END LINE PERFORMANCE EVALUATION: WORK PLAN Draft Report Submitted: 14 December 2020 Contract No: AID-OAA-I-15-00018/620-TO-16-00001 PREPARED FOR THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT IMPLEMENTED BY: DEVTECH SYSTEMS INC. 1700 N. MOORE STREET SUITE 1720 ROSSLYN, VA 22209 DISCLAIMER This report is made possible by the support of the American People through the United State Agency for International Development (USAID). The contents of this report are the sole responsibility of DevTech and do not necessarily reflect the views of USAID or the United States Government. 69 CONTENTS ACRONYMS 5 EXECUTIVE SUMMARY 6 INTRODUCTION 6 EVALUATION METHODOLOGY 6 DATA ANALYSIS 6 METHODOLOGICAL LIMITATIONS 7 FINDINGS AND CONCLUSIONS 7 INTRODUCTION 11 BACKGROUND OF LOCAL CONTEXT 11 ACTIVITY OVERVIEW 11 KEY PARTNERS AND ROLES 13 EVALUATION PURPOSE & QUESTIONS 14 EVALUATION METHODOLOGY 15 DESIGN 15 DATA COLLECTION 16 DATA ANALYSIS 16 EVALUATION LIMITATIONS 17 FINDINGS AND CONCLUSIONS 18 EVALUATION QUESTION 1 18 EVALUATION QUESTION 2 24 EVALUATION QUESTION 3 29 EVALUATION QUESTION 4 35 EVALUATION QUESTION 5 39 ANNEXES I ANNEX A: EVALUATION SCOPE OF WORK II ANNEX B: EVALUATION WORKPLAN AND DATA COLLECTION TOOLS XX 70 ACRONYMS BESDA Better Education Service Delivery for All CBMC Center-based management committee CDCD Country Development Cooperation Strategy COP Chief of Party CSO Civil society organization EGR Early grade reading EGRA Early Grade Reading Assessment FGD Focus group discussion FLC Formal learning center KII Key informant interview LGA Local Government Authority LGEA Local Government Education Authorities NCCE National Commission for Colleges of Education NEDS National Education Data Survey NEI+ Northern Education Initiative Plus NERDC Nigerian Education Research and Development Council NFLC Non-formal learning center OOSC Out-of-school children SAME State Agency for Mass Education SBMC School-based management committee SMOBSE/SMOE State Ministry of Basic and Secondary Education STWG State Technical Working Group SUBEB State Universal Basic Education Board TLMs Teaching and learning materials UBEC Universal Basic Education Committee USAID United States Agency for International Development 71 SECTION 1: Background 1.1. Context and Background Nigeria is critically important to the global stage as the largest economy, largest democracy, and most populous country on the African continent. Nonetheless, Nigeria faces daunting developmental challenges that threaten its potential as a world leader. Nearly 65 percent of Nigerians live in extreme poverty. Education sector statistics are equally underwhelming with an estimated 10.5 million Nigerian out-of￾school children (UNICEF OOSC 2012)7 between the ages of 5 and 14, the highest number in the world. Those children attending school are likewise inadequately learning. According to World Bank data (2018), 81 percent of young adults with only a primary education are unable to read8. Poor quality teaching and learning environments directly correlate with these low achievement levels. Crumbling physical infrastructure, a scarcity of teaching and learning materials (TLMs) and inadequate teacher training or professional development opportunities are drivers of educational failures. Regional disparities exist too, with northern Nigeria states performing well below the national average across education indicators such as attendance and completion rates. The 2015 National Education Data Survey (NEDS)9, facilitated by the National Population Council, net Attendance Ratio (NAR) of the combined North West and North East Regions was 46.6 percent, meaning that approximately 53.4 percent of the total number of children in the north of primary school age were not attending school. For example, 56.1 percent of children in Bauchi and 68.7 percent in Sokoto had never been to school, compared to the national average of 24.1 percent. Similarly, primary completion rates in Bauchi and Sokoto are 30.8 percent and 23.7 percent respectively, with the national average at 68.9 percent. Outcomes vary by gender as well, with NAR at 31.8 percent and 36.5 percent for girls in the north-east and north-west, respectively. In other words, more than half of girls in the northern region are not in school. In 2016, an Early Grade Reading Assessment (EGRA) baseline study, in Hausa and English and in grades two and three were conducted in Bauchi and Sokoto states. The results indicate that children do not have adequate foundational reading skills in Hausa, and those children in Primary 3 had not yet acquired them in English. The majority of students scored 0, meaning they could not provide a single correct response, on almost every subtask in the assessment. The large percentage of zero scores of course rendered the mean scores across subtasks very low, with almost no children read with 80 percent comprehension. Note that the use of early grade reading strategies had yet to occur, rather children were taught language (Hausa and English) skills but not reading as a subject. As a result, it is understandable that an assessment of reading ability result in low scores. The 2018 midline EGRAs showed a slight improvement in results and a decrease of zero scores in both states across the subtasks in Hausa and English. However, over half of the students in Primary 2 and 3 were still unable to read or respond to a single question-item correctly. The report concluded that many targeted students were not yet being effectively reached by the Initiative’s package of EGR interventions. 7 UNICEF (March 2012), Nigeria Country Study Conducted within the Conceptual and Methodology Framework (CMF) http://uis.unesco.org/sites/default/files/documents/out-of-school-children-nigeria-country-study-education￾2012-en.pdf 8 2018. Learning-Adjusted Years of Schooling (LAYS). World Bank 9 Nigeria 2015 Nigeria Education Data Survey (NEDS), https://ierc￾publicfiles.s3.amazonaws.com/public/resources/2015-NEDS-National-011716.pdf 72 1.2. PROJECT DESCRIPTION The Northern Education Initiative Plus (NEI+) is a five-year activity funded by USAID/Nigeria to strengthen the ability of Bauchi and Sokoto states to provide access to quality education and significantly improve the reading skills for more than one million school-aged children and youth, especially girls, orphans, and children enrolled in nontraditional schools. The Initiative employs a holistic approach, addressing a broad range of critical factors that affect learning, teaching, systems management, parental participation, and community engagement across two core intermediate results. Intermediate Result 1: Government systems strengthened to increase the number of students enrolled in appropriate, relevant, approved educational options, especially girls and out-of-school children in target locations, and Intermediate Result 2: Government systems strengthened to improve reading outcomes for primary grade learners in target locations. The Initiative focuses on strengthening technical and administrative capacity, commitment and accountability of federal, state and Local Government Education Authorities (LGEAs) to provide effective English and mother-tongue-based Early Grade Reading (EGR) teaching and learning to pupils; and to deliver literacy, numeracy, and life-skills to OOSC and youths. Whenever possible, interventions strive to mainstream these youth into formal schools for onward learning and social mobility. Regarding capacity strengthening, LGEAs are supported to plan, budget, supervise teachers, deliver high-quality TLMs and mobilize community involvement for increased enrollment and improved reading outcomes. Colleges of education are also empowered to integrate best practice EGR concepts, techniques and materials into their curriculum, while playing a key role within the education sector to promote EGR policy and research. Specific to strengthening efficiencies related to teacher payments, the Initiative also integrates the technology of mobile money to regulate payments and reduce absenteeism related to extended long-distance travel of individuals to collect salaries. The Initiative received two technical contract modifications over the life of the Activity. In March 2018, while maintaining the two core strategic objectives of the Activity, Modification 4 reduced the number of NEI+ implementation states from three to two and enhanced the focus on government systems strengthening, policy reform, and LGEA accountability and transparency. These modifications were Prime Implementing Partner: Creative Associates International Geographical Coverage: Bauchi and Sokoto States Activity Start and End Date: Oct 26, 2015 – Oct 25, 2020 and extended to May 25, 2021 Exhibit 1: Project Information Exhibit 2: CDCS Development Objectives DO 2: A healthier, better educated population IR 2.2: Improved learning outcomes in targeted states DO 3: Accountable, inclusive, and responsive governance strengthened IR 3.4: Increased accountability and effectiveness of public institutions 73 intended to ensure increased sustainability of USAID investments in the two target states. In May 2019, Modification 7 expanded the scope of NEI+ to include development of English language materials for Primary One (P1). SECTION 2: Evaluation Overview USAID awarded DevTech Systems, Inc. a task order to implement an End line Performance Evaluation of NEI+ on February 5, 2020. The evaluation will be divided into three work phases: 1) inception, 2) data collection and analysis, and 3) validating and reporting. The Inception Phase, leading to this workplan, took place from November 26 –December 14, 2020. During this period, the DevTech evaluation team (ET) reviewed background documents and conducted a stakeholder analysis to identify key respondent categories and conducted remote discussions to design the evaluation methodology and create tools for data collection. Due to the COVID-19 pandemic, all data collection will be collected virtually except for classroom observations which were re-added after the schools opened. Subsequent sections of this workplan describe the data collection and analysis, and validation and reporting phases in more depth. 2.1. Evaluation Purpose, Questions, and Use The purpose of the End line Performance Evaluation is to build on the Midline Evaluation to establish the progress of NEI+ against its objectives, and to produce lessons learned for application in future programming. Specifically, the evaluation will: ● Assess the extent to which the activity met its key objectives. ● Provide a better understanding of the progress made by each component of NEI+ on issues such as relevance, sustainability and cost-effectiveness. ● Identify implementation challenges, unmet needs, and unintended consequences or impacts of NEI+. The Performance Evaluation will achieve these goals by responding to the following five key evaluation questions at the level of the two NEI+ intermediate results mentioned above. The evaluation questions are taken from the six questions in the Scope of Work; review during the inception phase led to questions 4 and 5 being combined. Evaluation Questions 1. To what extent did the dialect of Hausa, chosen as the Language of Instruction (LOI) in Bauchi and Sokoto states target schools, impact learning outcomes in affiliated schools of the Initiative? How did the Initiative plan for dialectal variations within target schools? How were any inequities mitigated? 2. To what extent did “Fidelity of Implementation” factor into programmatic decisions, outcomes, and the overall impact of the Initiative? 74 3. How well did the Initiative influence factors affecting access to quality education for out-of-school children in target communities? What were the challenges to mainstreaming and retention, and how were they mitigated? 4. In what ways did the Initiative achieve institutionalization of efficiencies for more effective educational management by state and local education authorities? Where gaps exist, what factors limited progress? To what extent are these activities and activities across all Initiative Result Areas sustainable without USAID investment? Why? 5. To what extent has the Initiative integrated a conflict-sensitive approach that addressed or reduced the safety concerns of communities and beneficiaries (e.g., school-related violence, gender￾based violence, etc.), including but not limited to learners, parents, school-based management committees, etc.? The audience of the evaluation report will be USAID/Nigeria, specifically the Education Office, implementing partners, key officials from the national, state, local government and community levels, and other development partners. An Executive Summary narrative with recommendations and a PowerPoint presentation will be provided to USAID/Nigeria in advance of the full report completion. These documents will synthesize findings and recommendations for future programming. The Government of Nigeria, USAID, and its implementing partners will use the evaluation findings to make decisions for possible follow￾on activities. 2.2 Evaluation Approach and Design The evaluation will use a mixed-method approach in which more than one method for data collection will be used to answer the five (5) key questions. The methodological approach will be predominantly qualitative as prescribed in the SOW and will draw upon a range of primary and secondary data. The primary data will consist of qualitative data collection methods including semi-structured key informant interviews (KIIs), focus group discussions (FGDs). Classroom observations will be conducted at the end of the planned data collection and the timeline has been revised accordingly. The secondary will include reviews of key documents and government and other relevant organizational data. Every attempt will be made to interview the respondents in the plan, but in case they are not available, suitable substitutions may be made. NEI+ has provided a list of people to interview and will be consulted if someone is not available to find alternatives. For teachers and facilitators, the head teachers or center coordinators will be instrumental in providing names and a random sampling will be done. Triangulating field data with data from program reports, and previous evaluations will guide the identification of findings, conclusions and recommendations in response to the evaluation questions and the findings in the mid-term evaluation. Triangulation will also enable the evaluators to identify gaps, and missed opportunities. The mixed-method approach aims to provide a holistic understanding of the project’s end line status. Standard data collection tools and data analysis templates will be developed to ensure quality control across collected data. The team will confirm proposed interview and focus group discussion times and make necessary arrangements for permission to virtually access civil society groups, key informants and other relevant stakeholders. This will either be done in calls to groups if stakeholders are able to meet with social distancing, or individually by phone. During field work the team will divide into two teams, Team A and Team B, to cover both 75 states. Team members will be responsible for collecting data on specific components in parallel with the 5 key evaluation questions. Virtual interviews will be highly participatory for the purpose of engaging key stakeholders and beneficiaries and to promote learning. Interactive processes can facilitate reflection and can result in mutual learning. All respondents will be included if it is a group call and evaluators will make sure to involve everyone. The following chart illustrates data collection methods and sources aligned with the 5 key evaluation questions. DATA SOURCES Key Evaluation Question Data Collection Methods Data Collection Source 1. To what extent did the dialect of Hausa, chosen as the Language of Instruction (LOI) in Bauchi and Sokoto states target schools, impact learning outcomes in affiliated schools of the Initiative? How did the Initiative plan for dialectal variations within target schools? How were any inequities mitigated? ● Key informant interviews ● Focus group discussions ● Document review ● NEI+ ● STWG ● LGEA ● LGA ● NCCE ● BESDA ● SMOBSE/SMOE ● SAME ● NERDC ● SUBEB ● NFLCs ● LGAs ● Head teachers ● Trained teachers ● Community members ● Classroom Observations 2. To what extent did “Fidelity of Implementation” factor into programmatic decisions, outcomes, and the overall impact of the Initiative? ● Key informant interviews ● Focus Group Discussions ● Document review ● NEI+ ● FME ● NERDC ● BESDA ● NCCE ● SMOBSE/SMOE ● SAME ● LGA ● STWG ● UBEC ● LGEA ● SUBEB ● NFLCs ● LGAs 76 ● CSOs ● Grantees 3. How well did the Initiative influence factors affecting access to quality education for out-of-school children in target communities? What were the challenges to mainstreaming and retention, and how were they mitigated? ● Key Informant Interviews ● Focus Group Discussions ● Document review ● NEI+ ● FME ● NERDC ● BESDA ● NCCE ● SMOBSE/SMOE ● SAME ● LGA ● STWG ● UBEC ● LGEA ● SUBEB ● NFLCs ● LGAs ● CSOs ● Grantees ● Head teachers ● Trained teacher ● Community members 4. In what ways did the Initiative achieve institutionalization of efficiencies for more effective educational management by state and local education authorities? Where gaps exist, what factors limited progress? To what extent are these activities and activities across all Initiative Result Areas sustainable without USAID investment? Why? ● Key Informant Interviews ● Focus Group Discussions ● Document review ● NEI+ ● SAME ● NCCE ● NERDC ● BESDA ● LGA ● SMOBSE/SMOE ● UBEC ● LGEA ● SUBEB ● NFLCs ● LGAs ● CSOs ● Grantees 5. To what extent has the Initiative integrated a conflict￾sensitive approach that addressed or reduced the safety concerns of communities and beneficiaries (e.g., school￾related violence, gender-based violence, etc.), including but not ● Key Informant Interviews ● Focus Group Discussions ● Document review ● Project Documents ● NEI+ ● LGEA ● SMOBSE/SMOE ● BESDA ● SUBEB ● LGAs ● CSOs ● Grantees 77 limited to learners, parents, school-based management committees, etc.? ● Community members Each data collection method is described briefly below. DESK REVIEW The team will conduct a desk review beginning during the inception phase and continuing during data collection. The desk review provides context for the evaluation and will consolidate secondary source data that respond to evaluation questions. The scope, and focus, of the desk review involves a thorough analysis of three types of documents: 1) NEI+ project documents and reports, 2) NEI+ Midline Evaluation report and data, 3) State government education sector reports 4) Relevant financial and cost data from the implementing partner and state education authorities. The complete list of documents the evaluation team has received to date are provided in Annex One. VIRTUAL DATA COLLECTION Virtual data collection is designed to: I) elicit qualitative data on stakeholder perspectives on the most important achievements, factors contributing to or inhibiting success, and areas for improvement through KIIs and FGDs. 2) obtain qualitative data on the effects of the teacher training initiatives through classroom observations. The evaluation team will work remotely and collect data over a period of two weeks; the team will consist of a team leader, a basic education expert/evaluator, a reading expert/evaluator, an education expert/evaluator, a data analyst, and two local evaluation assistants. Methods used to collect data will be: phone calls or Google Meet calls for semi-structured key informant interviews with a sample of activity stakeholders and beneficiaries; and calls to schools involving semi￾structured key informant interviews (KIIs), and focus group discussions (FGDs). The interviews and discussions will include officials from: Federal Ministry of Education (FME), State Ministry of Basic and Secondary Education (SMOBSE/SMOE); Nigerian Education Research and Development Council (NERDC); Local Government Education Authority (LGEA); State Universal Basic Education Board (SUBEB); State Agency for Mass Education (SAME); Universal Basic Education Committee (UBEC), State Technical Working Group (STWG). Head Teachers, Formal Learning Center (FLC) teachers and pupils, Non-Formal Learning Center (NFLC) facilitators, Representatives from: Civil Society Organizations (CSOs); School-based Management Committees (SBMCs); Center-based Management Committees (CBMCs); Parent Teacher Associations (PTAs); Better Education Service Delivery for All (BESDA); and NEI+. Full data collection tools are in Annex Four. CLASSROOM OBSERVATIONS 78 Classroom observations will be conducted using the Classroom Observation Protocol for formal or non￾formal education settings, as appropriate for acquiring data from respondents. Section 3: Methodology 3.1. Sampling Design DATA COLLECTION TOOLS MATRIX Participants for the KIIs will be chosen using a purposive sampling approach based on the positions they hold in their organizations. Individuals will be chosen who have a good background and working knowledge of NEI+’s activities and purpose. If participants initially chosen are not available, others will be chosen. Heads of organizations will be asked to recommend those who worked most closely on the project. For teachers and facilitators, they will be chosen if they have completed NEI+ trainings. Participants for FGDs will be chosen using purposeful or convenience sampling. For example: members of School Board Management Committees or PTAs may be chosen for their role in the organizations but the people available at the right time may lead to convenience or snowball sampling. There are 14 data collection tools, with Tool Two divided into six sub-tools. Tools 1-9 are divided into categories of respondents, Tool 10 is a comprehensive list of all questions to ask to NEI+ in Abuja and the states, Tool 11 is specifically for finance and budgeting questions and will be used to gather data during interviews with individuals holding relevant positions. Tool 12 is an interview for teachers and facilitators to try to ascertain their knowledge of NEI+ training methodology and its use in their classrooms. Tools 13 and 14 are checklists for lessons observed in formal and non-formal classroom settings respectively. The list of data collection tools is in the table below: DATA COLLECTION TOOLS TOOL 1: HIGH LEVEL/POLICY MAKERS ● NERDC ● NEI+ Desk Officer ● SAME ● Permanent Secretary ● STWG ● Chairman ● Secretary ● Other Members ● SMOBSE/SMOE ● Commissioner ● Permanent Secretary ● SUBEB ● Executive/Permanent Secretary ● BESDA ● NEI+ Desk Officer ● LGA ● The Executive Chairman ● The Secretary ● LGEA ● Education Secretary 79 TOOL 2: TECHNICAL TOOL 2A: DESK OFFICERS ● FME ● NEI+ Desk ● NCCE ● NEI+ Desk Officer/ Director Academic Planning ● NERDC ● TWG (NEI+) Desk ● UBEC ● NEI+ Desk ● SAME ● Executive Director • SUBEB ● NEI+ Desk TOOL 2B: SCHOOL SUPPORT OFFICERS ● SMOBSE/SMOE ● Director Planning, Research and Statistics ● LGEA ● Area Supervisors ● NEI+ System Strengthening Officer (Abuja) ● NEI+ System Strengthening Officer (States) TOOL 2C: QUALITY ASSURANCE OFFICERS ● SMOBSE/SMOE ● Director, Quality Assurance ● LGEA ● Quality Assurance Officer (QA) TOOL 2D: ACCESS AND FRAGILITY OFFICERS ● LGA ● Access and Fragility Officer TOOL 2E: READING SPECIALISTS ● LGEA ● Reading Officer TOOL 2F: BESDA ● BESDA TOOL 3: ICT/EMIS ● LGEA ● EMIS Officers ● Information Officers ● SUBEB ● ICT & EMIS Director TOOL 4: LOGISTICS 80 ● LGEA ● Store Officer ● SAME ● Store Officer ● SUBEB ● Procurement Officer ● Finance Officer TOOL 5: EDUCATORS ● LGA ● Learning Center Coordinator ● LGEA ● Headteachers TOOL 6: COMMUNITY MEMBERS ● SBMC ● CBMC ● PTA ● CBOs TOOL 7: COMMUNITY MOBILIZATION ● LGA ● Community Mobilization Officer ● OTHERS (TBD) TOOL 8: CIVIL SOCIETY ORGANIZATIONS/GRANTEES ● FOMWAN ● CSACEFA ● RAN ● ANFEA ● LGA ● Lead Program Officer ● Program Officers TOOL 9: WOMEN’S GROUPS TOOL 10: NEI+ ● NEI+ ABUJA ● COP/DCOP ● Senior Reading Specialist ● Reading Specialist (Hausa) ● Reading Specialist (English) ● Teacher Education Specialist ● NEI+ STATES ● State Team Lead ● System Strengthening Officer ● Monitoring and Evaluation Officer ● ICT Officer 81 ● Access and Fragility Specialist ● System Strengthening Specialist ● Procurement/Finance Officer ● Database Specialist ● ICT Specialist ● EMIS Specialist ● Gender Specialist ● Assessment Specialist ● Community Mobilization Specialist ● Monitoring and Evaluation Specialist ● Reading Specialists (English and Hausa) ● Procurement Officer ● Finance Officer ● Community Mobilization Officer TOOL 11: FINANCE AND BUDGETING Relevant Finance Officers, Executive Secretaries and Directors. TOOL 12: TECHNICAL INTERVIEWS WITH TEACHERS AND FACILITATORS ● NEI+ Trained Teachers AGLC Facilitators TOOL 13: FORMAL SCHOOL CLASSROOM OBSERVATION To be used in LGEA classrooms. TOOL 14: NON-FORMAL LEARNING CENTER LESSON OBSERVATION To be used in NFLC and AGLC classrooms VIRTUAL SCHOOL ‘VISITS’ PLAN AND CLASSROOM OBSERVATIONS During the 10 days of fieldwork in Bauchi and Sokoto, the evaluation plan is to interview 40 teachers from formal schools and 20 teachers or facilitators from non-formal schools or Adolescent Girls Learning Centers. Evaluators will talk to teachers from 1st, 2nd and 3rd grade classes at formal schools and facilitators for classes for ages 6-10 and 11-17 in non-formal sites. School Head Teachers will be interviewed and focus group discussions with teachers and facilitators, parents/PTA, and other local officials or key groups that support school activities will be held. Classroom observations will be held in Sokoto and Bauchi for grades 4 and 5. Up to 4 classes at each school/site will be the team’s goal. Each classroom visit will be no less than ½ hour. 82 Schools and centers will be selected using a purposive sampling approach focusing on the variables listed below and after consultation with DEV TECH COP, USAID EDU, and other relevant partners. Some of the following variables listed will inform the selection of LGEA sites and schools: ● Urban and Rural ● Formal, Non-Formal Learning Centers, Adolescent Girls Learning Center and Youth Learning Centers ● Length of time participating in NEI+ Once schools are selected, the evaluation team will work with SUBEB and LGEAs to obtain approval for scheduling interviews with the relevant stakeholders in the schools and centers and the Evaluation Team will call to set up meetings. CSOs will be contacted in order to alert them of the impending data collection activities at NFLCs. School Based Management Committee (SBMC), Center Based Management Committee (CBMC), and PTA members may be alerted by head teachers, NFLC facilitators and CSOs. 3.2. Tools The following tools will be used for data collection. A list of the stakeholders for the evaluation is provided in Annex Two, a Question Matrix in Annex Three, and a full set of the tools is provided in Annex Four: DATA COLLECTION TOOLS. KEY INFORMANT INTERVIEWS (KIIs) The interviews are designed both to obtain specific information from technical or administrative officials and to triangulate data. Therefore, they contain some questions that are common to more than one interview and also individual questions tailored to a particular role. Data obtained from the interviews will be mostly qualitative. KIIs will be held for individuals. The data will contribute to understanding the outcomes of the Activity by soliciting opinions on activities leading to the achievement of the project indicators. Every effort will be made to triangulate information by asking the same questions to multiple stakeholders. Data will be disaggregated by respondent job category. A separate tool is included for NEI+ Abuja and NEI+ States – this contains all questions and will be completed by evaluators selecting the appropriate questions for each participant depending on their duties and involvement in the project. There is also a separate tool for budgeting and finance questions which will be asked during the relevant interviews. KIIs are planned to take not more than 90 minutes and to capture relevant data without causing respondent fatigue. FOCUS GROUP DISCUSSIONS The FGDs are tailored to specific groups of stakeholders as outlined in the Data Collection Tools Matrix above. Data obtained from the groups will be mostly qualitative. If it is not possible to get stakeholders together in one place for interviews, they will be interviewed individually. This may cause a drop in the predicated numbers due to timing. The data will contribute to understanding the outcomes of the Activity 83 by soliciting opinions on activities leading to the achievement of the project indicators. Data will be disaggregated by type of group. FGDs are planned to take not more than 90 minutes and to capture relevant data without causing respondent fatigue. TECHNICAL INTERVIEWS WITH TEACHERS AND FACILITATORS Evaluators will conduct interviews with teachers and facilitators. The data collection tool has been designed based on activities and techniques from the NEI+ trainings. Interviews are to try to establish understanding, retention and use of the taught methodology. This will not produce solid quantitative data but may give a snapshot of what methods teachers and facilitators are employing in the classroom, albeit from their own perspective and judgement. For example, teachers and facilitators will be asked which methods they use, how they teach “I do, we do, you do,” if they use groupwork and pair work and if all students have textbooks. CLASSROOM OBSERVATIONS Evaluators will conduct direct classroom observations using a checklist of teacher and learner behaviors, materials and visual aids, and general environment in the classrooms to ascertain if the methodology in the NEI+ teacher training sessions are implemented in the lessons. This will produce quantitative data to triangulate with qualitative data gathered in FGDs with teachers. 3.3. Data Collection and Quality Control Process The evaluation team will ensure adherence to sampling and tool administration protocols discussed during planning, such as completing checklists of respondents, correct completion of respondent information or numbers, reading out the informed consent information and providing contact information of evaluators. Interviews will be conducted by phone or, where possible, through Google Meet. The interviews and FGDs will be conducted directly by an evaluator who speaks the same language as the respondents or a translator will be provided by DevTech. The technical interviews with teachers and facilitators will be conducted by evaluators ho are both education experts and conversant with the content of the NEI+ trainings. A dry run of the tools will be conducted by the evaluators prior to using the tools in the field. The evaluation team will call or email respondents to arrange a time for KIIs and FGDs a few days prior and follow up with a call or email the day before the scheduled meetings. Head Teachers and Center Coordinators will provide contact information for teachers and facilitators. 3.4. Ethical Review and Informed Consent This evaluation is committed to upholding high ethical standards and will ensure high standards and systems for ethical review are upheld throughout design, data collection, analysis and report writing. All tools will include an informed consent statement that explains that participation is completely voluntary and they can end the interview or focus group discussion at any time with no negative consequences to them. Chatham House rules will apply for FGDs and KIIs so the names of any particular respondent will 84 not be mentioned or recorded. The report will merely indicate the group with whom the FGD was conducted or the position held in an organization or authority. Personally, identifiable information (PII) will not be collected, and to help maintain confidentiality, respondents will be assigned a number in the notes. The interview process will use the following steps in order to ensure complete confidentiality of respondents: ● Prior to beginning interviews or focus group discussions, the informed consent statement will be read and explained by interviewer and respondent(s) will be able to ask any questions. The statement will include purpose of study and request to record interview for pure purpose of transcribing interview at a later date. Consent will then be obtained prior to recording data. As the meetings will now be virtual, verbal consent will be requested from all stakeholders. ● The data collection team will provide contact information to respondent(s) verbally or by text, prior to beginning interview. This contact information can be used in case a respondent wants follow up information or to provide complaints or concerns about the study. Data management will follow DevTech’s internal data management protocols which establish processes and systems to manage primary source data throughout its lifecycle (from data collection to archival and destruction). These processes are designed to protect respondent anonymity and confidentiality in line with USAID ADS 201 and the mandatory reference for ADS 200 “Protection of Human Subjects in Research Supported by USAID (December, 2006). Processes that support this goal include: ● Recordings will be destroyed once transcripts are completed ● All interview and focus group transcripts will be labeled according to a naming convention that does not include personally identifiable information, with each participant being assigned a number in the notes. ● All final soft copies of qualitative transcripts (typed notes) will be stored in a secured (restricted) folder on DevTech’s protected server for 5 years and subsequently will be permanently deleted; these raw qualitative data will not be transferred outside of DevTech ● All hard copies of interviews and any longhand notes will be transferred to DevTech, where they will be stored for 3 months after the evaluation is completed, and then destroyed ● All consultants and subcontractors will permanently destroy electronic notes, recordings, and other data, after transferring to DevTech and verification of data ● Only individuals employed by DevTech and that are registered with an DevTech Microsoft account will be given access to any project information, and this will be based on a “need-to￾know” basis. 3.5. Data Analysis 85 The KIIs and FDGs will be recorded and transcription will be written by the evaluators with the aid of an online transcriber, Otter.ai so as to ensure speed and accuracy. The audio recordings will be used to later clean and verify accuracy of transcripts. Quantitative data collected via the classroom observations will be analyzed using Microsoft excel to provide descriptive statistics and evidence of change in key indicators where possible. Data will be disaggregated by student age group, sex, and by school setting (urban or rural). Qualitative data from the KIIs and FGDs will be coded thematically using NVivo, a platform application for qualitative and quantitative analysis to address each evaluation question. Prior to analysis, the team will create a codebook internally to ensure relevance of the coding structure to the data and consistent code application across analysts. 3.6. Limitations ● Classroom observations were initially cancelled and replaced with a technical interview with teachers and facilitators. On review, it seems that some schools are still running and so the classroom observation protocols were re-added to the evaluation. Extra days have been added to capture this data. However, difficulty accessing schools on short notice may result in less observations being conducted than originally planned. This will be mitigated somewhat by the technical interviews with teachers which are still going ahead, in conjunction with the observations. ● The evaluation team will be working remotely and conducting interviews by phone or online. In some ways this may be more productive as any missing interviews can be drawn from different schools or centers without the evaluators having to return to locate people in person. However, it may be a challenge to schedule the interviews and to get the contact numbers of the relevant people. There may also be a problem with connectivity in remote areas. This may have an impact on the numbers finally interviewed. ● The latest EGRA data from Sokoto and Bauchi is from the 2018 EGRA conducted by Creative Associates International. There will not be an end line EGRA. This means that, although respondents can give opinions on current reading levels, this will not be backed up by quantitative data. ● Evaluations that are primarily qualitative can generate a great deal of rich, contextual data. The sheer volume of such data invites a tendency to underuse, or even not to use, all the data collected during analysis. Insufficient time to analyze qualitative findings is a potential weakness for in-depth analysis. (Additional analysis time could reduce this limitation.) Secondly, an emphasis on quantifying qualitative data often excludes “critical incidents/cases” or “outliers.” In anthropological and sociological practice, “critical incidents/cases outliers” are given full credence as evidence of nascent or emerging trends. Efforts will be made to maximize the use of rich qualitative data. ● Forms of respondent bias such as social desirability bias where respondents answer in a way that is acceptable in their community and acquiescence bias where respondents respond positively to all questions can be an issue. Questions will be worded with as little ambiguity as possible and center on neutral issues. ● Access to contact information in the field can be a challenge as there is not a local phone book listing the phone numbers for formal schools and NFLC contacts are ever changing. SUBEB and 86 LGEAs should be able to provide contacts for the formal schools and CSOs may have updated information on the NFLCs which seem to operate for just 9 months at a time. Contact in rural areas is especially challenging as mobile phone coverage is limited and contact is reliant on the prosperity of the mobile phone owner. 3.7. Logistics NEI+ will provide an introduction to key stakeholders during the inception phase. USAID will provide input regarding evaluation respondent categories after reviewing the Workplan. DevTech will provide regular updates on the progress of the evaluation, via email, telephone or in-person meeting with USAID. To ensure the evaluation team is able to make timely edits to drafts, USAID will be given five business days for review of documents. USAID review of documents will be needed for the following submissions: ● Workplan and draft data collection tools ● Draft Evaluation Report ● Final Evaluation Report Section 4: Deliverables and Workplan Timeline 4.1. Schedule of Deliverables Deliverable Submitted to Deadline 1 DevTech submits Draft Work Plan USAID/Nigeria for approval 12/14/2020 2 In-brief/planning meeting with USAID Meeting with USAID 12/21/2020 3 Submit Preliminary Findings with PPT USAID/Nigeria for comments 2/3/2021 4 Debriefing with USAID to present Preliminary Findings USAID/Nigeria 2/4/2021 6 DevTech submits first draft Evaluation Report USAID/Nigeria for comments 2/4/2021 7 DevTech submits second draft Evaluation Report USAID/Nigeria for comments 2/20/2021 8 DevTech submits final Evaluation Report USAID/Nigeria 3/2/2021 87 4.2 Workplan Timeline and Responsibilities NEI+ End line Evaluation Workplan Phase Activity Responsible Party Time Schedule Award USAID awards contract USAID 02/05/2020 Phase1: Inception Background documents review Preparation of draft Workplan Review of draft Workplan Submission of final Workplan DevTech DevTech USAID DevTech 11/26/2020 – 12/14/2020 11/26/2020 – 12/14/2020 12/14/2020 – 12/18/2020 2/26/2021 Phase 2: Data collection from Sokoto and Bauchi and analysis Planning for data collection Conducting data collection remotely Preparation of preliminary findings presentation and draft evaluation report DevTech DevTech DevTech 12/30/2020 – 1/20/2021 Phase 3: Validating and reporting. Preliminary findings presentation submitted to USAID Presentation of preliminary findings to USAID First draft Evaluation Report submitted Review of draft evaluation report Submission of revised evaluation report to USAID Review of revised evaluation report Evaluation report finalized DevTech DevTech DevTech USAID DevTech USAID DevTech/USAID 2/3/2021 2/4/2021 2/4/2021 2/5/2021 – 2/12/2021 2/20/2021 2/20/2021 – 2/25/2021 3/2/2021 88 Section 5: Management & Staffing Approach 4.3. Evaluation Team Members & Qualifications Fiona Rowand, Team Leader/Senior Evaluation Expert (International) – Fiona holds an MA in education, and an MSc in international development and has strong technical skills in monitoring and evaluation methodology, data collection methods, including qualitative and quantitative data; extensive expertise in the area of basic education and significant international experience. She has worked for 30 years in the field of education, specializing in low resource environments. She worked as a teacher and teacher trainer for 20 years before becoming involved in project management and monitoring and evaluation for initiatives improving the quality of education, including on early grade reading basic education projects in Sub Saharan Africa. Benedicta Agusiobo, Local Evaluator /Basic Education Expert – Benedicta has more than 30 years’ experience in Education, holds a PhD. /MPhil. in Curriculum Studies, Diffusion and Innovation, M Ed. Education, BSc Hon& Education; certifications in Basic & Advanced Monitoring and Evaluation. She has taught in different levels of the educational system and in formal and non-formal settings. She consulted with DevTech Systems Inc. /USAID.N projects: on the NEI+ Mid-Term Evaluation Report 2017, `Education Crisis Response` Desk Review (2017). Other consultancies included USAID /NEI project engagements on CAII; WB/FME /Pwc, NIPEP, FGN Presidency–National Bureau of Statistics; International Consulting Expertise (ICE) EU project. She has strong skills in research and development interventions, qualitative and quantitative research, technical, analytical and reporting skills, integration of diverse data from a wide range of sources, learning assessments, EGRA assessments, designing interventions, strategies, approaches that have improved quality of learning outcomes in numeracy, literacy at the basic education level and reached marginalized beneficiaries, gender, women and girl child education. She has several publications including technical reports on basic/teacher education, policies, evaluations. She is a Hubert H. Humphrey Fellow of the US State Department of State. Mohammed Ibrahim, Local Evaluator/ Reading Expert (Local): Mohammed holds a PhD degree in Language Education and has been involved in teacher training and professional development for over 30 years. His special areas of interest include reading, teaching methodology and use of Information and Communication Technology (ICT) in Education. He has partnered with various organizations in Nigeria including National Teachers’ Institute and State Universal Education Boards (SUBEB) and state governments in in-service training and professional development of teachers at primary, secondary and tertiary levels of education. He has experience in data gathering field work for monitoring and evaluation of UNICEF, USAID and UKAID (FHI 360) projects. He currently lectures in the Department of Curriculum Studies and Educational Technology, Usmanu Danfodiyo University, Sokoto Ganya Adamu, Local Evaluator/ Education Sector - Adamu holds a Doctorate degree in Christian Education. He has worked for almost 20 years in the field of education as a teacher trainer with experience in primary, secondary and tertiary institutions. and is a national master trainer for ECCDE Teachers across the country. He has field experience in monitoring and mentoring, and monitoring and evaluation. He is currently a chief Lecturer and head of department at ECCE and has recently assumed the office of the Pioneer Head Teacher at Adamu Augie College of Education Demonstration (Nursery and Primary) School Argungu, Kebbi State, Nigeria. 89 Arowolo Ayoola Abimbola, Local Data Analyst Expert - Arowolo Ayoola has a Ph. D in Industrial Technical Education and has over 7 years’ experience consulting in various data science and research related roles. He is highly proficient in data analytics (qualitative, quantitative and geospatial methods) with extensive background in research, monitoring and evaluation and machine learning. Comfort Kaliyad Boman, Local Evaluation Assistant – Comfort has an MSc. in Economics and has over the years developing technical skills in monitoring and evaluation, data collection including qualitative and quantitative data collection methods. She has experience consulting with organizations where she has coordinated and supported the implementation of different evaluations. She has worked for over 10 years in the health sector and has specialized in data management, monitoring and evaluation, research, capacity building and health. Hadiza Shettima, Local Evaluation Assistant – Hadiza has an MPA in policy analysis, and relevant technical skills in monitoring and evaluation methodology, data collection methods, including qualitative and quantitative data; good knowledge and understanding of the Nigerian education system and hands-on experience in two private Nigerian school environments as an Administrator/ Administrative consultant. She has worked for 3 years in the field of monitoring and evaluation; she has been part of the mid line evaluation for the NEI plus EGR activity and has participated in two Field monitoring activities for the NEI plus EGR activity. 90 ANNEX ONE: DOCUMENTS FOR DESK REVIEW ● USAID Policy Briefs - Improving the Teaching of Early Grade Reading & the Transition to English (November 2017) - Testing for Improved Reading Outcomes (August 2018) - Appropriate and Relevant Texts for Improved Reading Outcomes (August 2018) - Providing Enough Instructional Time for Children to Learn to Read (November 2017) - Learning to Read in Hausa and the Transition to English (November 2017) ● USAID Country Development Cooperation Strategy (CDCS) (August 11, 2020 – August 10, 2025) ● USAID Northern Education Initiative Plus NEI Plus Midline Institutional Capacity Assessment (ICA) of Bauchi State Education Agencies and Local Government Education Authorities (LGEAs) (December 2018) ● USAID Monitoring and Evaluation Summary Report of the Nigerian Certificate of Education Pre￾Service Pilot Course – Introduction to teaching reading P1-P3 (January 2018) ● NEI+ Quarterly and Annual reports (April 2016 – June 2020) ● NEI+ Early Grade Reading Assessment Midline Report (October 2018) ● NEI+ BAUCHI CECA/Gender Analysis (June 2016) ● NEI+ Bauchi and Sokoto CECA/ Gender Analysis (January 2017) ● NEI+ SOKOTO CECA/ Gender Analysis (June 2016) ● NEI+ Northern Education Initiative Plus Report of Institutional Capacity Assessment of Basic Education Agencies (MoE, SUBEB, BASAME & LGEA) in Bauchi State (August 2017) ● NEI+ Northern Education Initiative Plus Report of Institutional Capacity Assessment of Basic Education Agencies (MoE, SUBEB, BASAME & LGEA) in Sokoto State (August 2017) ● NEI+ Midline Institutional Capacity Assessment (ICA) of Sokoto State Education Agencies and Local Government Education Authorities (LGEAs) (December 2018) ● NEI+ Baseline Assessment of Pre-Service Teacher Education Programs and Teacher Education Institutions in Bauchi and Sokoto States, Nigeria (September 2017) ● NEI+ Mainstreaming Pupils from Non-Formal Learning Centers into The Formal Education System (June 2020) ● NEI+ Early Grade Reading Assessment Midline Report (October 2018) ● State Education Fact Sheets ANNEX TWO: LIST OF STAKEHOLDERS Level Stakeholder National ● Federal Ministry of Education (FME) ● Nigerian Education Research and Development (NERDC) ● Universal Basic Education Committee (UBEC) ● Better Education Service Delivery for All (BESDA) ● National Commission for Colleges of Education (NCCE) State ● State Universal Basic Education Board (SUBEB) ● State Agency for Mass Education (SAME) ● State Technical Working Group (STWG) ● State Ministry of Basic and Secondary Education (SMOBSE/SOME) ● NEI+ Local ● Local Government Education Authority (LGEA) ● Local Government Area (LGA) 91 ● Civil Society Organizations (CSOs) ● Grantees ● Headteachers ● Teachers and Facilitators ● Women’s Groups ● Community members 92 ANNEX THREE: DATA COLLECTION QUESTIONS MATRIX KII AND FDG DATA COLLECTION QUESTIONS MATRIX Questions are divided by Evaluation Question + one for general questions that may fit in anywhere. Tools are divided by Stakeholder category. Questions are triangulated except for specific technical questions. 1: HIGH LEVEL/POLICY MAKERS 2A: DESK OFFICERS 2B: SCHOOL SUPPORT OFFICERS 2C: QUALITY ASSURANCE 2D: ACCESS & FRAGILITY 2E: READING SPECIALISTS 2F: BESDA 3: ICT/EMIS 4: LOGISTICS 5: EDUCATORS ME 6: COMMUNITY MBERS/PARENTS 7: COMMUNITY MOBILIZATION 8: CSOs/GRANTEES 9: WOMENS GROUPS GENERAL 0 What was the role of your organization/your role in implementing NEI+ activities? What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? What do you know about the early grade reading activity carried out by NEI+ funded by USAID? Do you have anything else you would like to add or ask? Are you a parent or guardian of a child/children in this school? If yes, how many children do you have in this school? Do you have children enrolled in Primary 1 to Primary 3 in this school? EVALUATION QUESTION 1 Describe what you think were the benefits and challenges of NEI+ teacher training? What was your assessment of the learning outcomes of the learners? Do you have any evidence/records to show? Do you have any suggestions for what could have been better in the trainings relating to Hausa instruction? 93 How did NEI+ plan for dialectical variations within target schools? What were the successes and challenges? What steps were taken in mitigating the dialectical inequities between the two states? To what extent have international evidence -based strategies, techniques and best practices for improving reading outcomes been applied? What impact has Hausa as language of instruction had on pupils’ ability to read? Were there any challenges accessing TLMs and what impact did that have on EGR instruction? What aspects of the TLMs distributed by NEI+ impressed you most? What aspects of the TLMs did you find challenging to implement and how did you overcome those challenges? What effect, if any, has the local dialect on the understanding of TLMs? Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? What factors could be responsible for poor performance of pupils in FLCs and NFLCs? What factors make it difficult for classroom teachers/ facilitators to apply EGR skills in lesson delivery? To what extent did the NEI+ teachers/NFLC facilitators training curriculum address the capacity needs of the teachers/facilitators? How do you think your community members/ other parents are perceiving the impact of the NEI+ intervention? Was your child provided with books (Mukaranta and Lissafi) as a personal copy or does your child have to share a book with other children? From your perception, do you think that the NEI+ intervention has positively impacted on the learning gains of your child/ward? Kindly expatiate on your reasons. What gains in learning have you observed from your child (ren) as a result of the NEI+ intervention? 94 If the NEI+ intervention is to continue, or if another similar intervention were to be designed for this community, what advice can you give for the improvement of such intervention. Does BESDA support acquisition of NEI+ teaching and learning material in participating states? Explain whether you have enjoyed such the support your state? What strategies are there within BESDA activity that would improve on capacity of teachers and other education players to improve on quality of education? EVALUATION QUESTION 2 Did the NEI+ project support you with mentoring and coaching? Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? Do you think the project activities had any negative effects? Could you describe them? What were the challenges you faced as an implementer of NEI+? Explain the steps you took to ensure policies and procedures were adhered to. How are NEI+ strategies adapted to local contexts and what improvements could be made? What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? Do you feel that the core activities of NEI+ are well covered by the results areas of BESDA? EVALUATION QUESTION 3 Which factors affecting access to quality education for OOSC did NEI+ address and how successful do you think they were? What strategies exist within BESDA frame work to provide access to quality education for Out -of-school children OOSC? 95 Which NEI+ activities affecting access to quality education for OOSC were you involved in and how successful do you think they were? What is your assessment of BESDA progress so far in addressing the problems of OOSC in the state? What figures support your assessment? Do you think the results of the activities for OOSC are visible in the community? Have people been positively (or negatively) influenced by the initiative? Please give examples. What more do you think could be done to help OOSC get a quality education and stay in school? EVALUATION QUESTION 4 What was the amount allocated by the government for NEI+ activities? What amount of this budgeted funding has been released? What percentage of the funding is that? What were the major activities or procured items that the funding was spent on? Do you have any documentation you could give/send us to show these figures? How do you plan to sustain the funding provided by USAID now that the project is finished? Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? Please explain the systems NEI+ put in place to strengthen education management systems. Were they successful and do you have any suggestions for improvements? Please explain the systems NEI+ put in place to strengthen financial systems. Were they successful and do you have any suggestions for improvement? Please explain the systems NEI+ put in place to strengthen ICT/EMIS systems. Were they successful and do you have any suggestions for improvement? Please explain the systems NEI+ put in place to strengthen quality assurance systems. Were they successful and do you have any suggestions for improvement? 96 Did you have any challenges with the distribution of TLMs? What were they and how did NEI+ help you deal with this? What factors limited your progress in implementing activities? Which systems could be put in place to make the process more robust? How much fund has BESDA ear marked for your State? What are the conditions for accessing the funds? What is your assessment of BESDA progress so far in addressing issue of system strengthening in the state? What figures support your assessment? EVALUATION QUESTION 5 Has there been an increase or decrease in safety in your school/center due to NEI+ activities addressing violence? What has the change been? What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? What were the measures taken to overcome gender violence at school, community and SBMC levels? Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? What future activities would you like to see to address violence in the community? How do you think women and girls could be further empowered in your community? 97 ANNEX FOUR: DATA COLLECTION TOOLS TOOL 1: HIGH LEVEL/POLICY MAKERS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # COMMENTS/NOTES a) What was the role of your organization/your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) What was the amount allocated by the government for NEI+ activities? 4 98 d) What amount of this budgeted funding has been released? What percentage of the funding is that? 4 e) What were the major activities or procured items that the funding was spent on? 4 f) Do you have any documentation you could give/send us to show these figures? 4 g) How do you plan to sustain the funding provided by USAID now that the project is finished? 4 h) What were the challenges you faced as an implementer of NEI+? 2 i) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? 2 j) To what extent have international evidence - based strategies, techniques and best practices for improving reading outcomes been applied? 1 k) Which factors affecting access to quality education for OOSC did NEI+ address and how successful do you think they were? 3 l) What more do you think could be done to help OOSC get a quality education and stay in school? 3 m) What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? 5 n) Explain the steps you took to ensure policies and procedures were adhered to. 2 o) Please explain the systems NEI+ put in place to strengthen financial systems. Were they successful and do you have any suggestions for improvement? 4 p) Please explain the systems NEI+ put in place to strengthen education management 4 99 systems. Were they successful and do you have any suggestions for improvements? q) Please explain the systems NEI+ put in place to strengthen quality assurance systems. Were they successful and do you have any suggestions for improvement? 4 r) Please explain the systems NEI+ put in place to strengthen ICT/EMIS systems. Were they successful and do you have any suggestions for improvement? 4 s) Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 t) Do you think the project activities had any negative effects? Could you describe them? 4 u) Do you have anything else you would like to add or ask? 0 100 TOOL 2A: DESK OFFICERS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What was the role of your organization/your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 d) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? 2 101 e) Explain the steps you took to ensure policies and procedures were adhered to. 2 f) What were the challenges you faced as an implementer of NEI+? 2 g) What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? 2 h) What factors limited your progress in implementing activities? Which systems could be put in place to make the process more robust? 4 i) What more do you think could be done to help OOSC get a quality education and stay in school? 3 j) What future activities would you like to see to address violence in the community? 5 k) Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 l) Do you think the project activities had any negative effects? Could you describe them? 2 m) Do you have anything else you would like to add or ask? 0 102 TOOL 2B: SCHOOL SUPPORT OFFICERS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to-day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What was the role of your organization/your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) Did the NEI+ project support you with mentoring and coaching? 2 d) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 e) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? 2 103 QUESTIONS # NOTES/COMMENTS f) Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? 1 g) To what extent have international evidence￾based strategies, techniques and best practices for improving reading outcomes been applied? 1 h) Describe what you think were the benefits and challenges of NEI+ teacher training? 1 i) Do you have any suggestions for what could have been better in the trainings relating to Hausa instruction? 1 j) What impact has Hausa as language of instruction had on pupils’ ability to read? 1 k) Were there any challenges accessing TLMs and what impact did that have on EGR instruction? 1 l) What factors could be responsible for poor performance of pupils in FLCs and NFLCs? 1 m) What more do you think could be done to help OOSC get a quality education and stay in school? 3 n) Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? 5 o) What factors limited your progress in implementing activities? Which systems could be put in place to make the process more robust? 4 p) Please explain the systems NEI+ put in place to strengthen quality assurance systems. Were they successful and do you have any suggestions for improvement? 4 q) Explain the steps you took to ensure policies and procedures were adhered to. 2 r) Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 s) Do you think the project activities had any negative effects? Could you describe them? 2 t) Do you have anything else you would like to add or ask? 0 104 TOOL 2C: QUALITY ASSURANCE OFFICERS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to-day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What was the role of your organization/your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? 1 d) Were there any challenges accessing TLMs and what impact did that have on EGR instruction? 1 e) What impact has Hausa as language of instruction had on pupils’ ability to read? 1 105 QUESTIONS # NOTES/COMMENTS f) Describe what you think were the benefits and challenges of NEI+ teacher training? 1 g) Do you have any suggestions for what could have been better in the trainings relating to Hausa instruction? 1 h) Did the NEI+ project support you with mentoring and coaching? 2 i) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 j) What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? 2 k) Please explain the systems NEI+ put in place to strengthen quality assurance systems. Were they successful and do you have any suggestions for improvement? 4 l) Explain the steps you took to ensure policies and procedures were adhered to. 4 m) What were the challenges you faced as an implementer of NEI+? 4 n) Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 o) Do you have anything else you would like to add or ask? 0 TOOL 2D: ACCESS AND FRAGILITY Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. 106 The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What was your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) Did the NEI+ project support you with mentoring and coaching? 2 d) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 e) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? 2 f) What were the challenges you faced as an implementer of NEI+? 2 g) What factors limited your progress in implementing activities? Which systems could be put in place to make the process more robust? 4 h) Do you think the activities/systems supported by NEI+ will be sustainable now 4 107 the project has finished? What measures have you taken to sustain them? i) Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? 1 j) What impact has Hausa as language of instruction had on pupils’ ability to read? 1 k) What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? 5 l) What were the measures taken to overcome gender violence at school, community and SBMC levels? 5 m) Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? 5 n) What future activities would you like to see to address violence in the community? 5 o) Which NEI+ activities affecting access to quality education for OOSC were you involved in and how successful do you think they were? 3 p) Do you think the results of the activities for OOSC are visible in the community? Have people been positively (or negatively) influenced by the initiative? Please give examples. 3 q) What more do you think could be done to help OOSC get a quality education and stay in school? 3 r) Do you think the project activities had any negative effects? Could you describe them? 2 s) Do you have anything else you would like to add or ask? 0 108 TOOL 2E: READING SPECIALISTS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # RESPONSE a) What was the role of your organization in implementing NEI+ activities? 0 b) To what extent have international evidence￾based strategies, techniques and best practices for improving reading outcomes been applied? 1 c) How are NEI+ strategies adapted to local contexts and what improvements could be made? 2 d) What impact has Hausa as language of instruction had on pupils’ ability to read? 1 e) Describe what you think were the benefits and challenges of NEI+ teacher training? 1 109 f) Do you have any suggestions for what could have been better in the trainings relating to Hausa instruction? 1 g) Were there any challenges accessing TLMs and what impact did that have on EGR instruction? 1 h) What percentage of pupils in the school / LGEA / State would you estimate had access to TLMs? 1 i) How did NEI+ plan for dialectical variations within target schools? What were the successes and challenges? 1 j) What steps were taken in mitigating the dialectical inequities between the two states? 1 k) What factors could be responsible for poor performance of pupils in FLCs and NFLCs? 1 l) To what extent did the NEI+ teachers/NFLC facilitators training curriculum address the capacity needs of the teachers/facilitators? 1 m) What factors make it difficult for classroom teachers/ facilitators to apply EGR skills in lesson delivery? 1 n) What was your assessment of the learning outcomes of the learners? Do you have any evidence/records to show? 1 o) Did the NEI+ project support you with mentoring and coaching? 2 p) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 1 q) Do you have anything else you would like to add or ask? 0 TOOL 2f: SUBEB Date: State: Location/City/Village: 110 Interviewer/Observer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # RESPONSE a) What do you know about the early grade reading activity carried out by NEI+ funded by USAID? 0 b) Do you feel that the core activities of NEI+ are well covered by the results areas of BESDA? 2 c) Does BESDA support acquisition of NEI+ teaching and learning material in participating states? Explain whether you have enjoyed such the support your state? 1 d) What strategies exist within BESDA frame work to provide access to quality education for Out-of-school children OOSC e) What strategies are there within BESDA activity that would improve on capacity of teachers and other education players to improve on quality of education? 1 111 f) What is your assessment of BESDA progress so far in addressing the problems of OOSC in the state? What figures support your assessment? 3 g) How much fund has BESDA ear marked for your State? What are the conditions for accessing the funds? 4 h) What is your assessment of BESDA progress so far in addressing issue of system strengthening in the state? What figures support your assessment? 4 i) Do you have anything else you would like to add or ask? 0 TOOL 3: ICT/EMIS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. 112 QUESTIONS # COMMENTS/NOTES a) What was the role your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 d) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? e) What were the challenges you faced as an implementer of NEI+? 2 f) Explain the steps you took to ensure policies and procedures were adhered to. 2 g) Please explain the systems NEI+ put in place to strengthen ICT/EMIS systems. Were they successful and do you have any suggestions for improvement? 4 h) Please explain the systems NEI+ put in place to strengthen education management systems. Were they successful and do you have any suggestions for improvements? 4 i) What factors limited your progress in implementing activities? Which systems could be put in place to make the process more robust? 4 j) Did you have any challenges with the distribution of TLMs? What were they are how did NEI+ help you deal with this? 4 k) Did the NEI+ project support you with mentoring and coaching? 2 l) What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? 2 113 m) Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 n) Do you have anything else you would like to add or ask? 0 TOOL 4: LOGISTICS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # RESPONSE a) What was the role of your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is 0 114 your assessment of the improvements? How can you see/measure them? c) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 d) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? 2 e) Please explain the systems NEI+ put in place to strengthen financial systems. Were they successful and do you have any suggestions for improvement? 4 f) What was the amount allocated by the government for NEI+ activities? g) What amount of this budgeted funding has been released? What percentage of the funding is that? h) What were the major activities or procured items that the funding was spent on? i) Do you have any documentation you could give/send us to show these figures? j) Please explain the systems NEI+ put in place to strengthen education management systems. Were they successful and do you have any suggestions for improvements? 4 k) Did you have any challenges with the distribution of TLMs? What were they are how did NEI+ help you deal with this? 4 l) What factors limited your progress in implementing activities? Which systems could be put in place to make the process more robust? 4 m) Did the NEI+ project support you with mentoring and coaching? 2 n) What kind of training were you or your organization given in order to effectively 2 115 monitor the project activities? How did it help you? o) Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 p) Do you have anything else you would like to add or ask? 0 TOOL 5: EDUCATORS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # RESPONSE a) To what extent did the NEI+ teachers/NFLC facilitators training curriculum address the capacity needs of the teachers/facilitators? 1 116 b) What was your assessment of the learning outcomes of the learners? Do you have any evidence/records to show? 1 c) Describe what you think were the benefits and challenges of NEI+ teacher training? 1 d) Do you have any suggestions for what could have been better in the trainings relating to Hausa instruction? 1 e) What impact has Hausa as language of instruction had on pupils’ ability to read? 1 f) Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? 1 g) Did the NEI+ project support you with mentoring and coaching? 12 h) Were there any challenges accessing TLMs and what impact did that have on EGR instruction? 1 i) What aspects of the TLMs distributed by NEI+ impressed you most? 1 j) What aspects of the TLMs did you find challenging to implement and how did you overcome those challenges? 1 k) What effect, if any, has the local dialect on the understanding of TLMs? 1 l) What factors could be responsible for poor performance of pupils in FLCs and NFLCs? 1 m) What factors make it difficult for classroom teachers/ facilitators to apply EGR skills in lesson delivery? 1 n) Has there been an increase or decrease in safety in your school/center due to NEI+ activities addressing violence? What has the change been? 5 117 o) Which factors affecting access to quality education for OOSC did NEI+ address and how successful do you think they were? 4 p) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 q) Do you have anything else you would like to add or ask? 0 TOOL 6: COMMUNITY MEMBERS/PARENTS Date: State: Location/City/Village: Interviewer/Observer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. 118 QUESTIONS # NOTES/COMMENTS a) What was the role of your organization in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) What were the challenges you faced as an implementer of NEI+? 2 d) Explain the steps you took to ensure policies and procedures were adhered to. 2 e) What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? 2 f) How are NEI+ strategies adapted to local contexts and what improvements could be made? 2 g) Did the NEI+ project support you with mentoring and coaching? 2 h) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 i) Which factors affecting access to quality education for OOSC did NEI+ address and how successful do you think they were? 3 j) Which NEI+ activities affecting access to quality education for OOSC were you involved in and how successful do you think they were? 3 k) Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? 1 l) Do you think the results of the activities for OOSC are visible in the community? Have people been positively (or negatively) influenced by the initiative? Please give examples. 3 119 m) What more do you think could be done to help OOSC get a quality education and stay in school? 3 n) What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? 5 o) Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? 5 p) What future activities would you like to see to address violence in the community? 5 q) Are you a parent or guardian of a child/children in this school? If yes, how many children do you have in this school? 0 r) Do you have children enrolled in Primary 1 to Primary 3 in this school? 0 s) How do you think your community members/ other parents are perceiving the impact of the NEI+ intervention? 1 t) Was your child provided with books (Mukaranta and Lissafi) as a personal copy or does your child have to share a book with other children? 1 u) From your perception, do you think that the NEI+ intervention has positively impacted on the learning gains of your child/ward? Kindly expatiate on your reasons. 1 v) What are the learning gains you have observed from your child (ren) benefitting from the NEI+ intervention? 1 w) If the NEI+ intervention is to continue, or if another similar intervention were to be designed for this community, what advice can you give for the improvement of such intervention. 1 x) Do you have anything else you would like to add or ask? 0 120 TOOL 7: COMMUNITY MOBILIZATION OFFICERS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What was your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) Did the NEI+ project support you with mentoring and coaching? 2 121 d) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 e) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? 2 f) What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? 2 g) Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? 1 h) Which NEI+ activities affecting access to quality education for OOSC were you involved in and how successful do you think they were? 3 i) Do you think the results of the activities for OOSC are visible in the community? Have people been positively (or negatively) influenced by the initiative? Please give examples. 3 j) What more do you think could be done to help OOSC get a quality education and stay in school? 3 k) Do you think the activities supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 l) What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? 5 m) Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? 5 n) What future activities would you like to see to address violence in the community? 5 122 o) Do you think the project activities had any negative effects? Could you describe them? 2 p) Do you have anything else you would like to add or ask? 0 TOOL 8: CIVIL SOCIETY ORGANIZATIONS/GRANTEES Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What was the role of your organization in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 123 c) Did the NEI+ project support you with mentoring and coaching? d) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 e) What were the challenges you faced as an implementer of NEI+? 2 f) Explain the steps you took to ensure policies and procedures were adhered to. 2 g) What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? 2 h) What impact has Hausa as language of instruction had on pupils’ ability to read? 1 i) Would you say that reading ability has improved in your community due to NEI+ initiatives? Can you give examples of why/why not? 1 j) What factors could be responsible for poor performance of pupils in FLCs and NFLCs? k) Which NEI+ activities affecting access to quality education for OOSC were you involved in and how successful do you think they were? 3 l) Do you think the results of the activities for OOSC are visible in the community? Have people been positively (or negatively) influenced by the initiative? Please give examples. 3 m) What more do you think could be done to help OOSC get a quality education and stay in school? 3 n) What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? 5 124 o) Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? 5 p) What future activities would you like to see to address violence in the community? 5 q) Do you think the activities supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 r) Do you have anything else you would like to add or ask? 0 TOOL 9: WOMEN’S GROUPS Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. 125 Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What was your role in implementing NEI+ activities? 0 b) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 c) Did the NEI+ project support you with mentoring and coaching? 2 d) Do you feel that you were given enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 e) Do you feel your contributions and feedback were taken seriously by NEI+? Was the NEI+ team easily reachable and responsive to requests? 2 f) What kind of training were you or your organization given in order to effectively monitor the project activities? How did it help you? 2 g) Which NEI+ activities affecting access to quality education for OOSC were you involved in and how successful do you think they were? 3 h) Do you think the results of the activities for OOSC are visible in the community? Have people been positively (or negatively) influenced by the initiative? Please give examples. 3 i) What more do you think could be done to help OOSC get a quality education and stay in school? 3 j) Do you think the activities supported by NEI+ will be sustainable now the project has finished? What measures have you taken to sustain them? 4 126 k) What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? 5 l) What were the measures taken to overcome gender violence at school, community and SBMC levels? 5 m) Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? 5 n) What future activities would you like to see to address violence in the community? 5 o) How do you think women and girls could be further empowered in your community? 5 p) Do you think the project activities had any negative effects? Could you describe them? 2 q) Do you have anything else you would like to add or ask? 0 TOOL 10: NEI+ This tool is to be completed by interviewing multiple people at NEI+ Abuja and States. Use a different form for each person but only ask the questions you think are relevant to their position or experience. Dates: State: Interviewer/Evaluator: Duration: Position/Title Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. 127 The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS GENERAL q) What was your role in implementing NEI+ activities? 0 r) What would you say were the major achievements of the NEI+ program? What is your assessment of the improvements? How can you see/measure them? 0 TEACHING, TEACHER TRAINING AND TLMs a) Describe what you think were the benefits and challenges of NEI+ teacher training? 1 b) Do you have any evidence/records to show of the learning outcomes of the learners? 1 c) How do you think NEI+ could have improved trainings relating to Hausa instruction? 1 d) How did NEI+ plan for dialectical variations within target schools? What were the successes and challenges? 1 e) What steps were taken in mitigating the dialectical inequities between the two states? 1 f) To what extent have international evidence￾based strategies, techniques and best 1 128 practices for improving reading outcomes been applied? g) What impact has Hausa as language of instruction had on pupils’ ability to read? 1 h) Were there any challenges distributing TLMs and what impact did that have on EGR instruction? 1 i) Would you say that reading ability has improved in the communities supported by NEI+ initiatives? Can you give examples of why/why not? 1 j) What factors could be responsible for poor performance of pupils in FLCs and NFLCs? 1 k) How did the NEI+ teacher/s NFLC facilitators training curriculum address the capacity needs of the teachers/facilitators? 1 l) Did you have any challenges with the distribution of TLMs? What were they and how did NEI+ help streamline the process? 1 FIDELITY OF IMPLEMENTATION a) How did NEI+ support stakeholders with mentoring and coaching? 2 b) Do you feel that NEI+ gave stakeholders enough support on how to facilitate activities? Do you have any suggestions for improvement? 2 c) Do you think the project activities had any negative effects? Could you describe them? 2 d) What were the challenges you faced implementing activities? 2 e) Explain the steps you took to ensure policies and procedures were adhered to both within NEI+ and by the implementing partners? 2 f) What kind of training did you give organizations in to effectively monitor the project activities? How did it help them? 2 129 g) How are NEI+ strategies adapted to local contexts and what improvements could be made? 2 OOSC a) Which factors affecting access to quality education for OOSC did NEI+ address and how successful do you think they were? 3 b) Do you think the results of the activities for OOSC are visible in the community? Have people been positively (or negatively) influenced by the initiative? Please give examples. 3 c) What more do you think could be done to help OOSC get a quality education and stay in school? 3 CONFLICT SENSITIVE APPROACH a) Has there been an increase or decrease in safety in your schools/centers supported by NEI+ activities addressing violence? What has the change been? What evidence do you have? 5 b) What measures did NEI+ take to reduce risks associated with violence and conflict? How effective have these measures been? 5 c) What were the measures taken to overcome gender violence, at school, community and SBMC levels? 5 d) Are the effects of NEI+ initiatives addressing conflict visible in the communities? How was this achieved or how could it be improved? 5 e) What future activities would you like to see to address violence in the community? 5 SYSTEMS STRENGTHENING a) Do you think the activities/systems supported by NEI+ will be sustainable now the project has finished? What measures have you taken to help stakeholders sustain them? 4 130 b) Please explain the systems NEI+ put in place to strengthen education management systems. Were they successful and do you have any suggestions for improvements? 4 c) Please explain the systems NEI+ put in place to strengthen financial systems. Were they successful and do you have any suggestions for improvement? 4 d) Please explain the systems NEI+ put in place to strengthen ICT/EMIS systems. Were they successful and do you have any suggestions for improvement? 4 e) Please explain the systems NEI+ put in place to strengthen quality assurance systems. Were they successful and do you have any suggestions for improvement? 4 f) What factors limited progress in implementing activities? Which systems could be put in place to make the process more robust? 4 FINAL QUESTION a) Do you have anything else you would like to add or ask? 0 TOOL 11: FINANCE AND BUDGETING Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. 131 We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) What is your mandate in the management, allocation and release of education funding? 4 b) What role do you have in NEI+ as it pertains to improving the allocation of funding to schools? 4 c) How much of the current allocation of funds for basic education has been released for the current year? 4 d) What evidence indicates that release of funds? 4 e) As a result of your participation with NEI+ how has the system that allocates funds to schools been strengthened? 4 f) How prepared is your department/organization to sustain allocation of funds on its own? 4 g) To what extent have states and LGEAs assumed financial responsibility for activity implementation? 4 h) What has stood in their way if not yet achieved? 4 i) What could be improved to achieve educational systems financial support? 4 j) Which parts of the systems relating to finance have been strengthened more 4 132 quickly than others? What do you think is the reason for this? TOOL 12: TECHNICAL TEACHER AND FACILITATOR INTERVIEW Date: State: Location/City/Village: Interviewer/Evaluator: Duration: Organization/Group Position/Title: Introduction: My name is __________, and I’m working with DevTech Systems Inc, supporting USAID research on the NEI+ project. The entire interview will take no longer than 90 minutes. The information you provide will be used to help us learn more about NEI+ in order to measure success and inform future planning. We are taking notes to help us to collect information. We will also be recording your voice to help us accurately record your answers. The information you provide will be stored very safely and no one outside the immediate project staff will have access to it. Your answers will be treated with the utmost privacy and confidentiality and your name will not be recorded or appear in any reports. Your participation in this study does not pose any risks or benefits outside of those in your day-to￾day activities participating in your job. Thank you for participating in this interview. There is no obligation to answer the questions, and you are free to refuse any question you do not wish to answer. You have the right to withdraw your agreement to participate in this survey at any time during the interview. QUESTIONS # NOTES/COMMENTS a) Which NEI+ teacher training did you attend?1 b) What were the most valuable parts of the trainings for you? 1 133 c) In what way have you been able to use these training in your classroom? 1 d) Can you give examples of technics you learned on NEI+ training that has helped you improve your students’ reading skills? 1 e) What kind of groupings do you use in the classroom? Do you use whole class, pair or groupwork? Which do you think is most effective and why? 1 f) How do you use ‘I do, we do, you do’ in your class activities? 1 g) What does your classroom look like? Do you have posters and extra materials? How do you use them? 1 h) Does every student have a textbook? If not, how do you manage this situation? 1 i) Do you have teachers’ guides? How do you use them? 1 j) What does a reading lesson look like in your classroom? Please tell me some of the steps you take. 1 k) Which language do you teach in? Do you switch languages or translate sometimes? 1 l) How do you know when your students have learned what you are teaching? How do you evaluate them on a daily basis? 1 m) How do you plan your lessons? How do you set objectives for the day? 1 n) What gives you confidence in the classroom? Do you think NEI+ trainings helped with this? 1 o) What future training would you like to have? How would it help you? 1 p) Do you have anything else you would like to add or ask? 1 134 TOOL 13: FORMAL SCHOOL CLASSROOM OBSERVATION Date: Observer: School Name: State:  Bauchi  Sokoto Location: Urban  Rural Remote Rural  GPS: Grade: ______ Students Present: Boys _____ Girls _____ Total _____ Enrolled: Boys _____ Girls _____ Total _____ Teacher: Male  Female  Time of observation: Start: __________ End: __________ Length of lesson: __________ Language of the Lesson: Hausa  English  Switching between languages  Other language used: _______________ Mu Karata Teachers Guide  Reading Lesson Objectives Stated by Teacher: ______________________________________ Summary: To be completed after the lesson Are instructional strategies learned at NEI+ training evident? YES NO What strategies are used? Are NEI+ strategies adapted by class teachers to the local context which is familiar to students? YES NO Example: Did the teacher miss an instructional opportunity? How could the lesson be improved? 135 Is audio-visual or information communication technology use visible in the classroom? YES NO Example: TO BE COMPLETED DURING THE LESSON The following chart includes areas that will focus your attention during the observation. Tick descriptors that you observe in the classroom. You can tick more than one box for each question or no boxes. Use the comments box to add detail about the observation. THE TEACHER Area Observations ✔ Comments Lesson plan/scheme Teacher’s guide Self-made plan Plan on board No plan visible Role in lesson Lecturing Guiding Monitoring Questioning Teacher’s manner Encouraging Engaged On task Confident THE STUDENTS Area Observations ✔ Comments General engagement Enthusiastic Following instructions Having difficulty Disinterested Groups for instruction One-on-one with teacher Pairs Small groups Whole class Classroom environment Over crowded Crowded Sufficient space Roomy 136 READING INSTRUCTION Area Observations ✔ Comments Focus of lesson Orthographic awareness Phonological awareness Morphological awareness Fluency Vocabulary Before reading Review of previous lesson Teacher reads title Teacher shows pictures Students predict content After reading Discussion Reviewing predictions Asking questions Writing Applying information General Uses students’ background Problem solving I do, you do, we do EQUIPMENT/MATERIALS Area Observations ✔ Comments Instructional materials Leveled text Mu Karanta Story books Jolly Phonics Other Students’ books One copy each Shared copy No books Additional materials Games/puzzles Word cards Learning manipulatives Class library Newspapers etc. Posters 137 Words on wall Syllables on wall Other Equipment Chalkboard Whiteboard Computer Paper Exercise books Mobile phone Classroom environment Area Observations ✔ Comments Classroom space Over-crowded Crowded Sufficient space Roomy Ventilation Excellent Good Average Poor Lighting Electric Natural light Poor light Walls 4 2 1 Outside Noise level Productive Quiet Disruptive 138 TOOL 14: NON-FORMAL LEARNING CENTER LESSON OBSERVATION Date: Observer: Center Name: State:  Bauchi  Sokoto Location: Urban  Rural Remote Rural  GPS: Students Present: Boys _____ Girls _____ Total _____ Enrolled: Boys _____ Girls _____ Total _____ Age range: 6-10 ◻ 11-17 ◻ Mixed ◻ Teacher: Male  Female  Time of observation: Start: __________ End: __________ Length of lesson: __________ Language of the Lesson: Hausa  English  Switching between languages  Other language used: _______________ Lesson Objectives Stated by Teacher: ______________________________________________ Summary to be completed after the lesson: Are instructional strategies learned at NEI+ training evident? YES NO What strategies are used? Are NEI+ strategies adapted by class teachers to the local context which is familiar to students? YES NO Example: Did the teacher miss an instructional opportunity? How could the lesson be improved? Were the boys and girls sitting together? Did you notice any difference in their involvement in the class? 139 TO BE COMPLETED DURING THE LESSON The following chart includes areas that will focus your attention during the observation. Tick descriptors that you observe in the classroom. You can tick more than one box for each question or no boxes. Use the comments box to add detail about the observation. THE FACILITATOR Area Observations ✔ Comments Lesson plan/scheme Teacher’s guide Self-made plan Plan on board No plan visible Role in lesson Lecturing Guiding Monitoring Questioning Facilitator’s manner Encouraging Engaged On task Confident INSTRUCTION Activity ✔ Comments Introduces lesson by explaining what students will learn Reads aloud to students Responds to students’ questions Assists students one-to-one Facilitates group work Asks students to work in pairs Provides explanation if Learners don’t understand Demonstrates reading or writing skills Asks comprehension questions about lesson Encourages discussion Gives activities for students to practice new learning Checks homework from the previous lesson Gives homework for this lesson Translates when students do not understand Treats boys and girls equally THE STUDENTS Area Observations ✔ Comments General engagement Enthusiastic Following instructions Having difficulty Disinterested LEARNING 140 Activity ✔ Comments Writes on board Copies from board Reads aloud Answers oral questions Completes written work Choral or rote reading Reads independently Pair work Group work Discussion Role-play/skits Plays games/sings songs EQUIPMENT/MATERIALS Area Observations ✔ Comments Instructional materials Leveled text Mu Karanta Story books Jolly Phonics Other Students’ books One copy each Shared copy No books Additional materials Games/puzzles Word cards Learning manipulatives Class library Newspapers etc. Posters Words on wall Syllables on wall Other Equipment Chalkboard Whiteboard Computer Paper Exercise books Mobile phone Classroom environment Describe where the lesson is taking place. What kind of building is it? Is it outside? Do the students have chairs? Are there pictures on the walls?          !" #$$% ŝƐĐůŽƐƵƌĞŽĨŽŶĨůŝĐƚŽĨ/ŶƚĞƌĞƐƚĨŽƌh^/ǀĂůƵĂƚŝŽŶdĞĂŵDĞŵďĞƌƐ EĂŵĞ ƌDŽŚĂŵŵĞĚ/ďƌĂŚŝŵ dŝƚůĞ /ŶĚĞƉĞŶĚĞŶƚŽŶƐƵůƚĂŶƚ KƌŐĂŶŝnjĂƚŝŽŶ hƐŵĂŶƵĂŶĨŽĚŝLJŽhŶŝǀĞƌƐŝƚLJ͕^ŽŬŽƚ ǀĂůƵĂƚŝŽŶWŽƐŝƚŝŽŶ͍ dĞĂŵ>ĞĂĚĞƌ dĞĂŵŵĞŵďĞƌ ǀĂůƵĂƚŝŽŶǁĂƌĚEƵŵďĞƌ ;ĐŽŶƚƌĂĐƚŽƌ ŽƚŚĞƌŝŶƐƚƌƵŵĞŶƚͿ h^/WƌŽũĞĐƚ;ƐͿǀĂůƵĂƚĞĚ ;/ŶĐůƵĚĞ ƉƌŽũĞĐƚŶĂŵĞ;ƐͿ͕ŝŵƉůĞŵĞŶƚĞƌ ŶĂŵĞ;ƐͿĂŶĚĂǁĂƌĚŶƵŵďĞƌ;ƐͿ͕ŝĨ ĂƉƉůŝĐĂďůĞͿ /ŚĂǀĞƌĞĂůŽƌƉŽƚĞŶƚŝĂůĐŽŶĨůŝĐƚƐŽĨ ŝŶƚĞƌĞƐƚƚŽĚŝƐĐůŽƐĞ͘ zĞƐ EŽ /ĨLJĞƐĂŶƐǁĞƌĞĚĂďŽǀĞ͕/ĚŝƐĐůŽƐĞƚŚĞ ĨŽůůŽǁŝŶŐĨĂĐƚƐ͗ ZĞĂůŽƌƉŽƚĞŶƚŝĂůĐŽŶĨůŝĐƚƐŽĨŝŶƚĞƌĞƐƚŵĂLJŝŶĐůƵĚĞ͕ ďƵƚĂƌĞŶŽƚůŝŵŝƚĞĚƚŽ͗ ϭ͘ ůŽƐĞĨĂŵŝůLJŵĞŵďĞƌǁŚŽŝƐĂŶĞŵƉůŽLJĞĞŽĨƚŚĞ h^/ŽƉĞƌĂƚŝŶŐƵŶŝƚŵĂŶĂŐŝŶŐƚŚĞƉƌŽũĞĐƚ;ƐͿ ďĞŝŶŐĞǀĂůƵĂƚĞĚŽƌƚŚĞŝŵƉůĞŵĞŶƚŝŶŐ ŽƌŐĂŶŝnjĂƚŝŽŶ;ƐͿǁŚŽƐĞƉƌŽũĞĐƚ;ƐͿĂƌĞďĞŝŶŐ ĞǀĂůƵĂƚĞĚ͘ Ϯ͘ &ŝŶĂŶĐŝĂůŝŶƚĞƌĞƐƚƚŚĂƚŝƐĚŝƌĞĐƚ͕ŽƌŝƐƐŝŐŶŝĨŝĐĂŶƚ ƚŚŽƵŐŚŝŶĚŝƌĞĐƚ͕ŝŶƚŚĞŝŵƉůĞŵĞŶƚŝŶŐ ŽƌŐĂŶŝnjĂƚŝŽŶ;ƐͿǁŚŽƐĞƉƌŽũĞĐƚƐĂƌĞďĞŝŶŐĞǀĂůƵĂƚĞĚ ŽƌŝŶƚŚĞŽƵƚĐŽŵĞŽĨƚŚĞĞǀĂůƵĂƚŝŽŶ͘ ϯ͘ ƵƌƌĞŶƚŽƌƉƌĞǀŝŽƵƐĚŝƌĞĐƚŽƌƐŝŐŶŝĨŝĐĂŶƚƚŚŽƵŐŚ ŝŶĚŝƌĞĐƚĞdžƉĞƌŝĞŶĐĞǁŝƚŚƚŚĞƉƌŽũĞĐƚ;ƐͿďĞŝŶŐ ĞǀĂůƵĂƚĞĚ͕ŝŶĐůƵĚŝŶŐŝŶǀŽůǀĞŵĞŶƚŝŶƚŚĞƉƌŽũĞĐƚ ĚĞƐŝŐŶŽƌƉƌĞǀŝŽƵƐŝƚĞƌĂƚŝŽŶƐŽĨƚŚĞƉƌŽũĞĐƚ͘ ϰ͘ ƵƌƌĞŶƚŽƌƉƌĞǀŝŽƵƐǁŽƌŬĞdžƉĞƌŝĞŶĐĞŽƌƐĞĞŬŝŶŐ ĞŵƉůŽLJŵĞŶƚǁŝƚŚƚŚĞh^/ŽƉĞƌĂƚŝŶŐƵŶŝƚ ŵĂŶĂŐŝŶŐƚŚĞĞǀĂůƵĂƚŝŽŶŽƌƚŚĞŝŵƉůĞŵĞŶƚŝŶŐ ŽƌŐĂŶŝnjĂƚŝŽŶ;ƐͿǁŚŽƐĞƉƌŽũĞĐƚ;ƐͿĂƌĞďĞŝŶŐ ĞǀĂůƵĂƚĞĚ͘ ϱ͘ ƵƌƌĞŶƚŽƌƉƌĞǀŝŽƵƐǁŽƌŬĞdžƉĞƌŝĞŶĐĞǁŝƚŚĂŶ ŽƌŐĂŶŝnjĂƚŝŽŶƚŚĂƚŵĂLJďĞƐĞĞŶĂƐĂŶŝŶĚƵƐƚƌLJ ĐŽŵƉĞƚŝƚŽƌǁŝƚŚƚŚĞŝŵƉůĞŵĞŶƚŝŶŐŽƌŐĂŶŝnjĂƚŝŽŶ;ƐͿ ǁŚŽƐĞƉƌŽũĞĐƚ;ƐͿĂƌĞďĞŝŶŐĞǀĂůƵĂƚĞĚ͘ ϲ͘ WƌĞĐŽŶĐĞŝǀĞĚŝĚĞĂƐƚŽǁĂƌĚŝŶĚŝǀŝĚƵĂůƐ͕ŐƌŽƵƉƐ͕ ŽƌŐĂŶŝnjĂƚŝŽŶƐ͕ŽƌŽďũĞĐƚŝǀĞƐŽĨƚŚĞƉĂƌƚŝĐƵůĂƌ ƉƌŽũĞĐƚƐĂŶĚŽƌŐĂŶŝnjĂƚŝŽŶƐďĞŝŶŐĞǀĂůƵĂƚĞĚƚŚĂƚ ĐŽƵůĚďŝĂƐƚŚĞĞǀĂůƵĂƚŝŽŶ͘ /ĐĞƌƚŝĨLJ;ϭͿƚŚĂƚ/ŚĂǀĞĐŽŵƉůĞƚĞĚƚŚŝƐĚŝƐĐůŽƐƵƌĞĨŽƌŵĨƵůůLJĂŶĚƚŽƚŚĞďĞƐƚŽĨŵLJĂďŝůŝƚLJĂŶĚ;ϮͿƚŚĂƚ/ǁŝůůƵƉĚĂƚĞƚŚŝƐ ĚŝƐĐůŽƐƵƌĞĨŽƌŵƉƌŽŵƉƚůLJŝĨƌĞůĞǀĂŶƚĐŝƌĐƵŵƐƚĂŶĐĞƐĐŚĂŶŐĞ͘/Ĩ/ŐĂŝŶĂĐĐĞƐƐƚŽƉƌŽƉƌŝĞƚĂƌLJŝŶĨŽƌŵĂƚŝŽŶŽĨŽƚŚĞƌĐŽŵƉĂŶŝĞƐ͕ ƚŚĞŶ/ĂŐƌĞĞƚŽƉƌŽƚĞĐƚƚŚĞŝƌŝŶĨŽƌŵĂƚŝŽŶĨƌŽŵƵŶĂƵƚŚŽƌŝnjĞĚƵƐĞŽƌĚŝƐĐůŽƐƵƌĞĨŽƌĂƐůŽŶŐĂƐŝƚƌĞŵĂŝŶƐƉƌŽƉƌŝĞƚĂƌLJĂŶĚ ƌĞĨƌĂŝŶĨƌŽŵƵƐŝŶŐƚŚĞŝŶĨŽƌŵĂƚŝŽŶĨŽƌĂŶLJƉƵƌƉŽƐĞŽƚŚĞƌƚŚĂŶƚŚĂƚĨŽƌǁŚŝĐŚŝƚǁĂƐĨƵƌŶŝƐŚĞĚ͘ ^ŝŐŶĂƚƵƌĞ ĂƚĞ KĐƚŽďĞƌϭϵ͕ϮϬϮϬ 2 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Title Organization Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date +DGL]D 6KHWWLPD ■ 1(, SOXV 0LG7HUP (YDOXDWLRQ LPSOHPHQWHG E\ &UHDWLYH $VVRFLDWHV 0DUFK