This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the Monitoring, Evaluation and Learning program, DevTech Systems, Inc. NORTHERN EDUCATION INITIATIVE PLUS (NEI+): MID-TERM EVALUATION DR. JUANITA JENNIE CAMPOS FOR USAID USAID.GOV NEI+ MID-TERM EVALUATION REPORT | 1 This page intentionally left blank. USAID.GOV NEI+ MID-TERM EVALUATION REPORT | 2 ACKNOWLEDGEMENTS This document was produced by Dr. Juanita (Jennie) Campos, Team Leader; Janet K. Orr, Senior Education Expert; Dr. Benedicta C. Agusiobo, Education Expert; Hadiza Shettima, Education Expert. A Technical Committee composed of USAID/Nigeria’s Education Office (Croshelle Harris-Hussein, Dr. Janet Y. Thomas, Olawale Samuel and Sunny Fwogos); Dr. James Statman, Chief of Party, and Dr. Nura Ibrahim, Deputy Chief of Party, of the Northern Education Initiative Plus (NEI+) activity; and Traci Dixon, Lauren McCaw, Samuel Gyang, Yahuza Ahmed Getso, Dar Maxwell, Natalie Lam, and Lisette Anzoategui of DevTech provided support for the discussion and validation during the design, development, and report writing of the evaluation. The evaluation team also thanks NEI+ stakeholders at the Federal, State, LGEA and community levels for their contributions to this evaluation. We are appreciative of the numerous parents who showed up to offer their views, opinions, and recommendations. The team acknowledges the many classroom teachers and NFLC/AGLC learning facilitators who are at the closest interface with the focal point of this activity – children in schools and non-formal learning centers who are learning to read. Photography was conducted by Dr. Juanita Campos and Janet K. Orr during the October 2017 fieldwork. USAID.GOV NEI+ MID-TERM EVALUATION REPORT | 3 NORTHERN EDUCATION INITIATIVE PLUS (NEI+): MID-TERM EVALUATION Draft Report Submitted: November 27, 2017 Final Report Submitted: December 15, 2017 Contract No: AID-620-C-15-00002 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV NEI+ MID-TERM EVALUATION REPORT | 4 CONTENTS ACRONYMS 5 EXECUTIVE SUMMARY 7 INTRODUCTION 11 BACKGROUND 13 METHODOLOGY 15 FINDINGS 19 ISSUES 41 RECOMMENDATIONS 42 FUTURE DIRECTIONS 44 ANNEXES 46 ANNEX I: LIST OF DOCUMENTS REVIEWED 47 ANNEX II: EVALUATION SOW FOR NEI+ 50 ANNEX III: MID-TERM EVALUATION WORK PLAN 67 ANNEX IV: PROTOCOLS AND QUESTIONNAIRES 85 ANNEX V: NEI+ MID-TERM EVALUATION INTERVIEW PARTICIPANTS 107 ANNEX VI: DISCLOSURE/CONFLICT OF INTEREST FORMS 125 USAID.GOV NEI+ MID-TERM EVALUATION REPORT | 5 ACRONYMS AGLC Adolescent Girls Learning Center AC Area Coordinator AEO Area Education Officers BASAME Bauchi State Agency for Mass Education CAII Creative Associates International, Inc. CBMC Centre Based Management Committee CBO Community Based Organizations CECA Community Education Conflict Assessment COE College of Education COP Chief of Party CSACEFA Civil Society Action Coalition on Education for All CSO Civil Society Organizations DFID Department for International Development DPRS Director of Planning Research and Statistics ECR Education Crisis Response EDC Education Development Centre EGR Early Grade Reading EGRA Early Grade Reading Assessment EMIS Education Management Information System ES Education Secretaries F Formal Schools FOMWAN Federation of Muslim Women’s Association in Nigeria FGD Focus Group Discussion FSU Florida State University GEP Girls’ Education Project GON Government of Nigeria ICA Institutional Capacity Assessment ICT Information and Communication Technology IP Implementing Partner IR Intermediate Result LGA Local Government Authority LGEA Local Government Education Area M&E Monitoring and Evaluation MIS Management Information Systems MoE Ministry of Education MoU Memorandum of Understanding MT Master Trainer NCCE National Commission for Colleges of Education NEI Northern Education Initiative NEI+ Northern Education Initiative Plus NERDC Nigerian Educational Research and Development Council NFE Non-Formal Education NFLC Non-Formal Learning Centre NMEC National Mass Education Commission NGO Non-Governmental Organization OOSC Out of School Children OVC Orphans and Vulnerable Children USAID.GOV NEI+ MID-TERM EVALUATION REPORT | 6 PMP Performance Monitoring Plan PTA Parent Teachers Association PY 1 Project Year 1 PY 2 Project Year 2 QA Quality Assurance RAN Reading Association of Nigeria RANA Reading and Numeracy Activity RARA Reading and Access Research Activity REWRS Reading Early Warning Response System RF Results Framework R-CECA Rolling Community Education Conflict Assessment SAME State Agency for Mass Education SBMC School-Based Management Committees SEA State Education Account SIP School Improvement Plan SMoE State Ministry of Education SMS Short Message Service SSCOE Shehu Shagari College of Education SSO School Support Officer STL State Team Leader SWG SEA Working Group SOW Scope of Work SUBEB State Universal Basic Education Board Sub-IR Sub-Intermediate Result ToT Trainer of Teachers TWG Technical Working Group UBE Universal Basic Education UBEC Universal Basic Education Commission UNICEF United Nations International Children’s Education Fund UN United Nations USAID United States Agency for International Development YLC Youth Learning Centre 7 EXECUTIVE SUMMARY Mu Karanta! Let’s Read! is the concept and main learning material for USAID/NIGERIA’s Northern Education Initiative Plus (NEI+) project funded by the United States Agency for International Development (USAID). The five-year NEI+ project is strengthening the ability of the states of Bauchi and Sokoto to provide access to quality education—especially for girls, orphans, and children enrolled in non-traditional schools and to improve children’s reading skills for more than 2 million school-aged children and youth. DevTech Systems, Inc. (DevTech), with a four-person evaluation team, conducted the Mid-Term Evaluation presented in this Final Report. Creative Associates International, Inc. (CAII) is the lead implementing partner for the NEI+ activity. This report will be used to make mid-course changes as necessary for the duration of NEI+ and recommendations on related USAID early grade reading (EGR) activities. NEI+ is in implementation Year 3 and is expected to demonstrate significant improvement in EGR for approximately 1.6 million children in first to third grades and for more than 500,000 out-of-school children (OOSC) and youth attending some 11,000 Non-Formal Learning Centers (NFLCs), Adolescent Girls Learning Centers (AGLCs), and Youth Learning Centers (YLCs). CONTEXT Field work in Bauchi and Sokoto resulted in 223 key stakeholder interviews (177 males and 46 females) in nine Local Government Education Areas (LGEAs). Eight formal government schools were visited for observations of Primary 2 (P2) and Primary 3 (P3) classrooms, along with seven NFLCs. While the recent insurgency by Boko Haram has mainly impacted the northeastern states of Borno, Adamawa, and Yobe, it has caused communities in neighboring states to fear the possibility of terrorism and kidnapping in their communities. During a field visit to one AGLC in a remote area in Bauchi State, parents expressed their concern for the safety and security of their young daughters due to a kidnapping and rape that occurred in the area the previous year. During an interview with the Executive Secretary of The Bauchi State Agency for Mass Education (BASAME), the secretary stressed that instability in Northern Nigeria has presented the education/learning community with new challenges. These are: (1) the population of internally displaced persons (IDPs) and out of school children (OOSC) is growing; (2) 50% of the general population in Bauchi continue to be traumatized by the insurgency; and (3) a significant number of children aged six to nine are believed to be traumatized. BASAME anticipates that NFLCs for the general population, and specifically for nomads, IDPs, OOSCs, must be serviced over the next 20 years. While daily life appears somewhat stable in Bauchi and more so in Sokoto, community members in rural areas in both states remain fearful and unsure of the future. Although Bauchi and Sokoto have not been as severely affected by the violence as their neighboring states, the fear of violence has created insecurity in communities. Evidence of children’s concern for their futures is found in a UNICEF press release, dated November 20, 2017, found on the UNICEF Nigeria website. 1 UNICEF Nigerian Country Representative, 1 November 20, 2017. UNICEF, Nigeria. “Children in Nigeria call for leaders to listen on World Children’s Day.” Abuja, Nigeria. Web. https://www.unicef.org/nigeria/media_11706.html 8 Mohamad Fall stated, “Nigerian children are most likely to worry about poor education, violence against children, and terrorism affecting their peers,” highlighting the pervasiveness of this fear across the country. FINDINGS AND CONCLUSIONS The evaluation findings and conclusions are framed around five NEI+ areas: Application of International, Evidence-Based Best Practices – NEI+ is viewed as effective in its application of evidence-based and international best practices. Evaluators conclude that NEI+ is on target in this area. CAII implements a proven and direct pedagogical intervention in EGR. In formal classrooms and NFLCs, teachers and learning facilitators are methodical in delivery of instruction, which positively impacts the system. However, the effectiveness and ultimate success of NEI+ could be hampered by issues beyond CAII’s control – for example, the late delivery of Mu Karanta! textbooks by the State Government. In week five of the new school year, when evaluators were visiting EGR classrooms, textbooks had still not been delivered. Evaluators saw pupils using P1 level text books for P2 classes. Additionally, the State Government public school feeding initiative in Bauchi has caused enrollment to mushroom in EGR and non-EGR schools. Pupils in EGR classrooms are not guaranteed they will have Mu Karanta books in hand in Year 3. More critical, is the issue of high teacher absenteeism which has been documented in CAII spot￾check reports. Recommendation – Negotiate ways to ensure government will provide Mu Karanta textbooks for late registrants. In pre-service and in-service training, include strategies to enable classroom teachers to accelerate learning in cases where the pupil’s learning is out of sync with the textbooks due to late delivery of textbooks. Conflict Sensitivity. Safety and security standards addressed in gender analysis workshops in 2016 and 2017 include 33 security and safety risks, ranging from infrastructure to the more culturally sensitive topic of gender-based violence. Rather than address all 33 standards, CAII has chosen to focus on infrastructure in formal schools. Evaluators conclude this area is minimally on target, as evaluators visited classrooms in such disrepair that dilapidated roofs hung dangerously above pupils’ heads or bats flew out from exposed rafters. However, Community Based Management Committee (CBMC) members and parents in remote rural areas remain gravely concerned about girls and adolescent females for whom the threats of rape, kidnappings, and gender-based violence are heightened. Importance must be given to the more culturally sensitive topic of gender-based violence. Recommendation – Increase support of CAII’s Access and Fragility NFLC managers to further develop safety and security measures for vulnerable boys and girls around the more culturally sensitive topics of physical and verbal abuse and gender-based violence. Research Agenda: NEI+ plans to conduct research in three areas: (1) Transition from L1 to L2, (2) Time on Task, and (3) A study to understand why NFLC learners reach reading proficiency at a faster rate than pupils in formal schools. High teacher absenteeism, typically unaccounted for by State authorities, threatens the effectiveness of EGR in some formal schools. Researching factors that contribute to speedy achievement in NFLCs can potentially generate valuable lessons that can be applied in formal classrooms. Research could be better focused on what is involved in training classroom teachers about how to adjust from a teacher-led classroom approach, to the more child-centered approach used in EGR classrooms. 9 Teachers too often rely on rote learning where pupils have learned to memorize lessons rather than decoding or reading. It was observed that some formal teachers and learning facilitators have fallen into more teacher-led instruction rather than child-centered learning. Recommendation – Time and resources devoted to research on L1 – L2 transition to English could be better channeled to a topic that does not have the same breadth and depth of global knowledge this area enjoys. There is a global body of literature based on experiences in various country contexts that should be tapped and integrated into NEI+. Research should be conducted on what is involved in effectively breaking the pattern from how teachers themselves were taught under a more traditional, memorization approach, to how teachers should teach using the more child-centered method in the NEI+ EGR. Systems Strengthening: System strengthening in Nigeria is complicated, time consuming, and fraught with political obstacles. Strengthening implies “change,” which largely depends on political will and changes in behavior and attitude at the individual level. One GoN authority remarked, “The education system is not harmonized. The greatest threat to the education system is the system itself.” Educator, Bauchi State2 Critical to change is an organizational culture of connectedness, networking, sharing, and efficient information flow. These elements can disrupt bureaucratic inertia and resistance to change. Evident to evaluators was an absence of connectedness (e.g., the right hand not knowing what the left hand is doing), a lack of transparency especially regarding release of education funds, and an expressed unwillingness to share valuable databases that had been developed through personal hard work. An institutional capacity assessment of Basic Education Agencies in Bauchi and Sokoto states was conducted August 2017, and concluded that the system is fraught with inefficiencies and weakness in practically all domains. This is the environment in which NEI+ is implemented. 1 Recommendation - It was difficult for evaluators to put a finger on the “sweet spot” that could leverage or influence political will to bring about sustainable change when NEI+ funding ends. A future direction that holds the possibility of achieving activity goals is for the implementing partner to tap its best resources and knowledge gained during the life of the NEI+ activity. CAII can identify and network with organizational entities and individuals who have a proclivity to change, who aspire to instill a high level of integrity in all areas for which they are responsible, and who are ready to take action to thwart corrupt practices, even if only in small ways. Resources should be channeled to areas that have proven to positively impact the ultimate goal – children learning to read. This includes focusing on the micro-level, not just the macro￾level of Nigeria’s education system. Information and Communication Technology (ICT). ICT training has been extensive and, to some extent, effective. However, practical factors beyond CAII’s control inhibit successful application within the Education Management and Information System (EMIS). The lack of reliable and secure infrastructure; 2 USAID/NIGERIA. Northern Education Initiative Plus, Report of Institutional Capacity Assessment of Basic Education Agencies (MoE, SUBEB, BASAME & LGEA) in Bauchi and Sokoto State, August 2017. 10 unreliable or non-existent internet connectivity in many State offices; computer hardware that is inoperable; and the absence of sufficient ICT staff to repair and maintain hardware, are factors that inhibit successful application of ICT training. Recommendation – CAII’s Dashboard is a strong element that can provide a centralized source for the collection, retrieval, storage and analysis of vital education data. CAII is on target to meet ICT strengthening goals, but practicalities beyond their control can work against full achievement. It is recommended that CAII assess frequency of use, and prioritize most frequent users to determine which entities tap into Dashboard and use it productively. CAII should assess the return on investment of Dashboard and ensure that stakeholders who are in the most need of obtaining education statistics are, indeed, accessing data effectively. Also, they should determine if tablets and survey-to-go software are used correctly, and if these are being used to their full capacity, specifically by School Support Officers (SSO). Future Directions Weaknesses within the education system that are beyond CAII’s control may hamper achieving all activity goals. The evaluation team sees that CAII is moving in a direction that will allow it to be effective if it remains true to the philosophical and methodological principles of EGR. CAII can tap that “sweet spot” of sustainable change by focusing on and supporting entities or individuals who have an interest in institutional change, or are willing to change behaviors and attitudes at the individual level. Its greatest contribution can be realized by prioritizing the thousands of children who excitedly pick up Mu Karanta! and are eagerly learning to read. Mu Karanta! Let’s Read! 11 INTRODUCTION PURPOSE AND OBJECTIVES The purpose of this activity was to conduct a mid-term evaluation of the Northern Education Initiative Plus (NEI+), also referred to by stakeholders as The Initiative. NEI+ is a five-year activity funded by USAID/Nigeria. The period of performance for NEI+ is October 26, 2015 to October 25, 2020. The purpose of this mid-term evaluation is twofold: • Assess the extent to which the activity is on track to meeting its key objectives; and, • Identify promising practices, unmet needs, or unintended consequences from implementation of the activity. The assumption underpinning NEI+ is as follows: if state and local education actors can make education more child-friendly, relevant, flexible, and evidence-based, then greater numbers of children will have access to instruction that improves their reading and life skills. The objective of the evaluation was to respond to the following five key evaluation questions: 1. To what extent have international, evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? • How well are these strategies adapted to local contexts? What improvements could be made? 2. To what extent has the Initiative integrated a conflict-sensitive approach to education? • What measures has the Initiative taken to reduce risks associated with violence and conflict? How effective have these measures been? • Are the needs of underserved groups been met? If so, how well? What could be done better? 3. To what extent have systems strengthening outcomes and milestones been achieved? • To what degree have states and LGEAs assumed financial responsibility for activity implementation? How has the current economic climate affected release and expenditure of budgeted funds? • What could be done differently to better achieve educational systems strengthening and increased state and LGEA financial support? 4. To what extent has the NEI+ research agenda been formulated to address key challenges to effective EGR and access? How will the research results support project implementation and results? 5. In what ways does NEI+ utilize ICT to support project implementation, administration, monitoring and evaluation (M&E)? AUDIENCE The audience of the mid-term evaluation report is the USAID/Nigeria Mission, specifically the Education Team, the implementing partner, and key officials from the federal, state, and local government level. 12 Findings presented in this evaluation report provide ways to scale up best practices so the activity’s strengths can be optimized going forward. The outcome of the mid-term evaluation is designed to serve as a learning platform for all key stakeholders. 13 BACKGROUND Nigeria currently has the highest number of out-of-school children (OOSC), although data on the exact number of OOSC varies across development agencies. The World Bank reports that Nigeria has 12 million OOSC, UNESCO reports 8.7 million, and UNICEF reports 10.5 million. This reporting variance indicates a fundamental weakness – data gaps and inaccuracies that inhibit strategic planning for future school generations and limit the ability for stakeholders to determine where scarce resources should be channeled. Despite the differing statistics, the number of Nigeria’s OOSC is worrisome. The NEI+ project has built on previous education sector investments by USAID in Nigeria that have sought to assure equitable access as well as improved quality and efficiency of learning. USAID’s Literacy Enhancement Assistance Project (LEAP), 2001-2004, awakened community demand for better education in Nigeria’s primary schools by training and working with Parent-Teacher Associations (PTA). The Nigeria Reading and Research Access Activity (RARA) was a task order in 2014-2015 designed to carry out research in the areas of early grade reading and access to education in Nigeria. Built on these previous endeavors, the first NEI activity implemented by Creative Associates International, Inc. (2009-2014) had analogous goals as that of the current NEI+ activity: • Objective 1: Strengthened state and local government capacity to deliver basic education services; and • Objective 2: Increased access of orphans and vulnerable children (OVC) to basic education and other services. NEI+ PURPOSE AND PROGRAM DESCRIPTION The current NEI+ activity focuses on improved reading outcomes and increased access to learning, by strengthening both the administrative and financial management functions of the state entities that oversee and provide education. USAID, through NEI+, is concerned with the capacity of governmental and non￾governmental systems to manage themselves, both administratively and financially. NEI+ is designed to create sustainable education policies, strengthen planning, management and assessment systems, and build capacity in two states, Bauchi and Sokoto, to deliver and support education systems, with the end goal of increasing access to education and to improve learners’ reading outcomes. The five-year activity will strengthen the states’ ability to provide quality education—especially for girls, orphans, and children enrolled in nontraditional schools—and improve children’s reading skills in Bauchi and Sokoto states. Partnering with CAII are three U.S.-based international organizations—Education Development Center (EDC), Florida State University (FSU), Overseas Strategic Consulting (OSC)—and four local organizations—Value Minds, Association for Education Development Options (AEDO), Civil Society Action Coalition on Education for All (CSACEFA), and the Federation of Muslim Women’s Associations in Nigeria (FOMWAN). The activity works in LGEAs in each target state, with half (10) of the LGEAs in each of the two states supported directly by NEI+ and the other half supported by the state. Criteria for the selection of current target LGEAs include gross enrollment ratio (GER), net enrolment ratio (NER), literacy rate, number of OOSC, number of non-formal education learning centers, percentage of qualified teachers in the state 14 disaggregated by gender and local government; and the number of facilitators in the non-formal education centers. CAII and its partners focus on building programmatic ownership among federal, state and LGEAs, as well as increase their commitment to quality early grade reading instruction and increased access. The NEI+ activity aims to strengthen LGEA’s ability to better train and manage teachers, along with improving their capability to plan, budget and deliver on professional development, ensure school governance, mobilize community support for learning and the timely distribution of materials. 15 METHODOLOGY The evaluation team was comprised of two international team members, including the Team Leader/Senior Evaluation Expert and an Education Sector Expert and two local education experts, in addition to support provided by two full-time DevTech staff based in Nigeria, who arranged visits, coordinated logistics, and conducted interviews and class observations during fieldwork. The team read and reviewed NEI+ program documents and gathered additional references once in Abuja. A list of key stakeholders who could provide relevant information that addressed specific key evaluation questions was developed. Protocols were developed for (1) federal authorities, implementing partners and collaborators in Abuja, (2) State education authorities in Bauchi and Sokoto, (3) Local Government Education Area (LGEA) authorities including SSOs, (4) formal school head teachers and class teachers from grades two and three, and SBMC members, (5) non-formal learning facilitators, CBMC members and area coordinators and (6) CSOs, village elders, religious leaders, and parents. After conducting structured interviews at the federal level, the evaluation team divided into two sub￾teams for fieldwork in Bauchi and Sokoto. A mixed-methods approach was used to collect data in Abuja and the two states. A classroom observation protocol collected quantitative and qualitative data, whereas structured interviews gathered qualitative data using the protocols in Annex III. All protocols were submitted to USAID for review prior to fieldwork. Standard data collection tools, a team inter-rater reliability and data collection protocols were developed to ensure consistency and data integrity as data were being collected in the two field sites simultaneously. KEY PARTNERS AND ROLES The table below lists key partner institutions and their roles. These stakeholders were instrumental key informants. In addition, other key stakeholders and partners include USAID Nigeria Education Office, the implementing partner CAII, the state governments of Bauchi and Sokoto, LGA religious and community leaders, village elders, SSOs, CSOs, SMBCs, CBMCs, and parents. At the first team meeting in country, it was determined which evaluation questions would best be addressed by the partners and stakeholders noted in the table following: TABLE 1. PARTNER INSTITUTIONS AND ROLES STAKEHOLDER ROLE & RESPONSIBILITY Local Government Education Area (LGEA) Implements all programming in both reading and access. Collaborates with NEI+ activity personnel and CSOs to achieve improvements in reading and access. Formal Schools Head Teacher is the school administrator. Class Teacher delivers instruction. School Based Management Committee (SBMC) serves as liaison between the community/parents and school. 16 ANALYSIS Following data collection, three forms of analysis were used. The first and most straightforward was a desk review of background documents, complemented by updated documents gathered once in Nigeria. Baseline observations and data collected by CAII in 2015, as well as reported actions in quarterly reports, guided the development of the instruments. Key documents were read pre- and post-field work to both inform and validate field work findings. See ANNEX: References. The Dedoose qualitative software program was used to code formal school and AGLC/NFLC interview transcriptions and comments based on the evaluation questions. Classroom observations were analyzed using a simple Excel spreadsheet to tally frequency and to identify trends and themes. All other transcriptions were read, re-read, and themes were identified by hand using colored markers. Data gathered from state authorities, collaborators, and other Federal education authorities were reviewed to identify trends and patterns gathered using semi-structured protocols with individual interviewees and Adolescent Girls Learning Center (AGLC) Non-Formal Learning Center (NFLC) Grantees are Civil Society Organizations that coordinate a number of centers. Facilitator delivers instruction. State Universal Basic Education Boards (SUBEB) Coordinates the development of statewide policies, standards, frameworks, and monitoring tools for both reading and access. Provides financing and logistical support to LGEAs for materials procurement, teacher training, monitoring, and testing. State Ministry of Education (SMOE) Approves statewide policies, standards, frameworks, and monitoring tools for reading and access. Leads the development and approval of appropriate materials for formal schools and non-formal learning centers. Nigerian Educational Research and Development Council (NERDC) Ensures conformity of teaching and learning materials with national primary school curriculum. Universal Basic Education Commission (UBEC) Ensures the allocation of funds for basic education. Acquisition and distribution of textbooks. National Council for Colleges of Education (NCCE) Develops and approves pre-service coursework designed to prepare new primary teachers. Colleges of Education (CoE) Pre-service training and implementer of recently developed pre￾service reading instructional program. Federal Ministry of Education (FME) Endorses decentralized efforts to improve reading and access. Incorporates best practices into national-level plans. Approves national policies on education. Department for International Development (DFID) and UNICEF. Development partners and donors with programs in Northern Nigeria Tangential program implementers and collaborators. 17 focus group discussions. Quantitative and qualitative data are presented in this report both graphically and in narrative form. Collectively, the evaluation report synthesizes both quantitative and qualitative data drawn from the variety of data collection methods used. SITE SELECTION In preparation for the five days of field visits, the evaluation team established site selection criteria to identify formal schools and NFLC/AGLCs. A key consideration was a variety of school settings: rural and urban schools. The Implementing Partner supplied the team with a list of all schools in which NEI+ is being implemented in the two states. School sites where EGRA had been administered in 2016 were selected as the potential list of sites to be included in this evaluation. From that list, an Excel random selection was conducted to reduce the list to 40 schools in each state. An additional Excel random selection reduced the sample to four formal urban schools, four formal rural schools, four urban NFLC/AGLCs, and four rural NFLC/AGLCs, in each state, with an additional eight site alternates. (Additional site details contained in the electronic file annex.) CONSTRAINTS AND GAPS The databases available to the evaluation team often had out-of-date contact details due to staff changes and mobile phone changes, which made arranging site visits and interviews a challenge. In addition, NFLCs/AGLCs usually operate on Thursdays and weekends and only for a nine-month period. These constraints challenged the team to develop a fieldwork “action plan” in advance of arriving in Bauchi and Sokoto. Time that could have been devoted to conducting substantive interviews was compromised by logistics involved in tracking-down correct phone numbers for individuals or the location of NFLCs which tend to relocate frequently. This could have been better mitigated by NEI+ through facilitating contacts for the evaluation team and organizing meetings and interviews prior to the arrival of the team in Nigeria. USAID advised the team that visits to schools and NFLCs should be unannounced so the team could observe normal, not prepared, instructional delivery by classroom teachers and NFLC facilitators. Adhering to this request added to time spent arranging visits upon arriving in the field. On occasion, visits to NFLCs/AGLCs occurred by “accident” or because classes were held during an available time slot, which rushed the teams to utilize the surprise encounters. Table 2 below lists the actual sites visited, which varies from the sample methodology. As can be expected in busy office settings, some officials were not available due to travel, illness or previously scheduled meetings. Both teams attempted to reschedule interviews or interview deputies when available. The evaluation was allotted five days of fieldwork in each state, which the team found to be insufficient for the travel time and demanding interview schedule. With a short field visit and a very demanding interview schedule the type of trust and rapport that might have resulted in gathering pertinent and sensitive information may have been compromised. 18 TABLE 2. SITES VISITED State LGEAs Formal Schools/ Classes Observed NFLC/AGLC Urban/Rural Bauchi Bauchi & Ningi 4 schools 13 classes 2 NFLC 2 AGLC 4 Urban 5 Rural Sokoto Dambuwa, South Sokoto, Tambuwal, Wamakko, Wurno, Yabo, 4 schools 16 classes 2 NFLC 1 AGLC 1 Urban 6 Rural 19 FINDINGS In this section, findings and conclusions related to each evaluation question are presented. QUESTION 1: TO WHAT EXTENT HAVE INTERNATIONAL EVIDENCE-BASED STRATEGIES, TECHNIQUES AND BEST PRACTICES FOR IMPROVING READING OUTCOMES BEEN APPLIED? A preliminary assessment conducted in May 2016 in formal schools developed a general idea of literacy. A baseline assessment rooted in more current, evidence-based instructional strategies used in other USAID EGR activities looked at foundational reading skills in Hausa and English. The study in formal schools assessed foundational reading skills identified in well-established USAID-supported EGR programs in other countries. A clarification must be noted: “EGRA is an assessment of foundational reading skills. It is not an instructional approach, and it is not tied to (and does not seek to impose) a particular model of reading.” (Gove, Warrick, Yusuf and Bellow, 2016) Yet, the reading skills assessed were not commonly included in some previously used textbooks in Nigerian primary schools, especially phonemic awareness and the decoding process. Those textbooks focused on teaching language, especially vocabulary and sentence structure, rather than reading skills that allow the pupil to gain meaning from the page of print. The assumption is that if an individual can speak the language, he or she can automatically read what is written on a page. This assumption underlies the push in many countries to begin teaching English as a second language earlier and earlier (McCloskey, Orr, and Dolitsky, 2006) to increase the marketable skills of pupils. When NEI+ conducted the EGRA baseline in 2017 of the previous instructional strategy with its emphasis on language skills did not prepare pupils for items on reading, especially Hausa and English syllable knowledge resulting in low scores. The more current, evidence-based assessment that looks at reading more than language is a more relevant and appropriate approach to the eventual determination of improved reading outcomes. The shift from teaching Hausa and English language to teaching pupils to read in primary schools requires the implementation of internationally validated strategies, techniques and best practices to teach reading. Three foundational documents, (Snow, Burns, and Griffin, 1998; Elley, 1992; August and Shanahan, 2006) clearly delineate both reading skill knowledge and environmental characteristics that impact reading outcomes. These research-based features, USAID defined subtasks, and the program scope of work were first used to develop the classroom observation protocol for the Mid-Term Evaluation of the USAID/Jamaica Basic Education Project, February 2013. The protocol was later refined based on results, data collector input, and the Malawi EGRA program scope of work for us in The Mid-Term Evaluation of USAID/Malawi Early Grade Reading Activity (EGRA), January 2015. Further refinement of the instrument was completed for this mid-term evaluation after review of program report, Mu Karanta textbooks, and teacher guides. Additionally, Nigerian educators reviewed it thoroughly to assure the identification of relevant classroom behaviors and appropriate cultural specific vocabulary. To examine the use of identified best practices and EGR subtasks in classrooms, the evaluation team conducted 29 classroom observations focused on observable actions, such as; use of the teacher’s guide; instructional style: lecturing, guided practice, encouraging, guiding, enthusiastic, and the communication pattern of “I do, you do, we do”; teaching aides/visible print: Mu Karanta textbooks in hand or shared, word cards, posters, manipulates, blackboard, and exercise books; instructional focus: phonological, orthographic, and morphological awareness, comprehension and fluency; cognitive processing questioning 20 strategies and writing reinforcement. (See Annex: Protocols). When interviewed head teachers who emphasized that Mu Karanta (Hausa) and Let’s Read (English) teacher’s guides “made lesson delivery easy” and “brings the lesson to life.” In 76% of the 29 observed classes the teacher had a teacher’s guide in hand. To break that down, 86% of the 22 formal schools (i.e., government-run public schools) and 57% of 7 NFLC/AGLCs had teacher’s guides in hand. In Bauchi State, 3 out of the 4 NFLC/AGLC facilitators observed had a teacher’s guide in hand. The scripted lessons in the teacher’s guide provide a systematic approach to teaching reading, a very positive impact in the classroom. It is especially helpful in teaching English reading in Primary 3, as teachers expressed a lack of confidence in teaching English partially due to a need to gain a mastery of the vocabulary. One teacher requested that a glossary be added to the teacher’s guide so he could “know” the vocabulary. Use of the teacher’s guide has shifted the instructional style in the classroom, according to head teachers and SBMC members. Of the 29 classes observed 72% began with a review of the previous lesson and a song followed by guided practice. The instructional style was encouraging and enthusiastic, even in classes with close to 100 pupils. Only one out of twenty-nine classes observed had individualized instruction, in all others the instruction was provided to the whole class. During interviews, teachers said that they liked the systematic introduction of syllables and how those syllables formed words, sound families, and language patterns. One of the most popular instructional strategies stated by teachers was “I do, you do, we do.” This is a well￾recognized approach (endorsed by the Association of Supervision and Curriculum Development) for scaffolding instruction, shifting it from a teacher-centered demonstration to independent pupil practice. Greater access to print is a key ingredient to increasing reading competence. (Snow, Burns, and Griffin, 1998 and Elley, 1992). In observed classrooms in Bauchi and Sokoto, textbooks and the chalkboard are the two most available sources of print. Few teaching aides, posters or displayed pupil work was visible in classrooms. All classrooms had a board, 27 had a blackboard and 2 had white boards. The board often contained the page or chapter number of the day’s lesson. In addition, teachers used the board to reinforce instruction. Pupils were often enlisted to come to the board to point to words or syllables as the individual pupil or whole class read, “we do.” The other main source of print was the textbook, 69% of the 29 classes observed had NEI+ supplied textbooks. Some textbook sharing was visible in every class but the majority of classes had sufficient textbooks with no more than a pair of pupils sharing one book. One teacher explained that in the past she only had eight books for a class of over 30 pupils, so she thought the current situation was good. Teachers and parents encouraged pupils to take care of their books for the next generation of students. SBMC and CBMC members expressed that parents are more likely to send their children, especially girls, to school now that there are books. Parents know that having a book in hand means you are, or will become, a literate person. SBMC and CBMC members are spreading the importance of literacy in the communities. JANET K. ORR FOR USAID 21 At the schools, the most commonly voiced complaint about NEI+ is that the books are not delivered on time. Teachers explained that they are teaching P3 pupils with P2 term 3 books, or P2 pupils with P1 books. Others stated that the first and second delivery of books had only exercise books and textbooks, but no teacher’s guide. “There just haven’t been enough books.” Head teachers stated that, now that parents see books at the schools, they are more likely to enroll children, increasing the enrollees from the end of the previous term of school when the pupils were counted. A challenge in Bauchi state is the increase of school enrollment, due to the newly established feeding program and the number of IDPs in the region. NFLC/AGLCs do not seem to be facing this dilemma. BEST PRACTICES: POLICY NEI+ has focused on how to improve reading outcomes from a practical level over the past two years. Success has been demonstrated at the school and LGEA level, as it is often expressed that “pupils can read” now. Successful instructional strategies can now be codified to create policies that will sustain the program long after NEI+ support ends. It was reported in both NEI+ Second and Third Quarterly Reports for FY 2017 that six policy drafts entitled “5Ts + C” (Teaching, Time, Text, Tongue, Test, and Curriculum), have been developed as a result of policy dialogues with FMoE, NERDC, NCCE, and UBEC. NERDC is collaborating with NEI+ to develop a National Reading Framework, which is the first step in the development of standards and benchmarks to gauge pupil performance over time. Policy change takes a long time, according to NERDC staff, so in the meantime the focus is on integrating reading into the National Language Curriculum. That curriculum includes the language skills in nine critical Nigerian languages (Hausa, Ibo, Yoruba, Fulfulde, Efik, Tiv, Kanuri, Edo, and Izon). The Early Grade Reading Assessment has yet to be developed for these languages, but the groundwork has been laid to move forward. BEST PRACTICES: TEACHER TRAINING One teacher was asked during the evaluation interview, “Can you use one of those four student teachers to assist you with the reading class? I can see you need some assistance with 100+ pupils.” The teacher responded, “No, they haven’t had training to teach Mu Karanta.” That challenge is soon to be remedied, as a pilot reading methods course is in development, “An Introduction to Teaching Reading in Primary 1- 3: A Teacher Educator Guide.” NEI+ partnered with NERDC, NCCE, and the four Colleges of Education (CoE) in Sokoto and Bauchi states to develop the course, which is being taught this term (4th quarter, 2017). This is a sweeping change in methodology for the colleges, in that they presently teach methods of instruction for language learning, but not for reading. In addition, this EGR approach is more child-centered and activity-based than previous teacher training methods that are more focused on lecturing. Faculty at Kangare CoE in Bauchi and Shehu Shangari CoE in Sokoto interviewed for this evaluation expressed some challenges teaching the program, although many of the instructors also provide NEI+ in￾service training. They find it difficult to teach the jigsaw strategy and the various forms of writing, as they had only taught essay writing previously. Faculty also stated that a child-centered, activity-based approach to teaching reading in a class of 100+ pupils is difficult to replicate in a college classroom. Another aspect of the CoE/NEI+ collaboration is that twenty-four lecturers are enrolled in an FSU online course leading to a Post Graduate Diploma in Reading. This will build the capacity of the CoE to independently deliver rigorous course work in EGR to future teachers in training. Current teachers in the 22 classroom repeatedly stated that they recognize the need for additional training to assure success of EGR. HOW WELL ARE THESE STRATEGIES ADAPTED TO LOCAL CONTEXTS? There was unanimous agreement among all levels of evaluation interviewees that the strategies used in the NEI+ program are adapted to local contexts. The NERDC Deputy Director summed it up nicely, “The strategies are well adapted in the materials. Decodable readers are made to reflect the local environment. The increase in indirect reading through parents is a learning process; it is a new approach, which adapts what happens in the family to the classroom. This leads to easy comprehension thus creating pleasure in learning.” Class teachers praised the stories in the textbooks and the pictures as being easily understood by pupils. Although, one very experienced teacher did state she had “some conflict with ethical values in the book, like talking about diarrhea. In Hausa, we use a nickname so pupils are not embarrassed. It's a subject for adults, not to expose to children.” She also stated that in the Year 2 Hausa book that the dialect used was not that of Sokoto, so it was confusing for the pupils. The phonemic approach to reading works well for Hausa language speakers because it combines each consonant with a vowel. Observers regularly saw “da, de, do, du, de” or “ga, ge, go, gu, ge” on blackboards. Two syllables are then combined to form meaningful words. Both pupils and teachers are finding success with this approach to reading. In Bauchi, best practices in non-formal education (NFE) are evident in NFLCs. The state benefits from the legacy of the work of Dr. Musa Moda, an expert in non-formal education, who established Nigeria’s first Adult and Mass Education Institute at the Federal level and Bauchi’s satellite campus at what is now the College of Education at Kangare. In 100% of the NFLC sessions observed in Bauchi, facilitator guides and learner workbooks were in-hand and followed during instruction. In an interview with Dr. Moda, he explained that, “a relaxed atmosphere and spontaneous classroom dialogue related to the immediate environment engages and motivates learners.” The linkages between learning and living, including earning a living, may be one key ingredient in the achievement of learners in the non-formal setting. WHAT IMPROVEMENTS COULD BE MADE? The most important improvement that can be made is delivering textbooks on time. Teachers have gained confidence using the scripted lessons. They have not used Mu Karanta long enough to have fully incorporated the methodology into their instructional repertoire as they have many years’ experience using a lecture approach where the pupil is a passive learner. Refresher training for teachers is important to reinforce effective practices. One LGEA official shared his wife’s story: “I hate to admit it, but I married an illiterate woman. But now she is learning to read from the children. She helps at the school and enjoys reading with the children every day.” Example of phonemic exercise. JANET K. ORR FOR USAID 23 Pupils and parents have begun to equate school with having a textbook in hand. It is motivating to have a “real” book in hand, pupil self-esteem is raised, and pupils’ own vision of self as a literate person is established. Interviews with SBMCs and CBMCs of which parents are members indicated their support of literacy efforts. Teachers lack the confidence and resources to create supplemental materials. As a result, a print-rich environment was rarely evident. CoE and NEI+ in-service training could incorporate instruction on how to make and use teaching aides. The resource room at CoEs can feature a teaching aide demonstration each month for student teachers to learn how to make and use simple teaching aids using locally available materials. The role of SSO’s, employees of SUBEB, has shifted from inspecting to coaching and mentoring in the NEI+ program. They should be visiting classrooms regularly to support the program, but lack of funds for fuel and commitment to other duties have constrained field activity. There is no evidence that either teachers or SSOs in schools have conducted “cascade or step-down training.” Both groups must be trained on how to share their own training when they return to school and be provided with the resources to do it. DFID and UNICEF, who support EGR programming in other Northern Nigerian states, are beginning to focus on “funding for sustainability.” Their planning includes a “RANA Light” program for use when donor funds are depleted. NEI+ should consider this model also. QUESTION 2: TO WHAT EXTENT HAS THE INITIATIVE (NEI+) INTEGRATED A CONFLICT-SENSITIVE APPROACH TO EDUCATION? The extent to which NEI+ schools and NFLCs address conflict sensitivity varies among stakeholders. The range of concerns includes structural safety issues, availability of drinking water, road safety, corporal punishment, access to latrines, and gender-based violence. The evaluation team found that CAII does not focus on the more culturally sensitive topic of gender-based violence but is aware of the vulnerabilities of some learning environments. No evidence that gender-based violence is explicitly addressed was found in remote rural communities where the safety and security for adolescent girls is of concern to parents and young women. 24 CAII organized consultative workshops and site visits to schools and NFLCs to collaboratively develop a basis for understanding over 33 descriptors of security and safety standards in learning environments. A document resulting from those workshops provides a conceptual framework for naming safety and security issues and highlights four central domains of a “Do No Harm” approach. The domains are: (1) Physical Health, (2) Child Protection, (3) Conflict Risk Reduction, and (4) Disaster Risk Reduction. CAII’s Access and Fragility Team along with STTA consultant, Stephan Richardson, facilitated workshops with select state and LGEA stakeholders to discuss security and safety in the States today. 3 “Creative will need technical personnel to work with Nigeria’s Access & Fragility team to co-develop the training packages. Depending on the project’s schedule, it may be difficult to allocate Creative’s technical assistance in time. However, if not much thought and effort are put into ensuring all actors are adequately prepared and supported to address the safety risks at schools and NFLCs, the project runs the risk of supporting safety in discourse only” (USAID, NEI+ School/Center Safety Standards Implementation Guidance, Pg. 6, 2017). In order to reduce risks associated with violence and conflict, NEI+ developed stories in Mu Karanta textbooks that support peace, tolerance and respect. At formal schools, SBMCs and Head Teachers are the most likely to integrate a conflict-sensitive approach. Each village and town has different levels of threat and varieties of resources to cope with them. Schools in Sokoto, for instance, have crossing and gate 3 USAID/NIGERIA, NEI Plus, School/Center Safety Standards Implementation Guidance, September 30, 2016. Despite the dilapidated ceiling, pupils take their reading exercises seriously. DR.JUANITA JENNIE CAMPOS FOR USAID 25 guards near busy roads. Two formal schools in Sokoto have school buses donated by former students (Old Boys) or community members. These buses, only used by the schools pick up and deliver pupils in outlying areas to school and home again, assuring their safety. Parents are encouraged to report any incidents of corporal punishment to CBMCs and SBMCs, especially as the use of corporal punishment is punishable by law enforcement. Although it is often observed at schools that a teacher has a switch or pointer, it is said by some that it is only used as a warning. In contrast, an LGEA education secretary in Bauchi explained, “Caning is illegal but that is the only way children will learn.” Caning children at their ankles was observed at a school in Bauchi. Interviews with SSOs, Area Coordinators, SBMCs, and CBMCs revealed lingering concerns precipitated by recent talk of the resurgence of Boko Haram threats and, subsequently, the need for caution and vigilance while children are at school or on their way to and from school. A number of comments related to similar concerns follow: • “We have girl hawkers who are vulnerable on the streets and they need extra protection and watching” (BASAME Official, Bauchi). • “Our safety is OK; no security threats. But, in case, we have local vigilante groups created by the community when needed. They report threats to the police and village elder who is our leader” (LGEA member, Sokoto). • “We need more legislation on rape with more punishment as in The Girl’s Child Act” (SUBEB authority, Sokoto) • “SSOs report to SUBEB about natural disasters and lack of toilets in schools but nothing is ever done about this” (LGEA member, Bauchi) • “Snake bites, kidnappings, flooding disasters are worries to this village. A young girl died of a snake bite because the hospital is 8 hours away. In this community, where an only AGLC is located, girls are worried about sexual violence” (CMBC members, Ginji Mira, Bauchi). An absence of concern for a comprehensive and explicit approach to children’s safety and security among some MoE authorities is worrisome. Two officials explained, “The days of insurgency are a thing of our recent past, not to be relived again. We have no conflict now and we do not hold this in our minds. Boko Haram is in the past. We have no security problems here.” 26 While CAII is cognizant of the vulnerabilities surrounding schools and remote NFLCs, its resources are such that the IP is focusing on the safety and security of school structures rather than the more critical issue of gender-based violence. The term “conflict sensitivity” was understood by only one CSO area coordinator, and by CAII’s Access and Fragility Manager. Both are located in Bauchi. Channeling limited resources and time to select security and safety domains is within CAII’s purview. Additionally, explicitly stating that partners and key stakeholders are responsible for children’s safety and security is within CAII’s scope to require stakeholders to ensure security and safety for all NEI+ EGR children. ARE THE NEEDS OF UNDERSERVED GROUPS BEING MET? IF SO, HOW WELL? WHAT COULD BE DONE BETTER? Underserved populations are characterized by differences including gender, race, ethnicity, geography, and health/disability issues. Of immediate concern in these two states is the historic under-enrollment of girls in school. • Non-formal and Adolescent Girls’ Learning Centers target under-served adolescents. With the addition of facilitator training, a facilitator’s guide (“Jagorar Malanai”), and textbooks (“Mu Karanta” Literacy, “Lissaffi” Numeracy and Life Skills), an increase in enrollment and attendance is evident. • SBMCs and CBMCs members visit homes to encourage parents to send their girls to school. Married girls can attend formal school until becoming pregnant, at which time they are encouraged to attend an AGLC. • Teachers’ babies are visible at formal schools and NFLC/AGLCs both in and outside of the classroom. Other teachers assist with childcare. Young married women bring their infants to NFLCs. • One formal school in Sokoto reported that two years ago the enrollment of girls was higher than that of boys. The SBMC actively recruits pupils. • Two schools in Sokoto had school buses to transport children, especially girls, from the outskirts of town to school. • School crossing guards and community members keep an eye on children as they walk to school. Young child being given guidance by a respected village elder. DR.JUANITA JENNIE CAMPOS FOR USAID 27 • A World Bank program funded at 600 million Naira pays states or schools that enroll pupils ($100 for each boy, $125 for each girl). Pupils with disabilities are another under-served group of primary pupils. Most schools visited reported that there are one or two pupils who were either deaf or blind. Fifteen of sixteen SBMCs and CBMCs reported that children with disabilities do not attend school due to lack of special education services. Evaluators in Bauchi met one blind and one deaf pupil who attended school with the help of classmates. Services for handicapped children are only available in city centers often too far for many children to avail themselves of such services. One NFLC in Sokoto was on the grounds of an infectious disease hospital that specializes in treating lepers. They or their parents may have the disease, which may stigmatize them if attending a formal school. QUESTION 3: TO WHAT EXTENT HAVE SYSTEMS STRENGTHENING OUTCOMES AND MILESTONES BEEN ACHIEVED? To address this question, interviews were conducted with Commissioners of Education, Permanent Secretaries, Directors of Planning Research & Statistics, Heads of EMIS Units/departments State MoEs, SUBEB, SAME, LGEAs, and CSOs. Key documents were reviewed to cross-check field site observations. Interviews revealed the following: ICT – Great strides have been made in this part of the system with extensive ICT training provided by CAII. The variety of computer platforms and software packages that are used by different development donors inhibits the overall ICT system to operate efficiently. EMIS personnel interviewed stated that they are expected to manage different data systems for different donor expectations. The absence of efficient interfacing across the various systems and databases results in misuse of valuable time. Simple bits of information such as accurate and current phone numbers for SSOs, Area Coordinators, grantee managers, etc. were difficult to obtain by the evaluation team in Abuja and especially in the field. Policy – NEI+ EGR is central to the evolving National Policy on Reading stated the Deputy Director of NERDC. The present plan is to integrate EGR into the National Language Curriculum, not as a standalone curriculum. Reading is not adequately covered in the language curriculum, and there is no standalone curriculum for reading. Nigeria does not have a National Reading Framework, but examples of National Reading Framework from different countries are being reviewed. Budget and Planning – A one-day round table discussion in October 2017, was convened in Abuja during which key NEI+ stakeholders discussed basic education budget preparation, execution and performance. The discussion took place within the context of the signed Memorandum of Understanding between USAID and the Bauchi State government. Please see ANNEX: Communique – Summary of Round Table Discussion, NEI+ Stakeholders. Creative Associates International, Inc./Nigeria, October 2017. CONTRACTS WITH GRANTEES Cost reimbursement grants issued to FOMWAN AND CSACEFA and other grantees have shifted to become fixed account award, and as non-formal centers expand additional grantees are added to administer the NFLC/AGLC programs. In Sokoto, FOMWAN staff explained their program: • 72 centers in 2 LGEA in 2016; 28 • An additional 40 centers in 1 LGEA in 2017; • All funds pass through the Abuja office; and • There are 3 Abuja staff plus LGEA based staff this year who visit centers frequently. NEI+ reported the following progress in the PY2 Q3 Report: “Following a review of the cost reimbursement grants issued to FOMWAN and CSACEFA, it was determined that their financial management capacity is high risk to the extent that specific award conditions were not met and the quality of financial reporting was below standard. To improve performance and administration and reduce their financial risk, fixed amount awards have been approved for them so they are the same with all grantees engaged for PY2. Grants activities commenced effective June 2017 and grantees have eight milestones to complete within the period of performance ending April 2018. All 47 grantees have completed and reported activities implemented under the first milestone. The Initiative has reviewed activities reported and released payments for this milestone.” The Reading Association of Nigeria (RAN) is a professional organization affiliated with the International Literacy Association (formerly International Reading Association) that supports the expansion of reading outside the schools. RAN staff in Sokoto reported the following: “Project work has a time delay because the work plan has not been approved since August and now it is October. No funding without an approved work plan, we cannot carry out activities. Our contract is for 4 years (it began one year after the beginning of NEI+).” RAN staff explained that they are presently unable to get out into the field due to this contract delay. 29 TO WHAT DEGREE HAVE STATES AND LGEAS ASSUMED FINANCIAL RESPONSIBILITY FOR ACTIVITY IMPLEMENTATION? A number of obstacles that were verified during interviews with Senior State authorities, the IP, and Federal MoE officials, stand in the way of fully understanding the degree to which States have assumed financial responsibility for NEI+ implementation. These are: 1. Fiscal calendars that do not synchronize; 2. Lack of transparency at the State level of how currency moves through the system, what is actually distributed and to whom, and a general lack of accountability; 3. Ghost teachers who are government employees on paper but do not actually work– “they siphon money from the system” as one respected Senior authority commented on the issue of “disappearing money.” 1. Fiscal calendars that do not synchronize: The State Education Account (SEA) runs 1 – 2 years behind its budget projections and the government’s fiscal year must close to be certified by relevant government offices. Accurate or solid data is difficult to obtain even from the SEA. Secondly, the government does not disaggregate basic education from secondary and tertiary in its education budget. Nigeria’s government calendar, which is reflected in budget planning, is from Jan 1 – Dec 31. NEI+’s budgeted year is from October 1 – September 30. The school year is from mid-September to the following mid-September. The timing for disbursement of money budgeted for education is difficult to track given the out of sync calendars. However, there are a number of known factors that assist in understanding financial contributions made to NEI+ by States. The amount for NEI+ basic education is written in State budgets. However, this does not mean that money is easily available when needed. When funds are released it is some small percentage of what is budgeted. Released funds are supposed to be disbursed quarterly, however, released funds on paper do not effectively translate to being released to States. Communities offer great support for children in school as these parents pledge to begin laying the foundation of a new school room. DR.JUANITA JENNIE CAMPOS FOR USAID 30 CAII met with SUBEB Executive Chairman/Bauchi, November 2017. His Honorable Chairman reported that “there is a distinct item for NEI+ in the new budget. But how and when it gets released is in questions. It won’t get released in the current government or USAID year.” 2. Lack of transparency: One Senior Education Authority in Bauchi offered his view regarding lack of transparency, “Record keeping at State is designed to be obscure; it is not designed to be transparent. Budgets should be released quarterly but it typically is late, late, late and it is difficult to track down the reasons for the delays. Decisions made at the highest level, Accounting General, do not flow down to States so it is difficult to understand delays.” 3. Ghost teachers and disappearing money: Bauchi’s SUBEB Executive Commissioner explained to evaluators that both practices siphon money from the education budget and both are difficult to track, but efforts to identify the number of ghost teachers are being pursued by him in earnest. Ghost teachers are teachers whose names and relevant identification information are on the education payroll, but do not actually report for work anywhere in the system. The same concern relates to pupils whose names are on school registers but do not attend school. In the NEI+ Addendum #1 to the USAID-state government Memorandum of Understanding (MOU), a graduated contribution was committed: Year 3=30% for training and materials; Year 4=70%; Year 5=90%. NEI+ for technical resource. As of October 2017, Bauchi State Government has committed to increase Education budget allocation from about 20% to 26% in 2017. Contributions made as direct financial contributions and in-kind are detailed below: Actual financial contributions - In terms of actual financial contributions, not quantified in-kind contributions, are presented below: • In Year 1, Sokoto contributed 20 million Naira and 65 million more to scale up. • In 2016, Year 2, Sokoto made a cash contribution of 9.5 million Naira. • In 2017 Sokoto contributed 65 million Naira for teaching and learning materials to scale up NEI+ to 10 new LGAs. • Comparable financial contributions from Bauchi have not been made. 31 The most current data provided by CAII on project expenditures and contributions are provided in the tables below. The first table details the percentage of expected State contribution to support in-service training, teaching and learning materials, logistics support, and space for the duration of the Initiative project. Using in-kind contributions as an indicator of political will, the table below expresses the State’s political will. The two charts below list in-kind contributions covered by GoN for Project Year 1 and Project Year 2, as reported by CAII: Year MEL Plan Bauchi & Sokoto State, GON Expected Period of Contribution Annual Projections Training TLM Logistics Space 2015/2016 90% 10% Jan.-Sept. 2016 2016/2017 80% 20% Oct. 2016-Sept. 2017 2017/2018 70% 30% Oct. 2017-Sept. 2018 2018/2019 30% 70% Oct. 2018-Sept. 2019 2019/2020 10% 90% Oct. 2019-Sept. 2020 0 20 40 60 80 100 Yr 1 Yr 2 Yr 3 Sokoto 85 9.5 65 Bauchi 0 0 0 NGN in Millions Axis Title Yearly States Financial Contributions for NEI+ Sokoto Bauchi TABLE 3. IN-KIND CONTRIBUTIONS 32 In Bauchi, the counterpart funding expected from the state government is estimated at N65,675,967.36. The actual expenditure for the period was N27,173,117, which is 41% of the expected contribution a rise in performance from PY1 of 26% increase. This indicates additional strategies and mobilization from the state government in supporting teacher training and material distribution for all school types. Sokoto spent N49,146,000 from an expected contribution of N77,732,927.76, this shows a 63% level of commitment to the teachers training and material supplies within the state. In comparison to PY1, the government though increased spending still did not meet the level of commitment expected for the period. Title Background Cost Assumption for Bauchi State $344,379.82 $3,846,456.75 $4,190,836.57 1,173,434,239.60 117,343,423.96 % Contribution 17,736,117.00 117,343,423.96 15.11470895 15% Cost Assumption for Sokoto State $341,663.35 $309,941.84 $651,605.19 182,449,453.20 18,244,945.32 % Contribution 41,599,000.00 18,244,945.32 228.0028757 228% Training of Learning Facilitators, materials development and distribution in Sokoto State Indicator 26: Percentage of costs of in-service training and materials distribution covered by GON In the PY 1, expected USAID investment on training and TLM development and distribution is total sums of $4,842,441.76 for the supported states. 10% counterpart commitment of these amount will be the states government obligation to support cost of annual teacher training and materials distribution for all school types assumed by the Initiative States and LGEAs contributions. Training of Learning Facilitators, materials development and distribution in Bauchi State Reading Instruction – Training and material development, Printing of material and distribution in Bauchi State $1=N280 Grand Total in Naira 10% Expected State Counterpart Contribution Cost Share contribution 100% Expected Government Commitment % of costs of in-service training and materials distribution covered by Bauchi State Government % of costs of in-service training and materials distribution covered by Sokoto State Government Reading Instruction – Training and material development, Printing of material and distribution in Sokoto State $1=N280 Grand Total in Naira 10% Expected State Counterpart Contribution Cost Share contribution 100% Expected Government Commitment TABLE 4. COSTS COVERED BY THE GON 33 MILESTONES AND OUTCOMES ACCOMPLISHED The table below details milestones and outcomes that have been accomplished as of October 2017: TABLE 5. MILESTONES AND OUTCOMES AT THE MID-TERM OF NEI+ RESULTS SCHEDULE COMMENT Intermediate Result 1: Government systems strengthened to increase the number of students enrolled appropriated relevant and approved educational options especially for girls & out of school children OOSC in target locations Sub IR.1.1-Increased # of education options (formal/non-formal) meeting school quality and safety standards. Sub IR1.1 April-June 2017 Unknown if this task has been accomplished: Data analysis is on-going so results were not available or known at the time of the Mid-Term Evaluation. Sub IR1.2-Stregheneed systemic approach to school management and supervision. Sub IR1.2 April-June 2017 Accomplished: Access TWG reviews Project Year 1 Performance Sub IR1.3-Institutionalised model of NFLCs to ensure education for vulnerable children and youth. Sub IR1.3 April-June 2017 Accomplished: More than 27,000 learners pass NFLC examinations. SubIR1.4-Core curriculum is adopted in NFLCs. Sub IR1.4 April-June 2017 Accomplished: Core curriculum developed and being used in NFLCs. Sub IR1.5-Streghetend CSO capacity to mobilize PTA /SMBC and communities around reading and access. Sub IR1.5 April-June 2017 Accomplished: The Initiative issued grants to 47 CSOs to manage 1,400 NFLCs period of performance - June 2017 will end in April 2018. First Aid kits /Mobile money as in-kind contribution, Kits, 2017. Mobile Money when learning activities commence. Utilization of Mobile Money for financial transactions in process using e-Transact technology for Learning Facilitators (Bauchi & Sokoto Staff and partners. Accomplished 2: Trained CSOs to mobilize communities to support education; Accomplished 3: Produced behavior change media campaigns to complement community outreach. 34 HOW HAS THE CURRENT ECONOMIC CLIMATE AFFECTED RELEASE AND EXPENDITURE OF BUDGETED FUNDS? Nigeria’s current economic crisis impinges on the country’s resources, weakens the system horizontally and vertically, and can work against the accomplishment of the NEI+ activity goals. State authorities rely on strong advocates at the Federal level to lobby for resources when budgets are appropriated for a new fiscal year. While budgets are planned and committed in word, the actual release of funds to the States has consequences that are obvious at a surface level. Aspects that were apparent to the team included: Intermediate Result 2 - Government systems strengthened to improve reading outcomes for primary grade learners in target locations Sub I R 2.1-State and LGEA policies, time tables and standards for reading instructions and performance improved and implemented. Sub IR2.1 April￾June 2017 Accomplished: Reading Policy framework, standards and benchmarks ongoing with Federal and state partners. 38 Education Management Information Systems (EMIS) officers / Education Managers trained on the use of SPSC an application for measuring school performance. Sub IR2.2-Stae and LGEA systems for the development, approval and distribution of decodable readers, teachers’ guides and supplementary materials for EGR instruction improved and implemented. Sub IR2. 2. April￾June 2017 Accomplishment of this task is pending and dependent on the on-time delivery of learning material in formal schools in Bauchi. Mu Karanta ! Let’s Read! TLMs for P1-3 developed and improved but not fully or effectively implemented due to lack of appropriate learning material in all formal schools in Bauchi. Sub IR2.3-State and LGEA systems for providing in-service training to teachers in formal and NFLCs in the use of the evidenced based reading materials improved and implemented. Sub IR 2.3 April￾June 2017 Accomplished: Improved and implemented. Initiative engaged a producer to support production of media content for EGR support. SubIR2.4-LGEA systems for monitoring and coaching in-service teachers in EGR instruction improved and implemented. Sub IR 2.4 April￾June 2017 Accomplished - Conducted coaching and mentoring visits to schools. SubIR2.5-State and LGEA systems for EGR improved and implemented. Sub IR 2.5 April￾June 2017 Accomplished: Conducted EGRA Instrument review and Development Workshop. Conducted EGRA Data Analysis and Reporting Training SubIR2.6-State and LGEA systems for extending evidenced based reading instruction to nontraditional, non– formal schools improved and implemented. Sub IR 2.6 April￾June 2017 Accomplished: EGR Instruction to NFLCs extended. Sub IR2.7-State and Local government’s accountability towards the public regarding reading instruction increased. Sub IR 2.7 April￾June 2017 Accomplished: Community Reading Charter developed and validated. 35 (1) A lack of clarity where there should be predictable standards of procedures and processes, as in the book delivery system, (2) A sense of inertia at the administrative and managerial levels where one would expect busyness and efficiency, and (3) Disconnects and lack of effective interfacing in the technology sector where quick access to reliable school-related data is central to making data-driven decisions. These factors, left unresolved, can constrain effective planning, management, and administration for Nigeria’s education sector in general and for the achievement of NEI+ expectations in particular. Sokoto’s State budget for education is 23.7% and Bauchi’s is 20.0%. However, budgets that are approved on paper are not necessarily distributed. Shortages in the area of salaries, training, retirement/pensions and school materials are common. At one school in Sokoto, an inspection team from the LGEA was verifying the number of teachers and pupils at school in hopes of eliminating “ghost” teachers and pupils from official registers. Eliminating these non-contributing staff from the payment rolls will increase available funds for schools. State government officials have been forced to recognize the need for focused and strategic advocacy campaigning at the Federal level to ensure sufficient funds for education, especially early childhood education. A citizenry that now values education for boys and girls is a force that should be mobilized. WHAT COULD BE DONE DIFFERENTLY TO STRENGTHEN THE EDUCATIONAL SYSTEM? Below are recommendations that can assist in strengthening the educational system: • Strengthen the bottom to pressure the top – Communities surrounding formal schools and especially NFLCs have a sense of ownership of NEI+. Textbooks, training, and evidence of literate children encourage communities and parents to trust that the education system is working within the NEI+ context. Devote resources to this constituency as much as is devoted to the “top.” Most communities have an honest sense of community and school ownership, but it seems to dissipate moving up the administrative ladder. • Launch a media campaign to tap resources in the private sector, especially “Old Boy” networks. In Sokoto, “Old Boys” are men who are successful in the business sector and were raised in impoverished communities. One school in Yabo, established in 1992, has many alumni contributing to the success of the school. It was the best resourced school of all schools observed. • One high-ranking education authority was asked to describe the State’s sustainability plan once NEI+ ends. With an earnest expression on his face he answered, “We’ll just wait for the next development donor initiative.” Attitudes of dependency rather than attitudes of self-reliance need to be changed. Frank and carefully-facilitated discussions about the consequences of dependency￾thinking might promote micro-change that can nudge macro-change at the top. • Call on the most influential state government authorities and respected elder statesmen to advocate for transparency at Federal, State and LGE levels. • An MOU has been signed between USAID and Bauchi & Sokoto to ensure their commitment on financial responsibility. Ensure the States adhere to the agreement. • Government’s political will to support education is the greatest support needed. Sokoto state government recently declared a state of emergency for education in order to ensure a meaningful 36 impact in education. The Sultan (ruler of Sokoto) is advocating for education to ensure that all school age children are enrolled and attend school. Nurture this relationship. • Engage the Committee of Religious Leaders who have called on communities to encourage parents to access education in NFLCs. • Increase monitoring of LGEAs as it pertains to actual time on the job and the roles and responsibilities of SSOs who are responsible to report to their respective LGEAs. QUESTION 4: TO WHAT EXTENT HAS THE NEI+ RESEARCH AGENDA BEEN FORMULATED TO ADDRESS KEY CHALLENGES TO EFFECTIVE EGR AND ACCESS; HOW WILL THE RESEARCH RESULTS SUPPORT PROJECT IMPLEMENTATION AND RESULTS? The program focus has shifted since 2016 from learner-based to system-based reform. The program agenda as stated in the 2016 Monitoring and Evaluation Plan and Performance Indicator Reference Sheets (PIRS) does address key challenges to assure effective EGR and access: • “Focus on issues relating to factors affecting learner access and education quality.” (p.27) An update in the Year 2 (2017) Third Quarter Report focuses programmatic activities more explicitly on demonstrating: • “How systems can be strengthened to provide a strong education foundation for children to learn to read and write in Hausa and prepare for English in upper primary school.” (p.9) A shift to “systems strengthening” narrows the focus of program activities at this mid-point in program implementation. This question focuses extensively on the status of NEI+ systems strengthening milestones and outcomes in 2017 and beyond. Identified key challenges in systems strengthening that influence the success of pupils learning to read and write include: the timely delivery of textbooks plus the demand for additional textbooks due to ever increasing enrollment; assuring teacher quality through in-service and pre-service training; supervision and support for administrators; and the establishment of enduring policies accompanied by realistic sustainability plans. Monitoring, Evaluation, Assessment and Research (MEAR) Technical Working Groups (TWG) are established in both Bauchi and Sokoto to guide and recommend improvements across related focal areas. At the May 2016 joint meeting, it was decided to prioritize research areas identified in Year 2. Results from these research areas are more in-line with 2016 program agenda than with the updated 2017 one. Following is an examination of the research topics and a brief discussion of their potential for support of current program implementation and end of program results. Three potential topics were identified in Year 2 of NEI+ for further study. These research areas are meant to support project implementation and results: • The transition from Hausa (L1) to English (L2) as a language of instruction in schools • “Time on task” teaching reading • To compare the performance of NFLC/AGLC attendees and formal school pupils, with a potential follow-up study in year 3 on the readiness of NFLC learners for mainstreaming into formal schools in P4. 37 TRANSITION FROM HAUSA TO ENGLISH Many of the skills learned while reading and writing in Hausa will be used when pupils learn to read in English, (Cummins, 1979), because a student only needs to learn to read once. There are some challenges for Hausa speaking pupils as they will need to be able to associate the 44 sounds in English with just 26 letters. Hausa is a tonal language with approximately 26 (depending on the dialect) phonemes. Mu Karanta presents a clear method for teaching Hausa reading that appears successful. Learning spoken English in primary schools may be challenging for young learners, as some primary teachers stated during interviews that speaking English was a challenge for them. English is rarely heard in rural and remote rural environments, so there are few language models available. Some Primary 3 teachers confirmed that they did not teach Mu Karanta in English, only in Hausa. Teachers may not be providing a language rich environment for pupils as their proficiency may not have gained full fluency. Evaluation observation results indicate that 75% of the 29 classroom observations during this evaluation were conducted in Hausa, with 45% in 10 Primary 2 classes and 55% in 12 Primary 3 classes. Increasing teacher in-service and pre-service training conducted in English with an emphasis on expanding spoken language proficiency would be beneficial. The evidence presented in question 1 of this evaluation report demonstrates that evidence-based instructional strategies, techniques, and best practices are being applied but could be strengthened through additional training. An extensive body of research literature4 exists on teaching young learners as they transition from reading in their home language to reading in English. Research studies conducted in Africa, Europe and the United States conclude that the use of identified effective instructional strategies speed the acquisition of oral language and reading skills. Research focused on identification of effective strategies to ease the transition between Hausa reading and English reading could be conducted through a literature review to determine if a full action research study is warranted. TIME ON TASK Past research studies in EGR and similar USAID funded programs have shown that time on task is likely to increase pupil reading performance. For example, an EQUIP 123 case study in Guatemala found that “more of the school day should be effectively used and more time devoted to reading.” Similarly, the 2012 CAII-produced “Malawi National Early Grade Reading Assessment Survey” recommends action to “maximize time on task for reading practice (including lengthening the school day, and reinforcing reading concepts throughout the day, across subject areas).” It is logical that the more time spent teaching reading efficiently, and the more time with eyes on print in the classroom, the more likely pupils are to increase reading proficiency. The issue here is how to achieve this. Many head teachers and class teachers in formal schools stated that the Mu Karanta lessons were too long, and it was difficult to complete them in one extended class period. Non-formal centers provide Hausa and English reading instruction in a three-hour block, which allows more time for longer textbook lessons. In formal schools, some teachers are more specialized or trained, as a result they may teach Mu 4 One of the latest books with a current bibliography is Shin, J.K. and Crandall, J. (2014) “Teaching Young Learners English: From Theory to Practice.” National Geographic Learning/Heinle. 38 Karanta English to more than one class, requiring an on-time schedule. A time on task study may also focus on attendance of both teachers and pupils. Class size can double with latecomers. At one formal school visited, school inspectors were also counting the number of teachers in classrooms with the goal of eliminating “ghost” teachers. A time on task study is likely to focus on classroom actions, but a shift in focus to strengthen instructional systems could look at the length of the school day, efficient timetabling by head teachers, teacher attendance, and how reading skills can be integrated into subject area instruction such as science, mathematics, and social studies. Of critical importance to this study is the absence of teachers in classrooms. Unannounced visits by CAII personnel report that some teachers in formal schools are frequently not in the classroom teaching; they roam the halls, stand around chatting, or do not account for their absenteeism, which is high. Some schools have no teachers or too few for the number of pupils. Evaluators visited a formal school in Gwam, Bauchi, where there were 700 pupils for one head teacher/classroom teacher. Classrooms were supervised, not taught, by community volunteers who had cane “switches” in hand to manage the over-populated school. PERFORMANCE OF NFLC/AGLC PUPILS It has been noted in NEI+ quarterly reports that NFLC and AGLC pupils may be performing better than formal school pupils. Validation of this hypothesis can be analyzed after the EGRA assessment for 2017, which is currently underway, is completed. One of the benefits of the NEI+ EGR activity is the richness of performance data. These results may confirm the hypothesis but causality may be more difficult to pin point. Potential influences on pupil performance observed during this mid-term evaluation include: • NFLC/AGLC facilitators observed in Bauchi demonstrated creative instructional strategies adapted to the local context more often than formal school teachers. • Smaller class size especially at AGLC. • Classes are held in the afternoon and on weekends opposite the formal school schedule. Some pupils in NFLC/AGLCs classes held Mu Karanta textbooks that may have been distributed at a formal school. • Older young adults and adolescent girls are more committed to gaining literacy since they missed the opportunity earlier in life. QUESTION 5: IN WHAT WAYS DOES NEI+ UTILIZE ICT TO SUPPORT PROJECT IMPLEMENTATION, ADMINISTRATION AND M&E? Evaluators interviewed Education Management Information System (EMIS) officers who are responsible for maintaining information technology facilities and learned about the extensive training that CAII has provided that includes collection, retrieval, and storage of data. Great strides have been made in this part of the system with extensive ICT training provided by CAIII. However, the variety of computer platforms and software packages that are used by different development donors inhibits the overall ICT system to operate efficiently. EMIS personnel interviewed stated that they 39 are expected to manage different data systems for different donor expectations. The absence of efficient interfacing across the various systems and databases results in misuse of valuable time. Simple bits of information, such as accurate and current phone numbers for SSOs, Area Coordinators, grantee managers, etc. were difficult to obtain by the evaluation team in Abuja and especially in the field. Despite the system’s infrastructure weaknesses noted above, highlights of CAII’s efforts in this sector were evident: • CAII’s Dashboard offers relevant statistics in real time can serve as a centralized data system. Tracking the frequency of use and types of users could help determine the Dashboard’s return on investment. At the State, LGEA, and school level no users were identified. • UNICEF’s Annual School Census stands alone as the most reliable and trusted data set used by the education sector and CAII has offered training to the ICT sector on data analysis in order to maximize the use of UNICEF’s data. • An improved EMIS infrastructure is established and Excel training for 20 Sokoto LGAs in data collection, retrieval, storage and analysis has been provided. • ICT sector in Sokoto boasts its solar panels to charge computers in primary schools which enables EMIS personnel to effectively apply CAII training in data retrieval, storage and analysis. • CAII continues to provide ample training opportunities to upgrade stakeholder capabilities, especially for EMIS officers to increase the flow of data especially from Federal to State and from State offices to LGAs. The table below lists findings and issues evaluators heard during their interviews with State Directors of Planning, Research and Statistics and EMIS officers. The findings and issues are categorized into implementation, administration and management, and M&E. 40 FINDINGS AND ISSUES ICT Implementation Training in the use of BLOOM software for book production and producing leveled & decodable readers has been provided. Special attention to Hausa & English standards for reading specialists was include. Assistance in book distribution using a tech-supported paper-tracking system CommCare registration & attendance apps, community mapping apps, teaching & learning material distribution app, SMS tracking of teaching learning materials Trained head teachers to use tablet￾based observation process to conduct classroom observations & improve data management at schools. System designed to interface with Dashboard. SMS messaging to provide coaching to LGAs’, SSOs & trainers to support & supplement Mu Karanta! Coaching and Training guide. Tips support parents to reinforce reading at home by setting up reading corners & reading time. Mobile tablet-based school support systems include modules such as Classroom Observations in Hausa & English, interviewing head teachers, questions, pictures & guides on EGR best practices Behavior change communication messages/SMS, interactive voice response/IVR, radio jingles, public service announcements & drama skits Digital coaching & mentoring Training to use mobile technology for monitoring and coaching in-service teachers. SSOs. CSOs use What’s App SMS for the same & also for community mobilization Survey-to-Go application for EGRA tool ICT Implementation Issues UNICEF’s Annual School Census is the only data set unanimously trusted and used in the education sector. Donor use of various computer platforms and software to collect and analyze data. Platforms are incompatible with others which restricts access to previously collected data by other donors or projects. This results in time-consuming data collection to track down data. The effective flow of data up and down the information chain is constrained by attitudes and behaviors that are rooted in self-interest rather than the greater good. In Bauchi, computers and printers were not evident in formal schools or at NFLCs. IT personnel are not available to repair computers and printers at schools. Cost for repairs are sometimes covered out-of-pocket by a Head Teacher. In Sokoto, computers and printers were not evident in formal schools or NFLCs because the schools generally do not have reliable electricity. One exception was in Yabo, where two computer labs are established each with 10 computers for use by P4 – P6 pupils. One computer rooms was supplied by a generator and another one by a solar battery backup. Older computers were provided by UBEC. State officials who have computers and printers do not maximize their use as some officials are not computer literate. Rather than move the equipment so others can use the technology is, instead, left under plastic covers. EMIS personnel noted the “low capacity to analyze data”. during work hours. EMIS master trainers are available but offices lack the infrastructure to apply what is learned during training. Donors and NEI+ partners use various platforms and software including Structured Query Language/SQL, DFID uses Access database, UNICEF uses SQL, other use Excel. The single most used and trusted data set is UNICEF’s Annual School Census. EMIS personnel complained that state offices not have a single database that is harmonized/synchronized. Data is shared with head teachers only when it is requested. Big education expenditures are sometimes made for the wrong equipment or services. Administration & Management SharePoint / Dropbox management Training in Excel COMCARE – evidence based mobile platform, data flows into excel link with data management systems like DHIS2, Tableau, or SQL. Administration & Management Issues • EMIS officers often use personal hotspots or do some work at internet Café’s where they can use public Wi-Fi. • BASAME which managed 400 NFLCs in Yr2 & will manage 700 additional in Yr3 does not have a server. This limits its ability to collaborate and engage effectively with State stakeholders. • EMIS officers report it is not standard practice to share data and information across agencies. Two EMIS officers expressed a personal unwillingness to share databases they “developed through hard, hard work.” • SUBEB does not have dependable intel facilities • 20 local government personnel were trained but do not have access to suitable facilities including internet connectivity or working computers and printers. • Frequent power outages prevent a significant number of State personnel from maintaining reliable internet connectivity to keep data and information flowing easily and regularly between one state office and another. • Even though State officials remarked “we have moved from the old days of paper and pen/pencil” many must rely on the “old days” due to lack of computer technology in offices. • From one EMIS officer, “EMIS is not in charge of its own data. I manage equipment in 21 learning centers but none of the centers have internet connectivity. • The banking platform, E-Transect & other electronic strategies to pay NFLC facilitators struggles because users either do not have bank account numbers; do not use the required personal mobile number; or do not understand how to use the system even though they were trained. M&E SPSS to analyze NFLC EGRA datasets and to generate reports Tablets with Survey-to-Go for use by SSOs and Area Coordinators to conduct EGRA CommCarehq.com used in rapid appraisal for gender analysis making it the first time for many to use this digital process. (UAID/NIGERIA, Gender Analysis, 2017). Tablets to collect and assess data Training for M&E database and IMS to enable EMIS staff in collection, storage and retrieval; has increased data reliability Digital tablets for community mapping GIS to provide an overview of education access and to understand geographic factors impacting learning communities, i.e., frequent movement of IDPs SPSC and Excel to measure school performance Evidence-based mobile platform, COMCARE, in which data flows into Excel link with data management systems like DHIS2, Tableau, or SQL M&E Issues • M&E is a challenge due to a wide coverage area; the system lacks sufficient human resources to monitor effectively. • Data-driven decision-making for strategic planning or M&E is not standard practice. According to an M&E officer “personnel do not know which database to believe and many do not know how to analyze the data they collect”. • An EMIS member commented: “Each person develops his own database and keeps it private; he has worked so hard on it and is not willing to give it away.” • Systems that stand alone are doing well but cannot interface with other organizations. The system lacks reliable linkages across databases. • Strategic planning and decisions in general are not data￾driven. A State EMIS officer noted “Many can collect data; few can analyze the data they collect. There is no such thing as data-driven decision making.” 41 ISSUES A number of issues facing the mid-term evaluation team are described below: • NEI+ is a multi-layered, multi-faceted, dynamic activity being implemented within a context whose environment includes a national economic crisis, lack of transparent practices throughout ministries, and day-to-day practicalities that impinge on efficient, effective and productive work. Sufficient time to conduct quality data analysis must be provided to future evaluation teams in order to produce results that are meaningful, informative, and useful for what remains of program activity. Additional time was, eventually, granted and aided in completion of all tasks. • The SOW indicated that, prior to arrival in Nigeria, the evaluation team would be provided the following (1) arranged meetings and interviews supported by USAID and the Initiatives COP and (2) initial contact lists of beneficiaries, GON officials, other donors and other USAID activity representative. Instead, this information was gathered during the first two weeks of the consultancy which took valuable time away from field work preparations. • The significance and value of making unannounced visits to NEI+ activity sites are recognized as good evaluation practice in some instances. However, the evaluation team spent an inordinate amount of time obtaining reliable, up-to-date information on names and locations of schools and NFLCs, and names and contact information of key stakeholders. In addition, obtaining permission to meet government officials in Bauchi and Sokoto was complicated and time-intensive. Multi￾layered bureaucratic requirements meant developing formal letters to high-level State authorities, that were rarely requested once the team arrived at State offices. A streamlined process for gaining access to NEI+ stakeholders, especially government officials, and sites would have aided the evaluation team’s fieldwork. 42 RECOMMENDATIONS Following are recommendations based on findings in each of the evaluation questions: The use of international, evidence-based best practices – • On-time delivery of Mu Karanta textbooks to eliminate gaps in instruction. • An allotment of extra books for newly enrolled pupils or lost books must be provided for formal schools facing unanticipated mushrooming enrollment since the inception of State Government’s school feeding program commenced. Books in pupils’ hands are essential. Expand in-service training and include refresher training – • Retain NEI+ EGR trained teachers by expanding their knowledge base, skills and motivation with refresher trainings. • Focus more training on pupil-centered, activity-based classroom management techniques for teachers who will more than likely have classes of 100+ pupils. • For facilitators in NFLCs, incorporate topics such as tailoring, sewing, mechanics, carpentry, and technology as part of reading lessons to increase interest and motivation by associating learning to living. In formal schools, reading skill lessons can be incorporated into science, mathematics, and social studies instruction. • Assure a print rich environment by training teachers to make teaching aids during in-service and pre-service training. • Display examples of how to make teaching aids in the resource rooms where pre-service and in￾service is delivered especially in the CoEs’ EGR training laboratories. • Engage student teachers to develop a teacher “kit” of supplemental materials aligned with the Mu Karanta Teacher’s Guide to take to their job in the classroom. SSO roles should be clearly delineated with SUBEB and LGEA officials and accountability of SSO coaching and mentoring should be strengthened – • Ensure that the reporting system for SSOs accurately describes actual time coaching and mentoring classroom teachers. • Engage SSOs in Cascade/step down training in formal schools. • Prepare for sustainability by including the development of a basic “teacher’s tool kit” during pre￾service and/or in-service training for use post-training. • SSOs serve the government and NEI+ and are involved in concurrent activities which affects full participation in either of their two roles. They are overburdened between project activities and official assignments and some sacrifice the role and responsibility for one over the other. The challenge is how to integrate NEI+ activities into official assignments. Although work plans have been developed, there is concern that SSOs are not providing adequate coaching and mentoring in formal schools. Address gender-based violence issues in the conflict sensitivity domain – • Expand the curriculum to include girls’ learning needs and interests specifically health and vocational skills. o Respond to parents’ concerns regarding fear of rape, kidnapping, physical and verbal abuse, and gender-based violence by ensuring that parents whose children are enrolled in the NEI+ 43 activity are aware of the legal roles and responsibilities of local law enforcement as it pertains to the safety and security of all Nigerians especially children. Increase support of CAII’s Access and Fragility managers to further develop safety and security measures for vulnerable boys and girls around the more culturally sensitive topics of physical and verbal abuse and gender-based violence. A common belief expressed by an LGE education secretary in Bauchi was that “caning children is necessary to properly raise or educate children.” Such thinking and behavior must be discouraged and discontinued in NEI+ settings. The same evaluators witnessed children being disciplined with light “swipes” on their ankles by long canes held in the hands of male, classroom volunteers. To counter beliefs, attitudes and behaviors that harm children an inventive, culturally-appropriate approach such as a radio or print campaign or, at a minimum, talks with parents focusing on the value and importance of protecting children’s’ mind, body and spirit to support learning is advised. Rather than communicate punitive consequences of failing to protect children, emphasize the importance of respecting and protecting children. • Partner with special education NGOs or CSOs to better serve the needs of children with disabilities. o Address the needs of children with disabilities by providing physical accommodations, transportation, or other accommodations. o At a minimum, obtain and provide the Universal Sign Language chart in formal schools and classrooms, and NFLCs serving deaf children. o At a minimum, obtain and provide basic braille learning materials from a special school for disabled children and make them available in formal schools and classrooms, and NFLCs. Strategies for systems strengthening: • CAII must be strategic in focusing on domains within the education system that have a high probability of systemic change rather than attempt to focus on all twelve domains. (Refer to USAID/Nigeria’s Institutional Capacity Assessment document regarding the twelve domains). • Determine the real return on investment of CAII’s Dashboard. Identify frequency of use and who are most frequent users. Assess actual use of the Dashboard at the State level. • Identify bottle-necks in the book delivery system and find a leverage point to pressure States to make good on its commitment to provide teaching and learning materials on time especially for formal schools. • Answer the question: Is it known exactly what prevents book delivery from being on time? If not, answer the question and apply pressure on that point. • Identify which step(s) in the process is responsible for persistent delays in book delivery. • Identify the system’s “sweet spot” that could leverage influence or increase political will to bring about sustainable change after NEI+ funding ends. • Convene a type of “think tank” for senior education authorities who have an interest in change. Collaborate with them and channel support to areas or individuals who have proven to positively impact the ultimate beneficiary – children learning to read. Focus on micro- not macro-aspects of the “system”. • Do not permit school administrators to transfer NEI+ teachers for a minimum of 2 years after they’ve been trained. 44 Refine the research agenda: • Planned research topics should be reconsidered to assure alignment with the 2017 program agenda and expected results after two more years of implementation. • Tap the global body of literature on transitioning from L1 to L2 and draw from lessons learned over the past decades in the field of mother tongue and second language acquisition. • Provide a rationale for Time on Task research and develop a strategy to identify likely leverage points within the Nigerian education system to address high teacher absenteeism. Absence of teachers in EGR classrooms and ineffective time on task can adversely impact NEI+ reading performance goals. • Align timed textbook lesson activities with what actually occurs in classrooms. Help teachers prioritize lesson activities and assist them to learn how to accelerate learning when breaks or delays in the timed scripts are interrupted, particularly for classroom teachers who have 85+ students in one classroom. Strengthen ICT training: • Include simple data analysis techniques during ICT training and repeat this many times over. Provide context-specific examples on how to use data to make data-driven decisions. • Assess data analysis skill levels among State officials who are responsible for making data-driven decisions, and fill in knowledge gaps as needed. Assess their data-driven decision-making skills. • Assess how frequently and how effectively CAII’s Dashboard is used. Determine who are its most frequent users and what obstacles stand in the way of easy access at all levels of the education system. • Identify ways for EMIS personnel who lack reliable internet connectivity to access Dashboard. • Determine if tablets and survey-to-go software are used correctly and being maximized especially by School Support Officers (SSOs). • Conduct a mini-research to determine if SSO coaching and mentoring is viewed as being effective by classroom teachers. FUTURE DIRECTIONS The evaluation team sees CAII’s future direction as being one in which it will prove effective by remaining true to the philosophical and methodological principles of EGR approach. If it can tap that “sweet spot” for change, it can focus and support collaboration-friendly entities and like-minded stakeholders who have a proclivity for change and who demonstrate a willingness to change behaviors and attitudes at the individual level. Schools and communities are excited to see pupils learning and demonstrate it through visible support at schools and in their homes. Classroom teachers and learning facilitators, when trained effectively, rarely forget lessons learned. Rather, they typically are equipped to apply the fundamentals of teaching or facilitation in a variety of work situations and settings. The NEI+ activity has been established on sound pedagogical fundamentals, i.e. EGR’s philosophy and theory is congruent with its methodological practice. The evaluation team encourages support of this activity that stands on sound pedagogical ground. What the NEI+ Mid-Term Evaluation team observed and heard during field work encourages it to recommend that USAID continue to do the following: 45 • Assess the consequences of high teacher absenteeism on learner performance and achievement in formal classrooms; • Consider lost learning time caused by persistent late delivery of textbooks specifically in Bauchi State; • Strengthen the SSO component by ensuring that SSOs efficiently and effectively fulfill their coaching and mentoring responsibilities; • Strengthen data analysis capacity of key stakeholders outside of the ICT community; • Design an approach to change organizational culture as a way to target weak links within the educational institution that inhibit effective progress and efficiency and prohibit transformation; • Address the software/database interface disconnects across the ICT community. Addressing these areas can strengthen NEI+ for the time that remains. 46 ANNEXES 47 ANNEX I: LIST OF DOCUMENTS REVIEWED Bean, R.M. (2014). The Power of Coaching: Improving Early Grade Reading Instruction in Developing Countries. USAID: Washington, D.C: PP. 6 – 31. Crouch, L., & DeStefano, J. (2015). A practical approach to in-country systems research. Unpublished paper. RTI International, Research Triangle, North Carolina. Cummins, J. (1979). Linguistic interdependence and the educational development of bilingual children. Review of educational research, 49(2), 222-251. http://dx.doi.org/10.3102/00346543049002222 Early Grade Reading Assessment Survey, Final Assessment – November 2012 http://pdf.usaid.gov/pdf_docs/PA00JB9R.pdf Pflepsen, A., Gove, A., Warrick, R. D., Yusuf, M. B., & Bello, B. I. (2016). Real Life Lessons in Literacy Assessment: The Case of the Early Grade Reading Assessment in Nigeria. In Annual Review of Comparative and International Education 2016 (pp. 129-145). Emerald Group Publishing Limited. McCloskey, M. L., Orr, J., & Dolitsky, M. (2006). Teaching English as a Foreign Language in Primary School. Case Studies in TESOL Practice Series. Teachers of English to Speakers of Other Languages, Inc. Snow, C. E., Burns, M. S., & Griffin, P. (1998). Preventing reading difficulties in young children committee on the prevention of reading difficulties in young children. Washington, DC: National Research Council. USAID/Malawi. (August 2011). Malawi Teacher Professional Development Support (MTPDS). https://pdf.usaid.gov/pdf_docs/PA00HVSP.pdf USAID/Nigeria, Bauchi and Sokoto CECA/Gender Analysis. January, 2017. http://pdf.usaid.gov/pdf_docs/PA00MQV1.pdf USAID/Nigeria, Bauchi CECA/Gender Analysis, June 2016. http://pdf.usaid.gov/pdf_docs/pa00mvdb.pdf Creative Associates International, Inc. (CAII), Mercy Corps-2016-[PDF, 710 KB] USAID/Nigeria, Sokoto CECA/Gender Analysis, June 2016. http://pdf.usaid.gov/pdf_docs/pa00mvd9.pdf Creative Associates International, Inc. (CAII), Mercy Corps USAID/Nigeria, Bauchi and Sokoto R-CECA Data Collection Training and Assessment Report, November – December, 2016. USAID/Nigeria, Communique, Roundtable Meeting of Key Stakeholders on Basic Education Budget Preparation and Performance for Bauchi State, Grand Cubana Hotel, Abuja, October 5, 2017 USAID/Nigeria, NEI+, Introduction to Teaching Reading in Primary 1 – 3: Student Teacher Resources Book, NCE Pilot Course for Year Two, Pilot Edition, 2017. USAID/Nigeria, Introduction to Teaching Reading in Primary 1 – 3: Teacher Educator Guide, NCE Pilot Course for Year Two, Pilot Edition, 2017. USAID/Nigeria, Teacher’s Guide, Primary 3, Term Two, English, First Edition, 2016. 48 USAID/Nigeria, Mid-Term Performance Evaluation, Final Report, November 2013. USAID/Nigeria, Mu Karanta! Let’s Read! Guide to Coaching and Professional Learning Communities, First Edition, 2016. USAID/Nigeria, Mu Karanta! Let’s Read! Teacher’s Training Manual, First Edition, 2016. USAID/Nigeria Northern Education Initiative Plus, Monitoring and Evaluation Plan and Performance Indicator Reference Sheets (PIRS) February 12, 2016. USAID/Nigeria, Northern Education Initiative Plus Quarterly Report, First Quarter-October1 to December 31, 2016. http://pdf.usaid.gov/pdf_docs/PA00MVD3.pdf USAID/Nigeria Northern Education Initiative Plus, Quarterly Report, Second Quarter. January 1, to March 31, 2017. http://pdf.usaid.gov/pdf_docs/pa00mvcz.pdf 2017 - [PDF, 3804 KB] USAID/Nigeria, Northern Education Initiative Plus, Quarterly Report, Third Quarter. April 1, to June 30, 2017. USAID/Nigeria, Northern Education Initiative Plus, Project Year One Work Plan.JanuarySeptember,2016.http://pdf.usaid.gov/pdf_docs/pa00mmt8.pdf Creative Associates International - 2016 - [PDF, 2547 KB]. USAID/Nigeria, Sokoto CECA/Gender Analysis, June 2016. http://pdf.usaid.gov/pdf_docs/pa00mvd9.pdf Creative Associates International, Inc. (CAII), Mercy Corps USAID/Nigeria, Northern Education Initiative Plus, Project Two Work Plan – Central Level, October 2016 – September, 2017, USAID/Nigeria, Northern Education Initiative Plus, Report of Institutional Capacity Assessment of Basic Education Agencies (MoE, SUBEB, BASAME & LGEA) in Sokoto State. August 16th, 2017. USAID/Nigeria, Northern Education Initiative Plus, Report of Institutional Capacity Assessment of Basic Education Agencies (MoE, SUBEB, BASAME & LGEA) in Bauchi State. August 16, 2017. USAID/Nigeria, Northern Education Initiative Plus, School/Centre Safety Standards(S/CSS) and Benchmarks September 30, 2016. USAID/Nigeria, Northern Education Initiative Plus, School /Centre Safety Standards Implementation Guidance. September 30, 2016. USAID/Nigeria, Northern Education Initiative Plus Annual Report. October 1, 2015 to September30, 2016. http://pdf.usaid.gov/pdf_docs/pa00mmqd.pdf Creative Associates International USAID/Nigeria, NEI+ Sokoto Communications and Advocacy Strategy. March 2016. USAID/Nigeria, Trip Reports submitted to CAII, Abuja, 2016 - 2017. Using Opportunity to Learn and Early Grade Reading Fluency to Measure School Effectiveness in Guatemala: Case Study. http://www.equip123.net/docs/E2-SEGuatemalaCaseStudy.pdf 49 Warwick B. Elley. (1992). How in the world do students read? IEA Study of Reading Literacy. International Association for the Evaluation of Educational Achievement. 50 ANNEX II: EVALUATION SCOPE OF WORK FOR NORTHERN EDUCATION INITIATIVE PLUS (NEI+) Evaluation Scope of Work for Northern Education Initiative Plus (NEI+) BACKGROUND INFORMATION Activity Identification Data Development Objective Activity Title A healthier, more educated population in targeted states Northern Education Initiative Plus (The Initiative) Award Number Award Dates AID-620-C-15-00002 10-26-2015- 10-25-2020 Funding Implementing Partner(s) $117,499,302.00 Creative Associate International. Inc. COR Olawale Samuel 1. BACKGROUND INFORMATION Nigeria is central to the U.S. Government’s global development strategy. Although it is Africa’s largest economy, its most populous country and largest democracy, and a major oil exporter, Nigeria must overcome daunting development challenges if it is to fulfill its significant potential. Presently, nearly 65 percent of Nigerians live in extreme poverty. Already, Nigeria has the highest number of out-of-school children, UNICEF estimates that 10.5 million children are out of school, and those children who do attend school have learning outcomes that are among the worst in the world, the Nigerian National Bureau of Statistics reported that the average net attendance ratio of the northern states was 38.3 percent, meaning that 61.7 percent of the total number of children of primary school age were not attending school in the north. In 2016, USAID financed an Early Grade Reading Assessment (EGRA) in Hausa and English in grades two and three in Bauchi and Sokoto states. The results indicate that children lack foundational reading skills in Hausa, and those in P3 had not acquired them in English. The majority of students scored 0, meaning they could not provide a single correct response, on almost every subtask. Percentage of zero scores ranged up to 90% for Sokoto P2 pupils in oral reading fluency. The large percentage of zero scores of course rendered the mean scores across subtasks very low, with almost no children reading with at least 80% comprehension. The inadequate quality and relevance of education available in schools is the main contributing factor to low learning outcomes, made worse by the poor conditions of the learning environment to support teaching and learning. These include the poor condition of physical facilities, including water and sanitation facilities; inadequate teacher training and limited opportunities for professional development; and a lack of linkages to the needs of the labor market. Teacher qualifications and distribution have also been critical challenges to the successful delivery of basic education. Development Context In this context, USAID through the Initiative is most concerned with the capacity of “systems” (governmental and non-governmental) to manage themselves: both administratively and financially. The 51 primary intention is to focus on improved reading outcomes and increased access to learning by strengthening both the administrative and financial management functions of the state entities that oversee and provide education. The Initiative design draws upon several recent evaluations and analyses conducted in Nigeria. Household and schooling data collected from the Nigeria EdData Surveys, conducted in both 2010 and 2015 form the analytical basis of this activity design. In addition, the design was informed by the Focus States Strategy Assessment, the Mid-Term Performance Evaluation of the Leadership, Empowerment, Advocacy and Development Activity and, most importantly, the Northern Education Initiative Mid-Term Performance Evaluation (2011). Activity Objectives The key objective of the Initiative is to orient the functioning of government processes for the flow of sector information and resources with the specific outcomes of increasing equitable access and measurable achievement in early grade reading. The Initiative focuses on creating sustainable policies, strengthening planning, management and assessment systems, and building capacity of states, Local Government Education Areas, Civil Society Organizations, schools, and communities to deliver and support educational systems, with the end goal of increasing access to educational pathways and improving learners’ reading outcomes. By assisting partner states and LGEAs to structure both formal and non-formal learning opportunities that result in measurable improvements in reading, the Initiative is intended to: (1) advance the implementation of the basic education goals of USAID’s Education Strategy, (2) strengthen the systems and processes of select states and LGEAs specifically for service delivery in reading, and (3) provide children in the target areas with the skills needed to improve their learning outcomes, thereby lifting them out of abject poverty, and preparing them to participate as youth and adults in a democratic society. In so doing, the Initiative’s investments will catalyze measurable educational gains that will contribute in the long run to enhancing the stability and productivity of the US-Nigeria bilateral partnership. Development Hypothesis The Initiative’s development hypothesis is that achieving a better flow of information and resources will contribute to improved access and quality. First, this improved flow is expected to lead to an increase in the number of students, especially girls and vulnerable children (e.g., physically handicapped, and Out-of￾School Children (OOSC) enrolled in appropriate, relevant and approved educational options. Supported opportunities for out-of-school populations would also allow students, especially girls and vulnerable populations, to gain access to informal and formal education. Second, because quality is defined as improved reading skills in primary grades, the Initiative is also intended to contribute to improved reading outcomes for primary grade learners, a key skill needed by pupils to support learning in all other academic subjects. Expected Outcomes The two fundamental expected outcomes of the NEI+ activity are that: (1) all children in target LGEAs access learning (via public school, Non Formal Learning Center (NFLC), and (2) reading performance improves for grades one to three for children in all learning environments (public and NFLC) where the activity works. To achieve these two outcomes, LGEA, State Universal Basic Education Boards (SUBEB), and State Ministries of Education will work in partnership with the Initiative to ensure community mobilization for reading and access, improve data management targeting the expansion of access and improved reading performance. 52 To improve the functioning of state and local government systems towards reading outcomes, the Initiative is intended to build the capacity of LGEAs to train and monitor public school teachers in early grade reading and to procure and distribute learning materials for early grade reading for public and NFLC contexts, as well as learning materials for basic numeracy and life skills for non-formal learning environments. In terms of policy development, the Initiative is intended to support states develop and implement policies that support increased equitable access and improved reading performance, and to update those policies based on evidence provided through the Initiative. With the support of the Initiative, states and LGEAs will possess and use sets of validated materials for early grade reading in Hausa for grades one to three, for oral English instruction for grade two, and for English reading instruction for grade three. Furthermore, LGEAs will collaborate with partner CSOs to provide training for teachers in all school contexts on students’ social/emotional wellbeing and conflict resolution. With the support of the Initiative, states and LGEAs will possess and use research-based, criterion-referenced monitoring tools and classroom and student assessments that enable them to measure progress in expanding access, reducing risk, and improving early grade reading instruction. Critical Assumptions The current crisis is acute and highly destructive, and is rooted in deep issues that have been constant drivers of violence and exclusion in Nigeria and particularly in the North. While the assumption that if such conditions did not exist, outcomes would be more achievable, persistent conflict and insecurity and its impacts on education can affect the ability of programs operating in such an environment to fully reach expected outcomes. The Initiative is currently operating in the northern states of Bauchi and Sokoto, with the expected addition of a third state in the third year. The activity works in 10 LGEAs in each of the two states, making up a total of 20 LGEAs participating in the activity. Criteria for the selection of current target LGEAs include gross enrolment ratio (GER), net enrolment ratio (NER), literacy rate; number of out-of-school children, number of non-formal education learning centers; Percentage of qualified teachers in the state disaggregated by gender and local government; and number of facilitators in the non-formal education centers. The implementing contractor, Creative, is currently working on criteria for the selection of the third state in the north. In addition to the LGEAs and states, there are multiple other institutional partners in the Nigerian educational system that the Contractor engages in activity implementation. The table below lists key institutions the Initiative collaborates with, and also provides a brief description of the role of each institution. 53 NEI+ ACTIVITY RESULTS FRAMEWORK Partner Institutions and Roles Local Government Education Area (LGEA) • Implement all programming in both reading and access • Plan for and arrive at improved reading and access • Collaborate with NEI+ activity personnel and CSOs • to achieve improvements in reading and access State Universal Basic Education Boards (SUBEB) • Coordinate the development of statewide policies, standards, frameworks, and monitoring tools for both reading and access • Provide financing and logistical support to LGEAs for materials procurement, teacher training, monitoring, and testing State ministries of education 54 State Ministry of Education (SMOE) • Approve the final copies of statewide policies, standards, frameworks, and monitoring tools for both reading and access for use throughout the state • Report to the Federal Ministry of Education on the progress in reading and access in the State Agencies for Mass Education (or the equivalents) and state ministries of religious affairs • Lead the development and approval of appropriate materials for non-formal learning schools, and for non-formal learning centers Nigerian Educational Research and Development Council (NERDC) • Ensure conformity of teaching and learning materials with national primary school curriculum Universal Basic Education Commission (UBEC) • Ensure availability of basic education allocations to state ministries of finance National Council for Colleges of Education (NCCE) • Approve introduction of teaching and learning materials for primary grades into pre-service coursework designed to prepare new primary teachers National Council for Colleges of Education (NCCE) • Approve introduction of teaching and learning materials for primary grades into pre-service coursework designed to prepare new primary teachers Colleges of Education (COE) • Ensure availability of basic education allocations to state ministries of finance Federal Ministry of Education (FME) • Endorse decentralized efforts to improve reading and access • Incorporate best practices into national-level plans Approach and Implementation As expected by USAID, the Initiative is gradually applying and scaling-up evidence-based practices and approaches that have been proven effective in contexts like that of northern Nigeria. The Initiative currently plans for and implements the following approaches during activity implementation: Strengthening systems through embedded LGEA and state teams: By embedding the Initiative’s staff in focal states and local level education units, the contractor addresses priorities for human and resource management for improved reading outcomes. The contractor has developed and implemented tools and metrics for rigorous monitoring of capacity transfer in improving reading. Reducing risks related to learning: Sexual violence, psychosocial difficulties and physical violence are highly prevalent in Northern Nigeria. All activities both formal and non-formal work with local government and communities to help schools meet risk reduction indicators that address school-based violence, gender bias, ethnic bias, and promote social cohesion, equity and conflict resolution/resolution. Integrating a conflict-sensitive approach: Education can help promote social cohesion, contribute to identity formation, build peace, and bridge the gap between humanitarian assistance and sustainable development. However, education can also undermine these processes. When it is not provided 55 responsibly, education can be exclusionary, oppressive, exploitative and corrupt, and it can propagate extremism. All the Initiative’s activities in both the formal and non-formal sectors are designed to avoid reinforcing stereotypes and exclusion. Applying international, evidence-based best practices: The Initiative applies tested and proven strategies and techniques for improving reading outcomes, and reducing risks associated with education. These include community-based schooling; structured, phonetically-driven, sequenced reading instruction in languages children speak and understand; teaching children at their instructional level and providing remediation where needed; ensuring children have a solid foundation in Hausa to facilitate reading acquisition in English; attention to education needs of underserved groups (OOSC and youth, girls, and displaced populations); and do-no-harm provision of services for traumatized and endangered populations. Utilizing rigorous monitoring of progress towards outcomes: The Initiative is required to report on not only outputs (i.e. number of teachers trained), but also outcomes (i.e. percent of children achieving a passing score on an early grade reading assessment). Tools focused on quantifiable outcomes in reading are employed regularly in all the Initiative’s LGEAs. Data collected on increases in improvements in learning outcomes are expected to be reported both to USAID/Nigeria and to the development exchange center and the secondary Analysis Results Tracking (SART) mechanism funded by USAID/Washington. Existing Data In 2013, as part of the Education Data for Decision-Making (EdData) an EGRA was administered to 1881 pupils in Bauchi and 1674 pupils in Sokoto states in grades 2 and 3 in government and non-formal schools. 92 percent of 2nd graders in each of the two states could not read a single word on the test, although the test was administered in the Hausa language, a language the vast majority of the children tested do speak and understand. In addition to the EGRA baseline data, other documents that will be available to the evaluators will include the Initiative’s work plans, M&E plan, result framework and associated indicators, state government education sector reports and quarterly and annual reports. 2. EVALUATION RATIONALE Purpose The Northern Education Initiative Plus (also referred to as the Initiative), is a five-year program funded by USAID/Nigeria. The period of performance is October 26, 2015 to October 25, 2020. The purpose of this mid-term performance evaluation of the Initiative is twofold: • Assess the extent to which the activity is on track to meeting its key objectives; and, • Identify promising practices, unmet needs, or unintended consequences from implementation of the activity. The evaluation will be utilized to make decisions for possible modification of the activity for its duration and/or for follow-on activities. Audience and Intended Users The audience of the evaluation report will be USAID Mission, specifically the Education Team, key officials from the federal, state and local government level and the implementing partner. An executive summary and recommendations will be provided with recommendations to USAID on how it could use the evaluation findings to make changes and improve its current strategy of system strengthening at the state and local level to improve quality of education, offer a sincere hope of scaling up of best practices so that the audience will learn about the activity’s strengths and weaknesses and also use the outcome of the evaluation to serve as a learning platform. 56 Evidence of Participatory Approach Key government, community and other partners will be involved in designing data collection tools and evaluation methodologies to be adapted to suit institutional and community contexts; they will also be part of the research in the field as data collectors. The approach is to use the process to build local capacity and answer local needs. Dissemination Plan Final documents will also be available electronically through the Development Exchange Clearinghouse (DEC). The contractor may also wish to propose an alternative dissemination strategy for key stakeholders including the governments Bauchi and Sokoto state represented by their relevant agencies like the Adult and Non-Formal Education Agencies (ANFEA), State Agency for Mass Education, SUBEB, Colleges of Education (COE), Arabic and Islamic Education Board, officials from Local Government Education Authorities, CSOs, communities, schools, and stakeholders from Non-Formal Learning Centers. At the Federal level stakeholders include Federal Ministry of Education, Nigerian Educational Research and Development Council, Universal Basic Education Commission, National Council for Colleges of Education. Development partners especially DFID. Also, development partners including the United Kingdom Department for International Development (DfID), British Council, and UNICEF. Evaluation Questions The overarching development hypothesis for the program is: If state and local education actors can make education more child-friendly, relevant, flexible, and evidence-based, then greater numbers of children will have access to instruction that improves their reading and life skills. The contractor may revise the evaluation questions to test some aspect of the activity’s hypothesis in a logic model. The following questions cover the range of issues highlighted in the evaluation approach which the evaluation intends to address. 6. To what extent have international, evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? • How well are these strategies adapted to local contexts? What improvements could be made? 7. To what extent has the Initiative integrated a conflict-sensitive approach to education? • What measures has the Initiative taken to reduce risks associated with violence and conflict? How effective have these measures been? • Are the needs of underserved groups been met? If so, how well? What could be done better? 8. To what extent have systems strengthening outcomes and milestones been achieved? • To what degree have states and LGEAs assumed financial responsibility for activity implementation? How has the current economic climate affected release and expenditure of budgeted funds? • What could be done differently to better achieve educational systems strengthening and increased state and LGEA financial support? 9. To what extent has the NEIPlus research agenda been formulated to address key challenges to effective EGR and access; how will the research results support project implementation and results? 57 10. In what ways does NEIPlus utilize ICT to support project implementation, administration and M&E? 3. EVALUATION DESIGN AND METHODOLOGY This is a mid-term performance evaluation. The evaluation will use a non-experimental design and employ a range of quantitative and qualitative data collection methods in answering the evaluation questions. Although predominantly qualitative, the evaluator may make recommendations based on a mixed-methods approach involving (1) a desk review of available primary education documents; (2) semi-structured key informant interviews; and (3) site visits to schools involving semi-structured key informant interviews, focus groups, and mini-surveys and direct observation methods as appropriate for acquiring data from respondents. In the response, the evaluator should discuss the relative strengths and weaknesses of the proposed methodology. Data Collection Methods The Initiative’s reporting data will be sent to USAID/Washington in order to track progress against the Global Education Strategy’s Goal 1—100 million children with improved reading skills and Goal 3 which will increase access to education for 15 million learners in crisis and conflict affected countries. While Goal 1 has a sharp focus on early grade reading, education programming in conflict and crisis affected settings requires a broader and more contextual approach. Goal 3 will support learning opportunities for children and youth; strengthen crisis prevention efforts; and develop host country institutional capacity to provide education services. As soon as the sample is selected, the evaluators would work with USAID and the Initiative’s Contracting Officer Representative (COR) to obtain approval for contacting schools, NFLCs and other beneficiaries in order to alert them of the impending data collection activities. The contractor would organize a pre￾data collection visit, especially to selected schools and centers, to: a) gain advance entry into the schools/centers and explain the purpose of data collection, b) verify the eligibility of the schools for the intervention and data, and c) obtain student lists and contact information of School Based Management Committees (SBMCs) and Center Based Management Committee (CBMC) members and head teachers, NFLC facilitators and CSOs in order to facilitate communication with parents during evaluation’s field work. The contractor should use a qualitative approach to gain insight into the evaluation questions and accomplishment of the activity performance and achievement of deliverables. The methodology should consist of interviews and focus group discussions with teachers, instructors, administrators, school directors, local officials, communities, parents and/or parent teacher associations, as well as Ministry of Education officials and members of CSOs in Bauchi and Sokoto states, backed up by a thorough desk study of relevant documentation and performance indicators. Key approaches that would be used to collect and analyze data for the evaluation would be as follows: Review of Background Materials: Activity documents relevant to the evaluation for review and analysis, including activity designs, scopes of work, annual and quarterly reports, annual work plans, technical and training materials. Data Analysis Methods USAID/Nigeria expects the evaluators to propose an outline of this evaluation’s data analysis plans which will be reviewed and approved by the Mission’s education and program teams. It is expected that this plan will directly address each evaluation question with specific methods for collecting and for analyzing the data that will be used to answer it. Also, it is expected that all data reported in this evaluation, including outcomes and input data that illustrates the degree to which reading and access outcomes are impacted 58 and improved in the focal sites, and as a result of the Initiative’s system strengthening, must be reported on a sex-disaggregated basis. 4. EVALUATION PRODUCTS Expected Deliverables Required Deliverables Deadlines (contingent upon another prior deliverable or deadline) To Whom Remarks 1 Work Plan including the Methodology Plan No later than 6 days of work. (During the Technical Planning Meeting prior to implementation). To be submitted to the COR at USAID/Nigeria for approval A detailed work plan which will include the methodologies to be used in the evaluation 2 Present Preliminary Findings Midway during the data collection process USAID for preliminary comments A PPT with preliminary findings to provide some insights to USAID 3 Debriefing with USAID An outbrief for USAID before the evaluation team’s departure from the country (This should provide findings at length for USAID). USAID/Nigeria Presentation of major findings of the evaluation through a PPT presentation and will include a discussion of achievements and issues as well as any recommendations for possible modifications to activity approaches, results, or activities will consider and revise the draft report accordingly, as appropriate. 5 First Draft Evaluation Report USAID/Nigeria who will provide comments within two weeks of submission. Draft written report of the findings and recommendations; clearly describing findings, conclusions, and recommendations. 5 Final Report No later than five days after USAID/Nigeria provides written comments on the ET’s draft evaluation report USAID/Nigeria Report to incorporate the team responses to Mission comments and suggestions. Report format to include: - executive summary table of contents methodology, findings, and recommendations. Report will be disseminated within USAID. 59 A second version of this report excluding any potentially procurement￾sensitive information will be submitted (also electronically, in English) for dissemination among implementing partners and stakeholders. Reporting Guidelines USAID’s evaluation policy requires that all evaluation SOWs include USAID’s Criteria to Ensure the Quality of the Evaluation Report (see USAID Evaluation Policy Appendix I). In addition to adhering to the requirement criteria stipulated by the ADS, (appendix 1), it is expected that the format for this evaluation report will be as follows: 1. Executive Summary: To include concise statement the most salient findings, recommendations, and proposed next steps (2-4 pp); 2. Table of Contents (1 pp); 3. Introduction: purpose, audience, and synopsis of task (1 pp); 4. Background: brief overview of the NEI+ activity, USAID activity strategy and activities implemented in response to the problem, brief description of the implementing partner/s, purpose of the evaluation (2–3 pp); 5. Methodology: describe evaluation methods, including constraints and gaps (1 pp); 6. Findings/Conclusions/Recommendations: for each objective area; and also, include data quality and reporting system that should present verification of spot checks, issues, and outcome (17–20 pp); 7. Issues: provide a list of key technical and/or administrative, if any (1–2 pp); 8. Future Directions (2–3 pp); 9. References (including bibliographical documentation, meetings, interviews and focus group discussions); 10. Annexes must include: 1) The scope of work; and 2) Data collection and analysis tools such as questionnaires, checklists, survey instruments, and discussion guides; 3) Quantitative data collected by the evaluation should be provided in an electronic file in easily readable format and organized and fully documented for use by those not fully familiar with the project or the evaluation. 4) Other materials that document the sources of information, evaluation methods, schedules, interview lists and tables. These materials should be succinct, pertinent and readable; 5) Written disclosures of conflict of interest. These are to be submitted with the proposal. 6) A “statement of differences” may be included as an annex if there are significant unresolved difference of opinion by USAID, implementers, and/or members of the evaluation team. The final version of the evaluation report will be submitted to USAID/Nigeria in hard copy as well as electronically. The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report should not exceed 30 pages, excluding references and annexes. 60 Report Delivery: The evaluation team leader shall incorporate USAID’s comments and submit the final report to USAID in electronic format (Microsoft Word) as well as printed and bound copies (five copies in English) no later than six working days of the receipt of the comments. The evaluation team leader shall submit one either electronic or hard copy to the Development Experience Clearinghouse at http://dec.usaid.gov or M/CIO/KM, RRB M01, USAID, Washington DC 20523. 5. TEAM COMPOSTITION As required by the ADS, at least one member of the evaluation team (possibly the team leader) must be an evaluation specialist and must possess significant experience in designing evaluations and a strong understanding of data collection and analysis methodologies. Also, the team leader is further expected to provide samples of past evaluation reports produced under his/her direction. In addition to the team leader, the evaluation team will consist of 2 - 4 other technical experts who should consist of expatriate and local consultants. Personnel Qualification Team Leader/Senior Evaluation Expert – S/he should have a Master’s degree in education, substantial international development, or an applicable field. S/he should have at least 5 years of senior level experience in managing evaluative activities in basic education program in developing countries including Sub Saharan Africa. S/he will work together with team of experts comprised of Education/Conflict Expert (international), local expert(s) and local logistics assistant (optional). S/he should have competence in assessing priorities and in managing a variety of activities in a time-sensitive environment, and in meeting deadlines with attention to detail and quality, as well as being strategic thinker with interpersonal skills and managerial, coordination, and organizational skills. S/he will display cultural sensitivity in designing and implementing an evaluation in the USAID context. Education Sector Expert (International) – The person should possess considerable international working experience in USAID monitoring, and evaluation processes. A Master’s degree in education, international development or related field relevant to the broad areas of education is required. S/he should have strong technical expertise in monitoring and evaluation methodology, data collection methods, including qualitative and quantitative data; extensive expertise in working with a variety of internal and external stakeholders in a cross-cultural context, in planning, managing and using evaluation report findings, conclusions and recommendations. This Education Expert should have demonstrated experience in mother￾tongue based Early Grade Reading interventions, knowledge of current research in this field and have worked on basic education reform or systems-strengthening activities. Professional English￾language proficiency (both written and oral) is required. Education Sector Expert (2 Local Experts) – The persons should possess substantial working experience from the field of strengthening the systems and processes for service delivery in the basic education sector. A Master’s degree in education, development studies or related field relevant to the broad areas of education is required. One of the two local consultants has worked with state or local government in basic education. They should have a good understanding of developing and implementing assessments and basic education studies. Professional English￾language proficiency (both written and oral) is required. They should have knowledge and comprehensive understanding of the education sector with particular expertise in Northern 61 Nigeria. Previous experience in evaluating USAID-supported programs and proficiency in Hausa language would be an advantage. 11. PERIOD OF PERFORMANCE: The period of performance of this contract shall be 67 days (September 25, 2017-November 30, 2017, assuming 6-day workweeks). The Contractor shall propose a realistic level of effort in the performance of this Scope of work. 12. EVALUATION MANAGEMENT Logistics USAID/Nigeria will provide overall direction to the evaluation team, identify key documents, and assist in facilitating the evaluation work plan. USAID/Nigeria will also assist in arranging meetings with key stakeholders identified by the Team in consultation with the Mission prior to the initiation of this field work. The contractor is responsible for providing all personnel and materials as per their need for site visits around Abuja. The evaluation team is expected to be responsible for arranging other meetings as identified during the course of this evaluation and but advising USAID/Nigeria prior to each of those meetings. USAID/Nigeria personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. Scheduling The contractor shall provide an activity schedule before the outset of the evaluation reflecting a realistic period of performance near to the prescribed period. The activity schedule with timeline shall be specified in a timetable. USAID/Nigeria will be responsible for information to the evaluation team on any scheduling restraints that could affect the evaluation. Key Documents • Early Grade Reading Assessment Baseline Report, 2017 • Community Education and Conflict Assessment/ Gender Analysis (CECA/GA) • Community Mobilization and Communications strategy reports • Nigeria Education Data Survey (NEDS) 2015 • Institutional Capacity Assessment reports • Education Section Project Appraisal Document • Third State Selection Report • School/Center Safety Standards Implementation Guidance report • Monitoring and evaluation reports, studies, and activity information • State of Education Accounts reports 62 • School/Centre Safety Standards (S/CSS) and Benchmarks • Performance Management Plan • Quarterly reports • Annual Work plans • Short term consultants’ reports • The Memorandum of Understanding between the USAID and the State Government of Sokoto and Bauchi • Other relevant technical materials are available online at Development Experience Clearinghouse (https://dec.usaid.gov/). Item Period of Performance Number of Days Team Leader/ Int. Evaluation Expert Reading Expert Local Evaluator Local Evaluator Review background documents & preparation work Conduct remote interviews and discussions with activity staff; ensure that activity staff provide initial contact lists of beneficiaries, GoN officials, other donors, and other USAID activity representatives as needed. Most of this work will be done through emails and Skype. The team will work though USAID and Initiative’s COP to set as many meetings and interviews as possible prior to arrival in Nigeria. Preparation and submission of draft workplan & draft data collection tools (protocols) Oct. 1-5 5 4 4 4 Workplan/Protocols submitted to USAID for review Oct. 5 1 1 63 Item Period of Performance Number of Days Team Leader/ Int. Evaluation Expert Reading Expert Local Evaluator Local Evaluator International travel to Abuja, Nigeria Oct 7 1 1 In-brief / Team planning meetings with USAID Oct. 9 1 1 1 1 In-brief with implementing partner and education officials in Abuja; Oct. 10-11 2 2 2 2 Sub-teams travel • Team A: Sokoto • Team B: Bauchi Oct. 12 1 1 1 1 Morning in-brief with State Activity Team Leads and staff; finalization of local site visit schedule Oct. 13 1 1 1 1 Sub-team review meetings and initial internal write-ups; coordination with team lead Oct. 14 1 1 1 1 Data collection and observations in schools / NFE centers; visits to LGEA offices; Oct. 16-19 4 4 4 4 Travel to Abjua Oct. 20 1 1 1 1 Preparation of the draft evaluation report, preliminary findings/PPT Oct. 21-24 4 3 3 3 Submission of presentation to TLP; continuation of report draft Oct. 25 1 1 1 1 TLP submits presentation to USAID; evaluators continue report draft Oct. 26 1 1 1 1 64 Item Period of Performance Number of Days Team Leader/ Int. Evaluation Expert Reading Expert Local Evaluator Local Evaluator Presentation of preliminary findings, exit brief and submission of report draft to TLP Oct. 27 1 1 1 1 International travel Oct. 28 1 1 USAID provides comments to draft Nov. 3 Incorporation of USAID comments to the draft report Nov. 4 - 6 3 3 1 1 Submission of draft to TLP Nov. 6 Submission of draft evaluation to USAID Nov. 10 1 1 USAID reviews and comments on final draft evaluation report Nov. 17 Team addresses USAID comments and finalizes the report Nov. 19 - 26 5 5 3 3 Report to TLP Nov. 27 1 1 Report to USAID Nov. 30 1 1 TOTAL LOE 37 35 25 25 Timeline Task Accomplishment Duration Pre Field￾Work Obtain key documents, make key contacts, and plan for interviews and discussions in Nigeria with activity staff, beneficiaries, GoN officials, other donors, and other USAID activity reps as needed. Most of this work will be done through emails and Skype. The team will work though USAID and Initiative’s COP to set as many 5 days 65 meetings and interviews as possible prior to arrival in Nigeria. A part-time local hire can be brought on board to assist with this process. Week 1 In-Brief and Planning This period will consist of meetings with USAID, the staff of the Initiative, gathering and reviewing data not already available, and solidifying plans for visits to states and focal LGEAs. In the latter part of this week the interview process with beneficiaries and others will begin including Federal agencies and education sector donors. 3 days Week 2 Field Work The focus of this entire week will be on interviews and discussions with beneficiaries, donors, government officials, representatives of related USAID activities, participating colleges of education and others who work with or have been impacted by the Initiative. Team members will visit and assess activities in at least six LGEAs that have been involved in Initiative activities. Those selected will vary by key qualities like geographical location (rural/urban) and such related. As time allows, the team will begin preparing the first few sections of the final report on the background, setting, and previous evaluative efforts related to the set of the Initiative. 7 days Week 3 Final Interviews, Draft Report, and De-brief Any remaining interviews will be completed. Follow-up meetings to discuss questions arising from the interviews and to clarify and remaining issues will be held with the implementation teams of Initiative, analyze data and findings; draft report for USAID comment and De-brief USAID. 7 days Post Field￾Work Finalize draft final report and submit to USAID/Nigeria no later than two weeks following receipt of final comments from the Mission. 5 days 8. BUDGET To be determined 9. SECURITY REQUIREMENTS This assignment is designated as classified in accordance with ADS Chapter 567 “Classified Contract Security and Contractor Personnel Security Program” and FAR Subpart 4.4 “Safeguarding Classified Information within Industry”. As such, it is subject to the requirements of these regulations. However, only the Team Leader, working under this contract will require access to classified national security information/and/or to areas under the control of USAID deemed “Restricted” by USAID’s Office of Security. State level of clearance for third-party evaluator will be provided if applicable. USAID Evaluation Policy - Appendix 1 Criteria to Ensure the Quality of the Evaluation Report 66 • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the activity, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 67 ANNEX III: MID-TERM EVALUATION WORK PLAN 68 DRAFT MID-TERM EVALUATION WORK PLAN NORTHERN EDUCATION INITIATIVE PLUS Submitted October 5, 2017 (Revised) DISCLAIMER: This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Juanita Campos, Janet K. Orr, Dr. Benedicta Agusiobo, Hadiza Sheetima and Samuel Gyang through DevTech Systems, Inc. NORTHERN EDUCATION INITIATIVE PLUS FOR USAID/NIGERIA 69 This page is intentionally left blank. 70 CONTENTS ANNEXES 46 ANNEX I: LIST OF DOCUMENTS REVIEWED 47 ANNEX II: EVALUATION SCOPE OF WORK FOR NORTHERN EDUCATION INITIATIVE PLUS (NEI+) 50 1. BACKGROUND INFORMATION 50 2. EVALUATION RATIONALE 55 3. EVALUATION DESIGN AND METHODOLOGY 57 4. EVALUATION PRODUCTS 58 ANNEX III: MID-TERM EVALUATION WORK PLAN 67 BACKGROUND INFORMATION 71 MID-TERM EVALUATION RATIONALE 73 DATA COLLECTION 75 DATA SOURCES 77 DATA ANALYSIS 78 SAMPLING 78 LIMITATIONS 79 EVALUATION TEAM COMPOSTITION 79 LOGISTICS 79 REVISED EVALUATION TIMELINE: OCT 2 – NOV 30, 2017 82 ANNEX IV: PROTOCOLS AND QUESTIONNAIRES 85 ANNEX V: NEI+ MID-TERM EVALUATION INTERVIEW PARTICIPANTS 107 ANNEX VI: DISCLOSURE/CONFLICT OF INTEREST FORMS 125 71 BACKGROUND INFORMATION This work plan has been developed for the Mid-Term Evaluation of the Northern Education Initiative PLUS (NEI+) project which is funded by the United States Agency for International Development (USAID). The purpose of the five-year project is to strengthen the ability of the states of Bauchi and Sokoto to provide access to quality education – especially for girls and out-of-school children (OOSC) – and improve children’s reading skills for more than 2 million school-aged children and youth. Consistent with the USAID Forward strategy, this project focuses on building programmatic ownership among federal, state and Local Government Education Authorities (LGEAs), as well as increasing their commitment to quality early grade reading (EGR) instruction and increased access. In 2016, an Early Grade Reading Assessment (EGRA) baseline study, in Hausa and English and in grades two and three were conducted in Bauchi and Sokoto states. The results indicate that children lack foundational reading skills in Hausa, and those children in Primary 3 had not yet acquired them in English. The majority of students scored 0, meaning they could not provide a single correct response, on almost every subtask in the assessment. The large percentage of zero scores of course rendered the mean scores across subtasks very low, with almost no children read with 80% comprehension. Note that the use of early grade reading strategies had yet to occur, rather children were taught language (Hausa and English) skills but not reading as a subject. As a result, it is understandable that an assessment of reading ability result in low scores. Program Activity Objectives USAID’s development objective for NEI+ is “A healthier, more educated population in targeted states”. Given the prevailing context of Nigeria and the state of lower primary education in two northeastern states, Bauchi and Sokoto, USAID through the Northern Education NEI+ Plus, aims to strengthen capacity of “systems” (governmental and non-governmental) to manage themselves: both administratively and financially. Also, to orient the functioning of government processes for the flow of sector information and resources with the specific outcomes of increasing equitable access and measurable achievement in early grade reading. The goal of strengthening the country’s financial and administrative systems is designed to increase improved reading outcomes for a vulnerable sector of primary-aged school children. Expected Outcomes The two fundamental outcomes of the NEI+ activity are that: (1) all children in target LGEAs access learning (via formal school, Non Formal Learning Center (NFLC), and other instructional environments; and (2) reading performance improves for grades one to three for children in all learning environments where the NEI+ works. To achieve these two outcomes, LGEA, State Universal Basic Education Boards (SUBEB), and State Ministries of Education (SMoE) work in partnership with the NEI+ to ensure community mobilization for reading and access, improve data management targeting the expansion of access and improved reading performance. The program takes a holistic approach to systems strengthening in the two states of Sokoto and Bauchi. Systems strengthening means supporting the Nigerian Education Ministries at all levels of the system from the national down to community, school, teacher and student levels is ongoing and at the forefront of our approaches and the spirit of implementation. To improve the functioning of state and local government systems towards reading outcomes, the NEI+ is will build the capacity of LGEAs to train and monitor formal school teachers in early grade reading and to procure and distribute learning materials for early grade reading for local and NFLC contexts, as well 72 as learning materials for basic numeracy and life skills for non-formal learning environments. In terms of policy development, the NEI+ is intended to support states to develop and implement policies that support increased equitable access and improved reading performance, and to update those policies based on evidence provided through the NEI+. Furthermore, LGEAs will collaborate with partner CSOs to provide training for teachers in all school contexts on students’ social/emotional wellbeing and conflict resolution. With the support of the NEI+, states and LGEAs will possess and use research-based, criterion-referenced monitoring tools, classroom and student assessments that enable measurement of progress in expanding access, reducing risk, and improving early grade reading instruction. NEI+ ACTIVITY RESULTS FRAMEWORK Approach and Implementation As expected by USAID, the Initiative is gradually applying and scaling-up evidence-based practices and approaches that have been proven effective in contexts like that of northern Nigeria. The Initiative currently plans for and implements the following approaches during activity implementation: (QUESTION #3 of 5 key questions) Strengthening systems through embedded LGEA and state teams: By embedding the Initiative’s staff in focal states and local level education units, the contractor addresses priorities for human and resource management for improved reading outcomes. The contractor has developed and implemented tools and metrics for rigorous monitoring of capacity transfer in improving reading. (Does this relate to QUESTION #4?) Reducing risks related to learning: Sexual violence, psychosocial difficulties and physical violence are highly prevalent in Northern Nigeria. All activities both formal and non-formal work with local government and communities to help schools meet risk reduction indicators that address school-based violence, gender bias, ethnic bias, and promote social cohesion, equity and conflict resolution/resolution. (QUESTION #2) Integrating a conflict-sensitive approach: Education can help promote social cohesion, contribute to identity formation, build peace, and bridge the gap between humanitarian assistance and sustainable development. However, education can also undermine these 73 MID-TERM EVALUATION RATIONALE Purpose and Objectives The purpose of this contract is to conduct a mid-term evaluation of The Northern Education Initiative Plus (NEI+), a five-year program funded by USAID/Nigeria. The period of performance is October 26, 2015 to October 25, 2020. The objective of this evaluation is twofold: • Assess the extent to which the activity is on track to meeting its key objectives; and, • Identify promising practices, unmet needs, or unintended consequences from implementation of the activity. The team of international and local evaluators will convene by Oct 3rd and data collection will begin after this work plan and data collection tools (protocols) are approved by USAID on Oct 10th. During the second week in country, the evaluation team will conduct interviews with key informants at the national level including USAID staff, project implementers and Ministry stakeholders involved in the program. Federal level interviews will provide the team with a chance to gather relevant information on the overall activities of the program, obtain data on perceived systems strengthening, as well as fine-tune logistics for fieldwork. On Oct 14 – 15, sub-teams travel to Sokoto and Bauchi for in-briefs with State Activity Team Leads, SUBEB, SM0E and LGEA staff. School visits, head teacher, School Management Committees and Community Based Organizations will be held October 16 – 20. Preliminary findings will be presented to stakeholders in the field, where a participatory activity will be designed to generate input and recommendations for further refinement and increase the accuracy of the evaluation report. Analysis of the findings will be synthesized into a draft report. Final draft of the report will be submitted to USAID on November 17th for comment. Those comments will be reviewed and incorporated into the final report by November 30th, 2017. Evaluation findings, conclusions and recommendations will be utilized by USAID and its contractors to make decisions for possible modification of the activity for its duration and/or for follow-on activities. Critical Assumptions The overarching development hypothesis for the program is: If state and local education actors can make education more child-friendly, relevant, flexible, and evidence-based, then greater numbers of children will have access to instruction that improves their reading and life skills. The current crisis is acute and highly destructive, and is rooted in deep issues that have been constant drivers of violence and exclusion in Nigeria and particularly in the North. While the assumption that if such conditions did not exist, outcomes would be more achievable, persistent conflict and insecurity and its impacts on education can affect the ability of programs operating in such an environment to fully reach expected outcomes. The implementing contractor, Creative Associates International, Inc. and its partners are beginning their third year of activity. Results of this evaluation may inform project activities during years four and five. 74 Audience and Intended users The audience of the evaluation report will be USAID Mission, specifically the Education Team, the implementing partner and key officials from the federal, state and local government level. An executive summary with recommendations for program improvement to strengthen systems at the state and local level will focus on quality of education delivery. With the hope of scaling up best practices so that educators can share the activity’s strengths and weaknesses and also use the outcome of the evaluation to serve as a learning platform. Key Evaluation Questions The following questions cover the range of issues highlighted in the evaluation approach which the evaluation will address. 13. To what extent have international evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? • How well are these strategies adapted to local contexts? What improvements could be made? 14. To what extent has NEI+ integrated a conflict-sensitive approach to education? • What measures has NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? • Are the needs of underserved groups been met? If so, how well? What could be done better? 15. To what extent have systems strengthening outcomes and milestones been achieved? • To what degree have states and LGEAs assumed financial responsibility for activity implementation? How has the current economic climate affected release and expenditure of budgeted funds? • What could be done differently to better achieve educational systems strengthening and increased state and LGEA financial support? 16. To what extent has the NEI+ research agenda been formulated to address key challenges to effective EGR and access; how will the research results support project implementation and results? 17. In what ways does NEI+ utilize ICT to support project implementation, administration and monitoring and evaluation? Key Partners and Roles The table below lists key partner institutions and their roles. These stakeholders will be instrumental key informants in this evaluation. In addition, other key stakeholders and partners include USAID Education Team, the implementing partner-Creative Associates International, Inc., governments of Bauchi and Sokoto as state represented by relevant agencies like the Adult and Non-Formal Education Agencies (ANFEA), State Agency for Mass Education, SUBEB, Colleges of Education (COE), Religious Leaders, officials from Local Government Education Authorities, CSOs, communities, schools, and stakeholders from Non-Formal Learning Centers. At the first team meeting in-country, it will be determined which evaluation questions can best be addressed by the partners and stakeholders noted in the graph below. 75 Partner Institutions and Roles STAKEHOLDER Role & Responsibility Local Government Education Area (LGEA) • Implements all programming in both reading and access • Plans for and arrive at improved reading and access • Collaborates with NEI+ activity personnel and CSOs to achieve improvements in reading and access State Universal Basic Education Boards (SUBEB) • Coordinates the development of statewide policies, standards, frameworks, and monitoring tools for both reading and access • Provides financing and logistical support to LGEAs for materials procurement, teacher training, monitoring, and testing State ministries of education State Ministry of Education (SMOE) • Approves final copies of statewide policies, standards, frameworks, and monitoring tools for both reading and access for use throughout the state • Reports to the Federal Ministry of Education on the progress in reading and access in the State Agencies for Mass Education (or the equivalents) and state ministries of religious affairs • Leads the development and approval of appropriate materials for non-formal learning schools, and for non-formal learning centers Nigerian Educational Research and Development Council (NERDC) • Ensure conformity of teaching and learning materials with national primary school curriculum • Align the introduction of teaching and learning materials for primary grades into pre-service coursework designed to prepare new primary teachers National Council for Colleges of Education (NCCE) • Approves introduction of teaching and learning materials for primary grades into pre-service coursework designed to prepare new primary teachers Universal Basic Education Commission (UBEC) • Ensures availability the allocation of funds for basic education Colleges of Education (COE) • Pre-service training and implementer of recently developed pre-service reading instructional program • Federal Ministry of Education (FME) • Endorses decentralized efforts to improve reading and access • Incorporate best practices into national-level plans United Kingdom Department for International Development (DFID), British Council and UNICEF. • Development partners • Donors • Tangential program implementers and collaborators (GEP3) DATA COLLECTION 76 The evaluation will use a mixed-method approach in which more than one method for data collection will be used to answer the five (5) key questions. The methodological approach will be predominantly qualitative as prescribed in the SOW and will draw upon a range of qualitative data collection methods including open-ended questionnaires, semi-structured interviews, focus group discussions, structured school/classroom observation, reviews of key documents and government and other relevant organizational data. “Snowball sampling” may be utilized when an informant recommends other stakeholders who may provide additional relevant data. Both quantitative and qualitative data will be collected and analyzed. Triangulating field data with data from program reports, previous evaluations will guide the identification of trends, findings, conclusions and recommendations in response to the evaluation questions. Triangulation will also enable the evaluators to identify gaps, triggers and missed opportunities in project planning. The mix-method approach aims to provide a holistic understanding of the project’s mid-course status. Standard data collection tools and data analysis templates will be developed to ensure quality control across collected data. Inter-rater reliability exercises will be conducted among evaluators to assure data collection consistency. Initially, the team will conduct interviews at the Federal level while in Abuja. The team will confirm proposed site visits and make necessary arrangements for permission to access school sites, civil society groups, key informants and other relevant stakeholders. During field work the team will divide into two teams, Team A and Team B, to cover both states. Team members will be responsible for collecting data on specific components in parallel with the 5 key evaluation questions. The safety and security of all team members will be a determining factor in the team’s mobility during the fieldwork phase in Bauchi and Sokoto. If needed and possible, team members will determine in collaboration with officials in Bauchi and Sokoto how best to collect evaluation data. Fieldwork will be a rapid rural appraisal approach as well as being highly participatory for the purpose of engaging key stakeholders and beneficiaries and to promote learning. Interactive processes can facilitate reflection and can result in mutual learning. Participatory techniques will be applied when relevant and achievable. The following chart illustrates data collection methods and sources aligned with the 5 key evaluation questions. 77 DATA SOURCES Key Evaluation Question Data Collection Method Data Collection Source – #1 - To what extent have international, evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? How well are these strategies adapted to local contexts? What improvements could be made? • School & classroom observation • Semi-structured interview • Key informant interview LGEA, SUBEB, NFLCs/directors, school directors, trained teachers, relevant state education institutions. GEP reps or gender-focused advocates/groups, relevant TWG members #2 - To what extent has the NEI+ integrated a conflict-sensitive approach to education? What measures has the NEI+ taken to reduce risks associated with violence and conflict? How effective have these measures been? Are the needs of underserved groups been met? If so, how well? What could be done better? • Structured observation • Focus group discussions • Semi-structured interviews 1-to-1 or group Gender-related NGOs/advocates, relevant TGW dealing with the CS approach, SBMCs, UNICEF, NFLCs, LGEA or LGA, Education Crisis Response Project personnel, CBOs, CBMC, CSOs, Bauchi State Agency for Nomadic Education & Resettlement, BASANER/ parents/community members or PTAs, GEP reps #3 - To what extent have systems strengthening outcomes and milestones been achieved? To what degree have states and LGEAs assumed financial responsibility for activity implementation? How has the current economic climate affected release and expenditure of budgeted funds? What could be done differently to better achieve educational systems strengthening and increased state and LGEA financial support? • Semi-structured interview both 1-to-1 or group • In-depth interviews • Document review Relevant TWG members, heads of finance and account for SUBEB, BASAME, LGEAs, EGRAs strongest advocates at federal and state levels #4 - To what extent has the NEI+ research agenda been formulated to address key challenges to effective EGR and access; how will the research results support project implementation and results? • In-depth interviews • Semi-structured interview 1-to-1 or group LGEA, SUBEB, NFLCs/directors, school directors, trained teachers, relevant state education institutions, relevant TWG members #5 - In what ways does NEI+ utilize ICT to support project implementation, administration and M&E? • Document review, in￾depth interview with key ICT personnel or TWGs ICT and EMIS and TMIS units at federal and state levels, key personnel and/or relevant TWG groups/members, SUBEB, LGEA, BASAME 78 DATA ANALYSIS Standard data analysis tools will be developed. Raw data collected from key informant interviews and focus group discussion will initially be transcribed in an instrument template on a daily basis. At the end of each day, the team will begin to consolidate findings by key questions and themes. At the end of fieldwork the questionnaires with a large number of respondents will be entered into a central Excel spreadsheet and coded, consolidated and calculated to determine frequencies and percentages of responses. Descriptive data collected from school site visits and singular interviews will be recorded in a Word document where key findings will be consolidated into themes, trends or triggers. The findings from document review and analysis of the student-teacher classroom interaction, for instance, will be recorded in a table organized by key question and data source. This will be done prior, during and after fieldwork, as new insights are gleaned (i.e., snowballing) from document review post fieldwork and data can be triangulated. Each team member will cross-check the questions and responses for accuracy and consistency. The team member responsible for analyzing the designated question will review the data and draw the key findings, conclusions and recommendations. All data reported in this evaluation, including outcomes and input data that illustrates the degree to which reading and access outcomes are impacted and improved in the focal sites, and as a result of the NEI+ system strengthening, will be reported on a sex-disaggregated basis. A “side bar report” on outlying issues may be provided if issues outside the scope of this report warrant the information. SAMPLING The evaluation plan is to visit 6-8 school sites including both formal schools and non-formal learning centers in Bauchi and Sokoto during 4 days of fieldwork. Teams of two (one international and 1 national) will visit reading classes first – a local evaluator will visit 2nd grade classes and international member will visit a 3rd grade class. Up to 4 classes at each school will be the team’s goal. Each classroom visit will be no less than ½ hour. School headmasters will be interviewed and focus group discussion with parents/PTA, village chief, other local officials or key groups that support school activities will be held in the afternoon. School typically begins at 7am and ends at 1pm; times to be confirmed for each school site. Focus group discussions with parents or PTAs will take place after school when parents typically come to pick up children. Schools will be selected using a purposeful sampling approach focusing on the variables listed below and after consultation with DEV TECH COP, USAID EDU, and other relevant partners. Some of the following variables listed will inform the selection of LGEA sites and schools: • Urban and Rural • Participation in EGRA (the assessment) in 2016 • Formal, Non-Formal Learning Centers, Adolescent Girls Learning Center and Youth Learning Centers • Length of time participating in NEI+ Once schools are selected, the evaluation team will work with SUBEB and LGEAs to obtain approval for scheduling visits to schools and offices. CSOs will be contacted in order to alert them of the impending data collection activities at NFLCs. School Based Management Committees (SBMCs) and Center Based Management Committee (CBMC) members may be alerted by head teachers, NFLC facilitators and CSOs in order to facilitate communication with parents during evaluation’s field work. 79 LIMITATIONS Evaluations that are primarily qualitative can generate a great deal of rich, contextual data. The sheer volume of such data invites a tendency to underuse, or even not to use, all the data collected during analysis. Insufficient time to analyze qualitative findings is a potential weakness for in-depth analysis. (Additional analysis time could reduce this limitation.) Secondly, an emphasis on quantifying qualitative data often excludes “critical incidents/cases” or “outliers”. In anthropological and sociological practice, critical incidents/cases outliers” are given full credence as evidence of nascent or emerging trends. Efforts will be made to maximize the use of rich qualitative data. The scope of NEI+ is broad, deep plus circumstances including security and the availability of school and community stakeholders can potentially compromise the evaluation plan. Secondarily, the lives of persons living in poverty and children considered to be the “most vulnerable” might present unanticipated limitations, which can only be addressed while in the field. In addition, field visits are occurring during the rainy season, which may make some areas inaccessible by road. Access to contact information in the field can be a challenge as there is not a local phone book listing the phone numbers for formal schools and NFLC contacts are every changing. SUBEB and LGEAs should be able to provide contacts for the formal schools and CSOs may have updated information on the NFLCs which seem to operate for just 9 months at a time. Contact in rural areas is especially challenging as mobile phone coverage is limited and contact is reliant on the prosperity of the mobile phone owner. The absence of accurate data available to the evaluators regarding assessment and reporting at all levels from policy to classroom stands in the way of a holistic understanding of the current, mid-term status. Practicalities such as teacher absenteeism or students can effect data collection and analysis. Security The security and safety of all team members and informants will be of utmost consideration for the duration of this evaluation task. EVALUATION TEAM COMPOSTITION The team is comprised of four personnel: (1) Dr. Jennie Campos, Team Leader/Senior Evaluation Expert, (2) Janet K. Orr, Education Sector Expert/International, and (3) Dr. Benedicta Agusiobo and Hadiza Shettima, Local Education Sector Experts and (4) one logistics assistant. The team will conduct interviews at the federal level and split into two teams for fieldwork visits. Team members will be assigned specific research questions to focus data collection efforts on and to ensure a systematic approach to data collection and analysis. They might also be responsible for collecting data on other components. Therefore, standard data collection tools and data analysis templates will be developed to ensure quality control across the team. Training of team members on the instruments will occur before venturing into the field to assure inter-rater reliability. LOGISTICS USAID/Nigeria will provide overall direction to the evaluation team, identify key documents, and assist in facilitating the evaluation work plan. USAID/Nigeria and Creative Associates International, Inc. will also assist in identifying key stakeholders for the Team in consultation prior to the initiation of this field work. The contractor is responsible for providing all personnel and materials as per their need for site visits 80 around Abuja. The evaluation team is expected to arrange other meetings as identified during the course of this evaluation and advising USAID/Nigeria. 81 82 REVISED EVALUATION TIMELINE: OCT 2 – NOV 30, 2017 Timeline Task Accomplishment Duration October 2-6 Pre Field-Work Obtain key documents, make key contacts, and plan for interviews and discussions in Nigeria with activity staff, beneficiaries, GON officials, other donors, and other USAID activity reps as needed. Most of this work will be done through emails and Skype. The team will work though USAID and NEI+’s COP to set as many meetings and interviews as possible prior to arrival in Nigeria. A part-time local hire can be brought on board to assist with this process. 5 days October 7-10 In-Brief and Planning This period will consist of meetings with USAID, the staff of the NEI+, gathering and reviewing data not already available, and solidifying plans for visits to states and focal LGEAs. In the latter part of this week the interview process with beneficiaries and others will begin including Federal agencies and education sector donors. 3 days October 16 - 19 Field Work The focus of this entire week will be on interviews and discussions with beneficiaries, donors, government officials, representatives of related USAID activities, participating colleges of education and others who work with or have been impacted by the NEI+. Team members will visit and assess activities in at least six LGEAs that have been involved in NEI+ activities. Those selected will vary by key qualities like geographical location (rural/urban) and such related. As time allows, the team will begin preparing the first few sections of the final report on the background, setting, and previous evaluative efforts related to the set of the NEI+. 7 days October 20- 27 Final Interviews, Draft Report, and De-brief Any remaining interviews will be completed. Follow-up meetings to discuss questions arising from the interviews and to clarify and remaining issues will be held with the implementation teams of NEI+, analyze data and findings; draft report for USAID comment and De-brief USAID. 7 days October 28 – November 30 Post Field￾Work Finalize draft final report and submit to USAID/Nigeria no later than two weeks following receipt of final comments from the Mission. days 83 Required Deliverables Deadlines (contingent upon prior deliverable or deadline) Deadline To Whom Remarks 1 Draft Work Plan including the Methodology Plan No later than 6 days of work. (During the Technical Planning Meeting prior to implementation). Oct. 5 To be submitted to the COR at USAID/Nigeria for approval A detailed work plan which will include the methodologies to be used in the evaluation 2 Debriefing with USAID An out brief for USAID before the evaluation team’s departure from the country (This should provide findings at length for USAID). Oct 27, 2017 USAID/Nigeria Presentation of major findings of the evaluation through a PPT presentation will include a discussion of achievements, issues and recommendati ons for possible modifications to activity approaches. 3 First Draft Evaluation Report Oct. 27, 2017 USAID/Nigeria who will provide comments within two weeks of submission. Draft written report of the findings and recommendati ons; clearly describing findings, conclusions, 84 and recommendati ons. 4 Final Report No later than five days after USAID/Nigeria provides written comments on the ET’s draft evaluation report Nov. 30, 2017 USAID/Nigeria Report to team to revise based on responses by Mission personnel. A 2nd version of report excluding potentially procurement￾sensitive information will be submitted (electronically, in English) for dissemination among implementing partners and stakeholders. 85 ANNEX IV: PROTOCOLS AND QUESTIONNAIRES STATE – SUBEB  Executive Secretary  Desk Officer Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Organization & Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE Q 3 A. What is your mandate in the management, allocation and release of education funding? B. What role do you have in the NEI+ project as it pertains to improving the allocation of funds to schools? C. What do States do with the funds? D. How much of the current allocation of funds for basic education has been released for the current year? E. What evidence indicates that release of funds? F. As a result of your participation with NEI+ how has the system that allocates funds to school been strengthened? G. How prepared is this department to sustain allocation of funds on its own? H. Have parts of the system been strengthened more quickly than others? I. Which are they? Why have these parts been strengthened but not others? J. What are the plans to establish a national reading policy that is influenced by EGRA? K. What have you found to be the most challenging aspect of the project? L. How are you addressing this challenge? 86 Q3/SOW – Q3A To what extent have systems strengthening outcomes and milestone been achieved? Q3B To what degree have states and LGEAs assumed financial responsibility for activity implementation? Q3C What has stood in their way if not yet achieved? Q3D What are the obstacles? Q3E How has the current economic climate affected release and expenditure of budgeted funds? Q3F What could be improved to achieve educational systems strengthening and increased state and LGEA financial support? Q5/SOW – A. In what ways does NEI+ utilize ICT to support project implementation, administration and M&E? 87 STATE – SUBEB Technical Working Group (TWG) Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Organization & Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE Q 3 (related to systems strengthening) A. What is your mandate in the management, allocation and release of education funding? B. What role do you have in the NEI+ project as it pertains to strengthening the system that allocates funds to schools? C. What do States do with the funds? D. How much of the current allocation of funds for basic education has been released for the current year? E. What evidence indicates that release of funds? F. As a result of your participation with NEI+ how has the system that allocates funds to school been strengthened? G. How prepared is this office to sustain allocation of funds on its own? H. Which parts of the system have been strengthened more quickly than others? Why have these parts been strengthened but not others? I. What are the plans to establish a national reading policy that is influenced by EGRA? J. What have you found to be the most challenging aspect of the project? K. How are you addressing this challenge? Q3/SOW – 88 A. To what extent have systems strengthening outcomes and milestone been achieved? B. To what degree have states and LGEAs assumed financial responsibility for activity implementation? C. What has stood in their way if not yet achieved? D. What are the obstacles? E. How has the current economic climate affected release and expenditure of budgeted funds? F. What could be improved to achieve educational systems strengthening and increased state and LGEA financial support? Q5/SOW – A. In what ways does NEI+ utilize ICT to support project implementation, administration and M&E? 89 EMIS/Director of Planning, Research and Statistics Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Organization & Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE Q5 – A. In what ways does NEI+ use Information Communication Technology (ICT) to support project implementation, administration and M&E? B. What parts of the ICT system has been strengthened because of NEI+? C. Which parts of the ICT system remains weak and in need of improvement? D. How can these weaknesses be improved? E. What impact does a weak ICT system have on the educational system overall? F. How strong are individual staff members in the use of Excel? G. Are there staff members who have not grasped the NEI+ lessons well? H. What part of NEI+ systems strengthening was difficult for personnel to learn and use? Why? I. What part of NEI+ training was relatively easy for personnel to learn and use? J. How frequently are school data delivered/shared with Head Teachers/NFLC Head Facilitators? K. How strong do you believe ICT system is to be sustainable? 90 Adult Non-Formal Education Association/ANFEA Bauchi State Agency for Mass Education/BESEME Sokoto State Agency for Mass Education/SAME Executive Secretary & Desk Officer Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Organization & Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE Q1/SOW – A. To what extent have international evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? B. In what ways have the strategies been adapted to local contexts? C. What improvements could be made in teacher preparation? Q2/SOW – A. To what extent has the Initiative integrated a conflict-sensitive approach to LF preparation? B. What approach is used in the NFLCs to ensure equality of boys and girls? C. In what ways have the strategies been adapted to local contexts? D. What could be improved? 91 State Ministry of Education (SMoE) Permanent Secretary Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Organization & Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE What is your role and responsibility in the NEI+ project? Q1/SOW- To what extent have international, evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? What improvements could be made? Q2/SOW- To what extent has the Initiative integrated a Do No Harm conflict￾sensitive approach to education? A. What measures has the Initiative taken to reduce risks associated with violence and conflict? B. How effective have these measures been? C. Are the needs of underserved groups been met? D. If so, how well? E. What could be done better? Q3/SOW- To what extent have systems strengthening outcomes and milestones been achieved? A. To what degree have states and LGEAs assumed financial responsibility for activity implementation? B. How has the current economic climate affected release and expenditure of budgeted funds? C. What could be done differently to better achieve educational systems strengthening and increased state and LGEA financial support? 92 Q4/SOW - To what extent has the NEI+ research agenda been formulated to address key challenges to effective EGR and access; A. How will the research results support project implementation and results? Q5/SOW – A. In what ways does NEI+ utilize ICT to support project implementation, administration and M&E? 93 State Ministry of Education (SMoE) State Adult and Mass Education (SAME) Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE A. To what extent have systems strengthening outcomes and milestones been achieved? B. What do you understand by systems strengthening? C. What type of training have you received? D. To what degree have states and LGEAs assumed financial responsibility for activity implementation? E. How has the current economic climate affected release and expenditure of budgeted funds? F. What could be done differently to better achieve educational systems strengthening and increased state and LGEA financial support? G. To what extent has the NEI+ research agenda been formulated to address key challenges to effective EGR and access? H. How will the research results support project implementation and results? I. In what ways does NEI+ utilize ICT to support project implementation, administration and M&E? 94 State Ministry of Education (SMoE) – TWG-Budget Planning Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Position/Title Phone QUESTION RESPONSE Q3 – SOW A. To what extent have systems strengthening outcomes and milestones been achieved? B. To what degree have states and LGEAs assumed financial responsibility for activity implementation? C. How has the current economic climate affected release and expenditure of budgeted funds? D. What could be done differently to better achieve educational systems strengthening and increased state and LGEA financial support? E. What contributions from the State ensure effective EGRA and access to education? Q4 –SOW A. To what extent has the NEI+ research agenda been formulated to address key challenges to effective EGR and access; B. How will the research results support project implementation and results? Q5 – SOW A. In what ways does NEI+ utilize ICT to support project implementation, administration and M&E? 95 College of Education National Commission for Colleges of Education (NCCE) Desk Officer Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE Q1 (related to Q1/SOW) A. What is your major mandate in teacher preparation? B. What has been your role in the NEI+ project? C. What is your major mandate for the teacher preparation with the NEI+ EGRA approach to teaching reading? How effective has it been? D. What contribution has EGRA made to teacher preparation that has made teaching reading effective for children? E. During teacher preparation what areas of the EGRA approach seems to have been easy for student-teachers to understand then apply during teaching practice? F. How are they monitored thereafter? G. During teacher preparation what areas of the EGRA approach seems to have challenged student-teachers to understand then apply during teacher practice? H. How are they monitored thereafter? I. What steps have you taken to ameliorate these challenges? J. There is a critical mass of teachers who have been trained in EGRA to train student-teachers. What do these teacher-trainers do after their experience with NEI+? K. How is this valuable “EGRA” resource cycled back into the educational system/classrooms? 96 L. How is this valuable human resource monitored thereafter? M. How has EGRA instruction influenced teaching reading in the classroom? N. What can be improved? Q1/SOW – A. To what extent have international evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? B. In what ways have the strategies been adapted to local contexts? C. What improvements could be made in teacher preparation? Q2/SOW – A. To what extent has the Initiative integrated a conflict-sensitive approach to teacher preparation? B. In what ways have the strategies been adapted to local contexts? C. What could be done better? Q4/SOW – A. To what extent has the NEI+ research agenda been formulated to address key challenges to effective EGR and access; how will the research results support project implementation and results? 97 Nigerian Educational Research and Development Council (NERDC) Executive Secretary (ABUJA) Date: State: Location/City/Village: Interviewer/Observer Duration Organization/Group Person(s) Interviewed Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. QUESTION RESPONSE Q1/SOW￾A. To what extent have international evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? B. How well are these strategies adapted to local contexts? C. What improvements could be made? Q1 - Related to Q1/SOW A. What is your major mandate in curriculum development? B. What has been your role in the NEI+ project? C. What is your major mandate for the integration of NEI+ EGRA into the national reading curriculum? D. What contribution has EGRA made to the national curriculum that has made teaching reading effective for children? E. What impact has the EGRA-influenced curriculum had on helping students learn to read? F. What are the plans to establish a national reading policy that is influenced by EGRA? G. What have you found to be the most challenging aspect of the project? 98 Nigerian Educational Research Development Council/ NERDC TWG (NEI+ at States) Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Position/Title Phone QUESTION RESPONSE Q1/SOW￾A. To what extent have international evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? B. How are the strategies adapted to local contexts? C. What improvements can be made? Q2/Related to Q1/SOW A. What is your major mandate in curriculum development? B. What role do you play in integrating NEI+ EGRA into the national reading curriculum? C. What contribution has EGRA made to the national curriculum that has produced effective reading teachers? D. What makes EGRA effective? E. What are the plans to establish a national reading policy that is influenced by EGRA? F. What is the most challenging aspect of EGRA’s NEI+? G. How is this challenge addressed? 99 Nigerian Educational Research Development Council/ NERDC Desk Officer Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Position/Title Phone QUESTION RESPONSE Q1/SOW￾A. To what extent have international evidence-based strategies, techniques, and best practices for improving reading outcomes been applied? B. How are the strategies adapted to local contexts? C. What improvements could be made? Q1 - Related to Q1/SOW A. What is your major mandate in curriculum development? B. What has been your role in the NEI+ project? C. What is your major mandate for the integration of NEI+ EGRA into the national reading curriculum? D. What contribution has EGRA made to the national curriculum that has helped teachers teach children to read? E. What are the plans to establish a national reading policy that is influenced by EGRA? F. What is the most challenging aspect of the project? G. How is this challenged addressed? 100 SCHOOL BASED MANAGEMENT BOARD COMMUNITY BASED ORGANIZATIONS Date: State: Location/City/Village: Interviewer/Observer: Duration Organization/Group Person(s) Interviewed: Organization & Position/Title Phone QUESTION RESPONSE Q3 (related) – A. What is the main function of SBMB or CBO? B. How does the community get involved in supporting the school? C. Are there parents who could be more involved in the school but are not? D. What does the SBMB or CBO do to motivate parents to become involved in school activities? E. Are EGRA’s “Mu Karanta” teaching materials being delivered to the hands of teachers/facilitators as intended?  Yes, explain  No, explain F. What changes have you noticed in the community or in homes that tell you children enjoy reading? G. What changes have you seen that tell you teachers enjoy the EGRA approach to teaching reading? H. What changes have occurred in homes that show that EGRA has influenced the family? I. What does the community do to celebrate children’s success at learning to read? J. How does this group support NEI+ EGRA? K. What is done to ensure that learning opportunities are equal for boys and girls? L. Are there children in the community who could attend school but do not? M. How are the needs of children with disabilities included in school? 101 N. How frequently is the school monitored or visited by an SSO/School Support Officer? O. How can the SBMB/CBO ensure that the accomplishments of EGRA will be sustained? P. What can be done to improve the school reading program? 102 Grantees Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Grantee/Implementer/sub￾grantee Person(s) Interviewed: Organization & Position/Title Phone QUESTION RESPONSE A. What is your group’s mandate? B. What is your group’s role in NEI+ EGRA? C. Please describe ways that your group contributes to children learning how to read? D. Please describe ways that your group contributes to supporting teachers teach reading? E. What do you view as the best contribution of NEI+ EGRA? F. What changes have you seen in the community or homes that tell you that children are learning to read and enjoying it? G. What changes have you noticed in the school environment that tell you that children enjoy going to school and are becoming better readers? What improvements do you believe need to be made in the NEI+ EGRA project? 103 School Support Officer/SSO Formal or Area Coordinator/Non-Formal Date: State: Location/City/Village: Interviewer/Observer: Duration: Organization/Group Person(s) Interviewed: Organization & Position/Title Phone Introduction: Thank you for your time today. During this interview, I would like to learn about your view of the NEI+ project. Your opinion is important to us and your responses will remain confidential. SSO/Formal or Area Coordinator/Non-Formal: A. Please describe your role and responsibilities as an SSO. B. How often do you visit schools? C. Please describe what you do during a school visit? D. What types of things keep you from completing school visits? E. Please tell me what you know about NEI+ EGRA program. C. How much time do you SSO spend in EGRA classrooms? D. When you visit an EGRA classroom what impresses you about the teaching style being used? E. What changes have you noticed in children who learn to read in an EGRA classroom? F. In what way does your work/school visit help to improve conditions at the school? G. What needs to change to make your role and responsibilities to the educational system more efficient and effective? 104 CLASSROOM OBSERVATION PROTOCOL Date: ________________________ Observer: _____________________ School Name: _____________________________________ State:  Bauchi  Sokoto Location: Urban  Rural Remote Rural  GPS: N________________________E_____________________ Grade: 2 or 3 Students Present: Boys_____Girls ______Total___ Enrolled: Boys_____Girls _____Total___ Teacher: Male  Female  Start time of observation: ____ End time: ____ Lesson Duration: __________________________ Language of the Lesson: Hausa  English  Switching between languages  Other language used: ________________________ Mu Karanta Teachers Guide  Teacher Stated Reading Lesson Objectives: _____________________________________________________________________________ ___________________________ The following chart includes dimensions that will focus your attention during the observation. Tick descriptors that you observe in the classroom. Use the comments box to add detail about the observation. Space for additional comments follows at the end of the chart. Dimensions Tick what you observe in the classroom Observer Comments The Teacher Lesson Plan/Scheme Teacher’s Guide  Written Plan  Plan on Board  No plan visible  Role in lesson Telling/lecturer  Guiding Practice  Monitoring tasks  Questioning  Teacher Manner Encouraging  Guiding  Engaged  On Task  The Students Student engagement (generally in class) Enthusiastic about learning  Following instruction  Difficulty following instruction  Disinterested or Lost  105 Students grouped for instruction One-on-one with the teacher  Small groups (3-6 students)  Pairs  Whole Class  Classroom Environment Classroom Space: Over-crowded  Crowded  Sufficient space  Roomy  Ventilation Excellent  Good  Average  Poor  Lighting Electric  Good Windows  Ave. Windows  Poor  Walls 4  2  1  0 (outdoors)  Noise Level Productive  Quiet  Disruptive  Equipment/Materials Present Instructional Materials Leveled Text  Mu Karanta  Story Books  Jolly Phonics  Equipment/materials Games/puzzles   Word Cards Learning manipulatives  Class Library  Available Print What type of print do students have eyes on in class? Chalkboard  White board  Book in hand  Teacher’s book  Shared book  Paper  Posters  Exercise book  Computer  Mobile phone  Print content Subject content (non-fiction)  Literature/story (fiction)  Authentic text (newspaper)  Syllables---- Words   106 Reading Instruction Before reading Review of previous lesson  Reads title  Looks at pictures  Predict content  Comprehension strategies observed Uses students’ background  Sequencing  Problem Solving  I do, you do, we do  Reading lesson focus observed Orthographic awareness  Phonological  awareness Morphological awareness  Fluency  Vocabulary  Comprehension￾Sentence  Paragraph  Questioning Predicting  Give Facts  Reorganize information  Evaluating  Create  Post reading tasks Discussion  Reviewing predictions  Applying/using information  Writing  Observer comments: Are instructional strategies learned at NEI+ training evident? YES NO What strategies are used? Are NEI+ strategies adapted by class teachers to the local context which is familiar to students? YES NO Example: Did the teacher miss an instructional opportunity? How could the lesson be improved? Is audiovisual or information communication technology use visible in the classroom? 107 ANNEX V: NEI+ MID-TERM EVALUATION INTERVIEW PARTICIPANTS NEI+ Mid-term Evaluation Interview Participants, Sokoto State # DATE NAME ORGANISATIO NS POSITION GENDE R PHONE NUMBERS 1 17/10/20 17 Dr. Mohd Jabbi K Min. of Secondary & Basic Education Commissioner M 08034515132 2 17/10/20 17 Mohammed Yusuf Gama NEI + Sokoto Access and Fragility Officer M 08066464526 3 17/10/20 17 Mohammed Attahiru Ahmad NEI + STL M 08035378704 4 17/10/20 17 Sanni Ahmad Gwashi NEI + IT Officer M 08066121141 5 19/10/20 17 Shamsu Muhid Achida Model Primary Teacher M 07016167134 6 19/10/20 17 Abdullahi Garba AMPS Teacher M 08064376551 7 19/10/20 17 Alh Tudu AMPS P.T.A Teacher M 09063701977 8 19/10/20 17 Adamu Aliyu AMPS Teacher M 08066682979 9 19/10/20 17 Huloti Salihu AMPS S.B.M.C M 08065630287 10 19/10/20 17 Umar Abubakar AMPS HEAD TEACHER M 08031357135 11 19/10/20 17 Abdullahu Ab LGEA Wurmo Education Secretary M 080108112642 108 12 19/10/20 17 Manya sada Achies LGEA H/Q AUDITOR M 07060644529 13 19/10/20 17 Mukhtar Umar LGEA H/Q DEPUTY EDUC. SEC. M 07030561712 14 19/10/20 17 Ibrahim Ah Sheu LGEA H/Q S/ACC M 080364778773 15 15/10/20 17 Nura Ibrahim NEIPLUS DCOP M 08030722623 16 15/10/20 17 Muhammad Attahiru Ahmad NEIPLUS State Team Leader M 08035378701/0803537 8704 17 15/10/20 17 Muhammed Yusuf Gama NEIPLUS AFO M 08066464526 18 15/10/20 17 Mustapha Aliyu Usaman NEIPLUS M&EO M 07069442447 19 15/10/20 17 Umar Muhammed Illr NEIPLUS SS/EMIS officer M 07030922417 20 15/10/20 17 Innocent Chukwu NEIPLUS Assessment Specialist M 08033919620 22 15/10/20 17 Abdulkadir Usman NEIPLUS Database Officer M 08039616150 23 15/10/20 17 Muhammed Bello Yusufu NEIPLUS Reading Officer M 08065550405 24 15/10/20 17 Zalisau Abubakar Mashan NEIPLUS Teacher Education F 08032209922 25 16/10/20 17 Bello Yusufu Danchadi SUBEB Ex-Chairman M 08035075259 26 16/10/20 17 Fatima Bello SUBEB PM III F 08032876546 27 16/10/20 17 Muhammad u Dango SUBEB P. M I M 080235776609 28 16/10/20 17 Ibrahim N. Muhid SUBEB PRO M 08069061913 109 29 16/10/20 17 Abdulahi Adamu SUBEB DQA M 07034975101 30 16/10/20 17 Mamuda Galadinma SUBEB MEAR TWG M 08036074739 31 16/10/20 17 Garba Yusuf SUBEB EMIS M 08063290117 32 16/10/20 17 Farouk Shehu SUBEB SEC M 08035074273 33 16/10/20 17 Muhammed Attahiru Ahmad NEI PLUS STL M 08035378704 34 16/10/20 17 Sulaimon S. Fulani Ahmad SAME Permanent Sectary M 0903900004 35 16/10/20 17 Abubakar M. Alkamawa SAME Prog. Director M 08062988357 36 16/10/20 17 Mohamed Yusuf Gama NEI PLUS AFO M 08066464526 37 16/10/20 17 Muhammed Tambian Umar SAME DPRS M 08084262577 38 16/10/20 17 Sanusi Ahmad Gwashi NEI PLUS IT Officer M 08066121141 39 16/10/20 17 Mohammed Attahiru Ahmed NEI PLUS STL M 08035378701 40 16/10/20 17 Sulaimon S. Fulani Ahmad SAME Permanent Sectary M 0903900004 41 16/10/20 17 Abubakar M. Alkamawa SAME Prog. Director M 08062988357 42 16/10/20 17 Mohamed Yusuf Gama NEI PLUS AFO M 08066464526 110 43 16/10/20 17 Muhammed Tambian Umar SAME DPRS M 08084262577 44 16/10/20 17 Sanusi Ahmad Gwashi NEI PLUS IT Officer M 08066121141 45 16/10/20 17 Mohammed Attahiru Ahmed NEI PLUS STL M 08035378701 46 17/10/20 17 Dr. Muhammad Wadata Hakimi Shehu Shagari College of Education Provost M 08035964989 47 17/10/20 17 Dr. Abubakar M. B SSCOE SOK D Provost M 08038073961 48 17/10/20 17 Hadiza Salihu Koko SSCOE SOK Dean langs F 08036003447 49 17/10/20 17 Dangaladin ma Wadata SSCOE SOK Focal Person M 08035631365 50 17/10/20 17 Aminu Balarabe Kilqori SSCOE SOK IT Officer M 08130305801 51 17/10/20 17 Faruk M. Jega SSCOE SOK Educator M 08063035776 52 17/10/20 17 Salisu Tukur SSCOE Sokoto HOD (PED) M 08060752308 53 17/10/20 17 Aishatu Jibil Beth SSCO Sokoto Teacher Educator F 08032716849 54 17/10/20 17 Imrana Ibrahim Labbo Dogon Daji Head Teacher M 08032479511 55 17/10/20 17 Muritala Umar Labbo Dogon Daji Teacher M 08069452496 56 17/10/20 17 Nana Garba Labbo Dogon Daji Teacher F 080694524196 57 17/10/20 17 Inno Maduwaki Labbo Dogon Daji Teacher F 08035570804 111 58 17/10/20 17 Mariyam Musa Labbo Dogon Daji Teacher F 07030142339 59 17/10/20 17 Zainab Bello Labbo Dogon Daji Teacher F 08038646426 60 17/10/20 17 Saratu Bello Labbo Dogon Daji Teacher F 08038549172 61 17/10/20 17 Dr. Mohd Jabbi K Min. of Basic Education Hon Commissioner M 08034515132 62 17/10/20 17 Mohammed Yusuf Gama NEI plus Sokoto Access and Fragility Officer M 08066464526 63 17/10/20 17 Mohammed Attahiru Ahmad NEI Plus STL M 08035378704 64 17/10/20 17 Sanni Ahmad Gwashi NEI Plus IT Officer M 08066121141 65 18/10/20 17 Aminu Na’ Allah TBW LGEA Q.A. O M 08035144031 66 18/10/20 17 Bello Maishanu TBW LGEA Head Master M 09036866896 67 18/10/20 17 Marafan Naikada TBW LGEA T/O M 07068103392 68 18/10/20 17 Farouk Labaran TBW LGEA VHead M 08029564634 69 18/10/20 17 Imam Nasiru TBW LGEA Asst. H/Masters M 08074801980 70 18/10/20 17 Rashida Isal TBW LGEA VC F 09063473358 71 18/10/20 17 Yunusa Ibrahim TBW LGEA Teacher EGRA F 08144700612 72 18/10/20 17 Malami Umar TBW LGEA Teacher EGRA M 08140966892 73 18/10/20 17 Ibrahim Mode TBW LGEA Teacher M 081428058807 112 74 19/10/20 17 Bello Abubakar LGEA WURNO SSO M 07038672808 75 19/10/20 17 Zayyanu Seidu LGEA WURNO SSO M 07068237049 76 19/10/20 17 Hamza Abubakar LGEA M 08136121284 77 19/10/20 17 Ladana Muhammed LGEA M 08028596112 78 26/10/20 17 Dr. Gandu Deputy Director Special Duties Language Dept. NERDC M 79 17/10/20 17 RAN-Sokoto M 80 17/10/20 17 Sani Umar Jabbi Traditional Leader Gagi Community M saifullahi@gmail.com 81 17/10/20 17 Bello Sambo FOMWAN Sokoto M 08032404324 82 17/10/20 17 Fatima Attahiru FOMWAN Lead, Sokoto F 08069811933 83 CSACEFA Sokoto M 84 18/10/20 17 Bello Sani LGEA Dange Shuni LGEA Area Education officer/SSO M 08032444530 85 18/10/20 17 Nasim Shehu Kalambani Tumbuwal, LGEA Executive Secretary M 86 18/10/20 17 Abdulaki Abu Bakr Warno, LGEA Chairman M 87 18/10/20 17 Abdullahi Anyu Amanawa Kamfani, LGEA Dange Shuni AGLC Head Teacher M 88 17/10/20 17 Khadyah Muh’d Bello Sarkia Kebbi Shehu Model Primary School, Yabo Teacher F 113 89 19/10/20 17 Shamseden Abubakar Wamakko NFLC Teacher M 07034651824 90 20/10/20 17 Jimbal Z. Tabita SKS Model Primary School, Yabo Teacher M 91 17/10/20 17 Inno Madawaki Labbo Dogon Dagi Model School Teacher M 92 20/10/20 17 Umar Abubakar Achida Model Primary/Wurno Head Teacher M 0803138735 93 20/10/20 17 Imrana Ibrahim Labbo Dogon Daji Model School, Sokoto South Head Teacher M 08032579511 ibrahimimrana@gmail.co m 94 18/10/20 17 Bello Maishanu Maikada Model Primary School /Tambuwal Head Teacher M 09036866896 95 18/10/20 17 Ummaru Shehu Dange Shuni LGEA Secretary LGEA M 07039063685 96 19/10/20 17 Ustaz Dahini Shehu Warno Wurno, Sokoto Religious Leader/LGEA M 0706895285 97 19/10/20 17 Hulot Saluhu Wurno, Sokoto SBMC Wurno M 08065630287 98 18/10/20 17 Marafan Mai Kada Mai Kada, Tambuwal LGEA SBMC M 08029564634 99 19/10/20 17 Shuaibu Suleiman SBMC Labbo Dogon Daji SBMC Wurno M 07034642759 100 19/10/20 17 Ladan Muhammad Wurnto SBMC Achida Chairman SBMC M 08028596114 1010 1 19/10/20 17 Alhaji Mustapha Sokoto Janjona Wurno Traditional District Traditional Leader M 07066431665 114 102 16/10/20 17 Dr. Muhammad Jabbi Kilgori Ministry of Basic & Secondary Education Sokoto , State Honorable Commissioner for Education M 0803 507 4273 08097438979 103 19/10201 7 Abdullahi Garuba Achida Model Primary Sch Pry 2 Teacher M 104 17/10201 7 Murtala Umar Labbo Dogo Dayi Pry 3 Teacher M 105 18/10/20 17 Rashida Isah Model Primary School Pry 2 Teacher F 106 18/10/20 17 Nasiru Dan Galadima Makaranta Malari Abdullahi Aliyu NFLC M 107 20/10/20 17 Umaru Ahmad Sokoto State Ministry of Education Director Planning Research & Statistics M 08065628229 108 20/10/20 17 Umaru Muhammad Yabo State Universal Primary Education Board Director Planning Research & Statistics & Development M 080287002184 109 18/10/20 17 Nasiru Lawal Maimagani Centre for Social Advocacy Executive Director ,M 08039670580 110 12/10/20 17 Iro Umar Universal Basic Education Commision Head of Special Projects M 0803701184 111 12/10/20 17 Mr A.E Udoh Universal Basic Education Commission Director Head of EMIS M 08037878215 112 19/10201 7 Jamila Sa`ad Universal Basic Education Commission Senior Statistician/E MIS Dept F 08036169942 113 12/10/20 17 Dr Tokunboh Onosode Universal Basic Education Commision Director Planning F 0802325 3825 115 Research & Statistics 114 13/10/20 17 Dr Hamid Bobboyi UBEC Universal Basic Education Commision Executive Secretary M 08037054764 115 19/10/20 17 Yusuf Mohammad Sokoto South Local Government Education Authority School Support Officer M 07069034578 116 19/10/2017 Amina Shehu Abdullahi Sokoto South Local Government Education Authority School Support Officer F 08031330283 117 20/10/2017 Murtala Chika Head of EMIS PRS /Desk Officer Director NEI Focal person in SMOE State Ministry of Education M 08035938363 118 23/10/2017 Mr D M Yabani National Commission for Colleges of Education Special Assistant to NCCE EXECUTIVE Secretary M 08034032099 119 23/10/2017 Dr Sam Ugwuoti National Commission for Colleges of Education Director /Chairman Technical Working Group on EGRA M 08037073539 120 26/10/2017 Miki Koide UNICEF Education Specialist F 08035350983 121 27/10/2017 Fatima Sada DFID Education Adviser F 08036650016 116 NEI+ Mid-term Evaluation Interview Participants, Bauchi State # DATE NAME GENDER STATE ORGANISATIONS POSITION PHONE NUMBERS 1 10.16.2017 Abubakar Mansur Abdu M Bauchi SUBEB Director Planning and research 08064291522 2 10.16.2017 Lawan A Wumi M Bauchi SUBEB Director Upper Basic 08038049413 3 10.16.2017 Abdullahi Umar Abubakar M Bauchi SUBEB Head of EMIS and Research 08051639413 4 10.16.2017 Sulaiman Mohammed M Bauchi BASAME Head EMIS 08036184167 5 10.16.2017 Proffessor Yahaya Ibraheem Yero M Bauchi SUBEB Executive chairman 08035734231 6 10.17.2017 Isah Yahaya M Bauchi Bakatumbe Primary School Ningi LGA Head teacher 07082056757 7 10.17.2017 Hajiya Lami Maianguwa F Bauchi Bakatumbe Primary School Ningi LGA Women Leader SBMC N/A 8 10.17.2017 Hajiya Ladi Sawanu F Bauchi Bakatumbe Primary School Ningi LGA Member SBMC N/A 9 10.17.2017 Hannatu Nuhu F Bauchi Bakatumbe Primary School Member N/A 117 Ningi LGA SBMC 10 10.17.2017 Malam Sani Mai Unguwa M Bauchi Bakatumbe Primary School Ningi LGA Chairman SBMC 07084802911 11 10.17.2017 Sarkin Fawa M Bauchi Bakatumbe Primary School Ningi LGA Treasurer SBMC 08088099819 12 10.17.2017 Dauda Yakubu M Bauchi Bakatumbe Primary School Ningi LGA Member SBMC N/A 13 10.17.2017 Anas . S. Fawa M Bauchi Bakatumbe Primary School Ningi LGA Grade 2 Teacher N/A 14 10.17.2017 Zakari Idris M Bauchi Bakatumbe Primary School Ningi LGA Member SBMC N/A 15 10.17.2017 Abdul Hamid Riga M Bauchi Bakatumbe Primary School Ningi LGA P.R.O SBMC N/A 16 10.17.2017 Alahji S Idris Muhammad M Bauchi Bakatumbe Primary School Ningi LGA Traditional Ruler Sarkin Yamma N/A 17 10.17.2017 Idris Hassan Gwam M Bauchi Gwam Primary School Ningi LGA Head Teacher/ Grade 2 and 3 Class Teacher 08055855980 18 10.17.2017 Shaaibu Dan Gida M Bauchi Gwam Primary School Ningi LGA Chairman SBMC 08097831028 19 10.17.2017 Ibrahim Aliyu M Bauchi Gwam Primary School Ningi LGA Treasurer SBMC 08177487202 118 20 10.17.2017 Zakariya B Shehu M Bauchi Gwam Primary School Ningi LGA Member SBMC 08174518544 21 10.17.2017 Kabiru M Inuwa M Bauchi Gwam Primary School Ningi LGA SBMC Member 08092085319 22 10.17.2017 Yunusa Bala M Bauchi Gwam Primary School Ningi LGA SBMC Member 08182696944 23 10.17.2017 Aliyu Abdu M Bauchi Gwam Primary School Ningi LGA SBMC Member N/A 24 10.17.2017 Kande Audu F Bauchi Gwam Primary School Ningi LGA SBMC Women’s Leader 07036374731 25 10.17.2017 Binta Shuaibu F Bauchi Gwam Primary School Ningi LGA SMBC Secretary N/A 26 10.17.2017 Amina Garba F Bauchi Gwam Primary School Ningi LGA SBMC Member N/A 27 10.17.2017 Ramatu Aliyu F Bauchi Gwam Primary School Ningi LGA SBMC Member N/A 28 10.17.2017 Halima Yunusa F Bauchi Gwam Primary School Ningi LGA SBMC Member N/A 29 10.17.2017 Zarau Isyaku F Bauchi Gwam Primary School Ningi LGA SBMC Member N/A 30 10.17.2017 Zakari Barau F Bauchi LEA headquarters Ningi LGA Education Secretary 08036566295 31 10.17.2017 Yakubu Zakari M Bauchi LEA headquarters Ningi LGA Head of School Services 07088638343 32 10.17.2017 Kabiru Adam M Bauchi LEA headquarters Ningi LGA Accountant 08055553806 119 33 10.17.2017 Haruna Danladi M Bauchi LEA headquarters Ningi LGA School Support Officer 07086397289 34 10.17.2017 Ali Alhassan Tiffi M Bauchi LEA headquarters Ningi LGA Cashier 08020651563 35 10.17.2017 Nafiu Haruna Tiffi M Bauchi NFLC TIFFI Learning Facilitator 08131668787 36 10.17.2017 Shuaibu Adamu Mohammed M Bauchi NFLC TIFFI Access Coordinator 08087011111 37 10.17.2017 Muktar Siji M Bauchi NFLC TIFFI Area Organizer 081427847 38 10.18.2017 Tanko Abdulmuminu M Bauchi Baba Maaji Primary School Head Teacher 07069273109 39 10.18.2017 Habiba Ahmed F Bauchi Baba Maaji Primary School Grade 2 Class Teacher 08163926787 40 10.18.2017 Amina Bello Ibrahim F Bauchi Baba Maaji Primary School Grade 2 Class Teacher 07060930184 41 10.18.2017 Aliyu Bala M Bauchi Baba Maaji Primary School SBMC Chairman 08136941072 42 10.18.2017 Habiba B Aliyu F Bauchi Baba Maaji Primary School SBMC Member 08081290637 43 10.18.2017 Rabi Alhassan F Bauchi Baba Maaji Primary School SBMC Treasurer 08034763985 44 10.18.2017 Abubakar Baba Maaji M Bauchi AGLC Muassassatul Maratus Saliha Women Centre Bauchi LGA Area Organizer 08025712345 45 10.18.2017 Amina Usman F Bauchi AGLC Muassassatul Maratus Saliha Women Centre Bauchi LGA Learning Facilitator 07037169944 46 10.18.2017 Hassan Salmanu Usman M Bauchi LEA Headquarters Bauchi LGA Education Secretary 08067349998 120 47 10.18.2017 Nasiru M Yalwa F Bauchi Bauchi Ministry of Education Permanent Secretary 08034803456 48 10.18.2017 Dan Azumi Zakari T M Bauchi Bauchi Ministry of Education Director Planning 07038504520 49 10.18.2017 Ahmed A Umar M Bauchi Bauchi Ministry of Education Director School Services 08036932465 50 10.18.2017 Muhammad Musa Misau M Bauchi Bauchi Ministry of Education DSTVE 08034450320 51 10.18.2017 Hannatu Aliyu F Bauchi Bauchi Ministry of Education Deputy Director Admin 08027710272 52 10.18.2017 Musa Lois M Bauchi Bauchi Ministry of Education Director 08052361970 53 10.18.2017 Ado Tanko M Bauchi Bauchi Ministry of Education Principal Admin Officer 08067152406 54 10.18.2017 Abubakar Mohammed Sani M Bauchi NFLC Rafin Makaranta Learning Facilitator 08035810082 55 10.18.2017 Saleh Umar M Bauchi NFLC Rafin Makaranta CBMC Member 080504431704 56 10.18.2017 Abubakar Yusuf M Bauchi NFLC Rafin Makaranta CBMC Memeber 07060760530 57 10.18.2017 Bala Tanko M Bauchi NFLC Rafin Makaranta CBMC Secretary 08069389662 58 10.18.2017 Sukumun N Ezekiel M Bauchi Women Dev Ass for Self Reliance Sub Grantee Program Lead 08025710469 59 10.18.2017 Suleiman Usman Toro M Bauchi Muslim Aid Initiative Nigeria Sub Grantee Program Lead 08065606317 60 10.18.2017 Pst Sunday K Simon M Bauchi AONM Sub Grantee Program Lead 08095100882 61 10.18.2017 Alhassan Lawal M Bauchi Taimoko CDI Sub Grantee Program Lead 08034561613 62 10.18.2017 Talatu Musa M Bauchi FOMWAN LPO 07036104723 121 Sub Grantee 63 10.18.2017 Itanola A AbdulFatai M Bauchi CSACEFA Sub Grantee Ag LPO 08030687823 64 10.18.2017 Yahaya Santuraki M Bauchi Program Officer Sub Grantee RAN 08035072934 65 10.18.2017 Solomon Ezekiel Magado M Bauchi YMCA Sub Grantee Ass Program Officer 07064809921 66 10.18.2017 Dogara James Igbeji M Bauchi Development for Exchange Centre (DEC) Sub Grantee Program Manager 08088275818 67 10.18.2017 Haruna Mohammed Salisu M Bauchi Better Life Restoration Initiative (BERI) Sub Grantee Lead Program Officer 08063180608 68 10.18.2017 Fredson Ogbene M Bauchi WEIN Sub Grantee Lead Program Officer 07069466715 69 10.18.2017 Abdul Jabbar Abdullahi J M Bauchi SISWACHI Sub Grantee Lead Program Officer 08038373680 70 10.18.2017 Gideon N Dakup M Bauchi FACE-PAM Sub Grantee Lead Program Officer 080658008444 71 10.18.2017 Bilkisu A Sambo F Bauchi CIPRHES Sub Grantee Lead Program Officer 08067038324 72 10.18.2017 Safur Yakubu Aliyu F Bauchi CENCOHD Sub Grantee Lead Program Officer 08036071631 73 10.18.2017 Jeremiah Panshak Kaseem M Bauchi YLN Sub Grantee Lead Program Officer 07016612614 74 10.18.2017 Emmanuel Y Iliya M Bauchi RAHAMA Sub Grantee Lead Program Officer 08039676029 75 10.18.2017 Misbahu Kasim Isah M Bauchi BASNEC Lead M & E 08039135365 122 Sub Grantee 76 10.18.2017 Ibrahim Garba M Bauchi JNI Sub Grantee Lead M & E 08032060276 77 10.18.2017 Daniel Florence F Bauchi FAWOYDI Sub Grantee Lead Program Officer 08026334326 78 10.18.2017 Hussaini Ahmed M Bauchi RHISA Sub Grantee Lead Program Officer 07037074990 79 10.18.2017 Ali Chindo M Bauchi CIIHPS Sub Grantee Lead Program Officer 08036231207 80 10.18.2017 Aliyu Hajara F Bauchi ACE-HI Sub Grantee Lead Program Officer 07067457992 81 10.18.2017 Nicholas Osojah Afeso M Bauchi LEADTOTS Sub Grantee Program Manager 82 10.19.2017 Zainab Sanusi Ibrahim F Bauchi LEA Kofar Nassarawa Primary School Head Mistress 07034506489 83 10.19.2017 Hadiza Aliyu F Bauchi LEA Kofar Nassarawa Primary School Primary 2 Class Teacher 08169441212 84 10.19.2017 Aishatu Baba Bello F Bauchi LEA Kofar Nassarawa Primary School Primary 3 Class Teacher 08067043788 85 10.19.2017 Zubairu Mohammed usman M Bauchi LEA Kofar Nassarawa Primary School Deputy Chairman SBMC 08108731727 86 10.19,2017 Hadiza Usman Isah F Bauchi College of Education Kangare Ag Registrar 080325936694 87 10.19.2017 Aliyu Shuaibu M Bauchi College of Education Kangare Focal Person EGRA RTING Chairman 08065405586 123 88 10.19.2017 Mugana Yelmi Bisu M Bauchi College of Education Kangare Deputy Provost 08034256778 89 10.19.2017 Yelwa Abubakar Balewa F Bauchi BASAME Executive Secretary 08038565020 90 10.19.2017 Aliyu Gambo M Bauchi BASAME Director Planning Research and Statistics 080654373934 91 10.19.2017 Justina Daniel F Bauchi AGLC Miri Learning Facilitator 92 10.19.2017 Turaki Pastor Dauda M Bauchi AGLC Miri Head Pastor 07089257762 93 10.19.2017 Ayuba Mani M Bauchi AGLC Miri Proprietor 07034702660 94 10.19.2017 Hoshimu Musa M Bauchi AGLC Miri Member CBMC 08055174163 95 10.19.2017 Yahaya Saleh M Bauchi AGLC Miri Member CBMC 07085985821 96 10.19.2017 Haruna Saraki M Bauchi AGLC Miri Member CBMC 08072327036 97 10.19.2017 Elisha Saraki M Bauchi AGLC Miri Member CBMC 08024182286 98 10.19.2017 Murtala Nuhu M Bauchi AGLC Miri Member CBMC 08184501296 99 10.19.2017 David M Bauchi AGLC Miri Member CBMC N/A 100 10.19.2017 Bulus Maibawa M Bauchi AGLC Miri Member CBMC N/A 101 10.19.2017 Musa S Saraki M Bauchi AGLC Miri Member CBMC 08059072018 102 10.19.2017 Amir Saidu M Bauchi AGLC Miri Member N/A 124 CBMC 103 10.19.2017 Ibrahim Saraki M Bauchi AGLC Member CBMC 08089507285 125 ANNEX VI: DISCLOSURE/CONFLICT OF INTEREST FORMS 126 127 128