OCTOBER 19, 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by NORC at the University of Chicago. PERFORMANCE & IMPACT EVALUATION (P&IE) Final Performance Evaluation: School Health and Reading Program (SHRP) PERFORMANCE & IMPACT EVALUATION (P&IE) FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM (SHRP) October 19, 2016 PN 7384; USAID Contract No.: AID-617-C-12-00006 PRESENTED TO: USAID/Uganda Jana Wooden PRESENTED BY: NORC at the University of Chicago Jeffrey Telgarsky Executive Vice President 4350 East-West Highway, 8th Floor Bethesda, MD 20814 Telephone: (301) 634-9413 Fax: (301) 634-9301 Cover Photo: Kasimeri Primary School, Moroto, Uganda, taken by Betsy Bassan DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Executive Summary ...........................................................................................................................................................1 Section I. Evaluation Purpose and Evaluation Questions .........................................................................................4 Section II. Program Background.....................................................................................................................................6 Section III. Evaluation Design, Methods, and Limitations.........................................................................................8 A. Use of Best Practices..........................................................................................................................................8 B. Evaluation Design.................................................................................................................................................8 C. District and Schools Sampling Methodology .................................................................................................8 D. Data Collection and Analysis..........................................................................................................................10 E. Ethical Clearance................................................................................................................................................14 F. Design and Methodology Limitations............................................................................................................14 Section IV. Findings, Conclusions, and Recommendations....................................................................................15 Evaluation Question 1................................................................................................................................................15 A. Result 1 Reading ............................................................................................................................................15 B. Result 2 Health ..............................................................................................................................................21 Evaluation Question 2................................................................................................................................................25 A. Result 1 Reading ............................................................................................................................................25 B. Result 2 Health ..............................................................................................................................................29 Evaluation Question 4................................................................................................................................................39 Evaluation Question 5................................................................................................................................................48 Annexes.............................................................................................................A-Error! Bookmark not defined. Annex A. Evaluation Statement of Work (P&IE RFP)........................A-Error! Bookmark not defined. Annex B. Evaluation Timeline....................................................................B-Error! Bookmark not defined. Annex C. Data Collection Instruments.................................................C-Error! Bookmark not defined. C.1 National Stakeholders: MoESTS, USAID, RTI/SHRP, SHRP Sub-awardees...................C-Error! Bookmark not defined. C.2 District Stakeholder: DEO, DIS, PTC, CCT, LLB, FA .....C-Error! Bookmark not defined. C.3 Head Teacher Questionnaire ................................................C-Error! Bookmark not defined. C.4 Literacy-Trained Teacher Questionnaire............................C-Error! Bookmark not defined. C.5 Health-Trained Teacher Questionnaire..............................C-Error! Bookmark not defined. C.6 Classroom Observation Protocol ........................................C-Error! Bookmark not defined. C.7 SMC and PTA Questionnaire.................................................C-Error! Bookmark not defined. C.8 Adolescent Learner Interview and Participatory Activity Guide ....C-Error! Bookmark not defined. Annex D. Program Description...............................................................D-Error! Bookmark not defined. D.1 SHRP Program Description ....................................................D-Error! Bookmark not defined. D.2 SHRP Cooperative Agreement Modifications....................D-Error! Bookmark not defined. D.3 P&IE Contract ............................................................................D-Error! Bookmark not defined. Annex E. Matrix of Monthly Performance Feedback and SHRP Actions ......... E-Error! Bookmark not defined. Annex F. Sources of Information .............................................................F-Error! Bookmark not defined. F.1 List of Documents Reviewed..................................................F-Error! Bookmark not defined. F.2 Contact List.................................................................................F-Error! Bookmark not defined. Annex G. Summary Tables of SHRP Performance Against TargetsG-Error! Bookmark not defined. Annex H. Disclosure of Any Conflicts of Interest..............................H-Error! Bookmark not defined. LIST OF TABLES Table 1. SHRP Program Districts, Regions, Languages, and Interventions Table 2. Districts Selected Table 3. Schools Selected Table 4. SHRP Final Performance Evaluation: Key Questions Correlated to Data Sources and Methods Table 5. Estimated effect of SHRP on Cluster 1 Learners Performance Table 6. Estimated Effect of SHRP on Cluster 2 Learners Performance Table 7. Cluster 1 and 2: Oral Reading Fluency Results After Two and Three Years in Program Table 8. Cluster 1 and 2: Oral reading fluency targets and results at the end of P2 Table 9. SHRP Achievements in HIV/AIDS Knowledge, Attitudes, and Practices Table 10. Trained Teachers’ Classroom Effectiveness Table 11. Advocacy and Support for Reading Increased Indicators Table 12. Feedback Memo Analysis (May 2013 – January 2016) ACRONYMS AIDS Acquired Immunodeficiency Syndrome AOR Agreement Officer’s Representative CAM Continuous Assessment Monitoring CCT Coordinating Center Tutor CDCS Country Development Cooperation Strategy CHC Communication for Healthy Communities COR Contracting Officer’s Representative CPD Continuous Professional Development Training DEO District Education Officer DES Directorate of Education Standards DIS District Inspector of Schools EGRA Early Grade Reading Assessment EMIS Education Management Information System FA Field Assistant FGD Focus Group Discussion GoU Government of Uganda GPE Global Partnership for Education HIV Human Immunodeficiency Virus KAP Knowledge, Attitude, and Practice survey KII Key Informant Interview LARA Literacy Achievement and Retention Activity LLB Local Language Board M&E Monitoring and Evaluation MEEPP Monitoring and Evaluation of Emergency Plan Progress MEO Municipal Education Officer MoES Ministry of Education and Sports MIS Municipal Inspector of Schools MSS Monitoring and Support Supervision NAPE National Assessment of Progress in Education NCDC National Curriculum Development Centre NORC National Opinion Research Center PEPFAR President’s Emergency Plan for AIDS Relief PIASCY Presidential Initiative on AIDS Strategy for Communication to Youth P&IE Performance and Impact Evaluation PMP Performance Management Plan POC Point of Contact PTA Parent Teacher Association PTC Primary Teacher College RTI Research Triangle Institute SFI School Family Initiative SHRP School Health and Reading Program SMC School Management Committee UNEB Uganda National Examinations Board UNITY Ugandan Initiative for Teacher Development and Management System, a predecessor program to SHRP USAID United States Agency for International Development VSO Voluntary Services Overseas WEI World Education Incorporated FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 1 EXECUTIVE SUMMARY Introduction. The Final Performance Evaluation of USAID/Uganda’s School Health and Reading Program (SHRP) assesses RTI’s achievement of SHRP performance goals with regards to improving school reading and health outcomes. Given SHRP’s extension to August 2019, this final performance evaluation also provides recommendations to help maximize results in SHRP’s extension years. In addition, it informs scale-up efforts via LARA and the Global Partnership for Education as well as similar efforts in other countries. Given this evaluation was carried out as part of the Performance & Impact Evaluation Contract (P&IE), it also sheds light on the findings of the annual SHRP impact evaluations and is able to examine the degree to which SHRP utilized the continuous evaluation monthly feedback for adaptive management purposes. It is important to note the context in which SHRP operates involves more than 50% teacher absenteeism, approximately 25% student absenteeism, very high levels of gender-based violence, and widespread hunger. Thus, there are many factors at play affecting school-based initiatives. Key evaluation questions. The SHRP Final Evaluation examined six evaluation questions developed collaboratively with USAID/Uganda regarding the extent to which:  SHRP improved reading and health outcomes  SHRP’s capacity building approach was perceived to facilitate improvements in reading and health instruction  SHRP’s community mobilization approach was perceived to facilitate improvements in reading and health outcomes?  SHRP reading and health activities are on track to continue without USAID assistance  SHRP utilized monthly continuous evaluation data for adaptive management Methodology. This performance evaluation is cross-sectional and analytical, primarily based on qualitative data. We held 96 key informant interviews at the national and district levels, visited seven districts selected via purposeful sampling, observed 22 classes, and held 14 focus group discussions with community members and adolescent learners. We rigorously analyzed the resulting data. All interviews, observations, and discussions were recorded. We then carefully tabulated interview data and developed frequency distributions for each stakeholder group, and consolidated it to identify findings for each evaluation question. We used qualitative content analysis to identify themes associated with the evaluation questions in the classroom observations and focus group data. We further triangulated our findings with SHRP M&E data to arrive at the final findings, conclusions, and recommendations. Global significance. SHRP is a program of considerable interest to USAID worldwide, in at least three main ways:  It represents a flagship effort in promoting literacy through use of mother tongue and therefore its experience will inform efforts elsewhere  Given increasing Agency interest in achieving greater impact through multi-sectoral programming, SHRP sheds light on both the challenges and opportunities in the day-to-day of programming in more than one sector, in this case the health and education sectors  The design includes an innovative application of USAID’s Evaluation Policy, namely a parallel FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 2 contract, P&IE (providing continuous third party monthly feedback, annual impact evaluations, and two performance evaluations, mid-term and final), which creates learning opportunities on the value and structuring of this aspect of the Evaluation Policy Key Evaluation Takeaways Overall. SHRP is regarded positively for having pioneered in Uganda what is generally viewed as a promising local language early grade reading initiative while working very collaboratively with all elements of the government education system. SHRP created local language early grade reading materials in twelve local languages and English for P1 – P4, totaling 52 pupil primers and 52 teachers’ guides with scripted lesson plans which are now being scaled up beyond SHRP’s 31 districts into 55 other districts and the Global Partnership for Education, covering 86 of Uganda’s 112 districts, almost 80% of the country. SHRP has worked to build up a cadre of master trainers and district level trainers who in turn train and provide ongoing support to teachers. SHRP has trained 14,210 teachers, 9,277 head teachers, and approximately 153 CCTs in early grade reading methodologies and leadership. In total, the program will work in 4,317 schools – 810 schools in 17 districts also have R2 Health activities, which primarily consists of the School Family Initiative (SFI) in which mixed gender and age groups meet after school with teachers to learn about and discuss a wide range of sensitive health and HIV/AIDS topics. Reading Key informants interviewed for the performance evaluation perceive much more progress in reading as a result of the SHRP EGR intervention than is substantiated by impact evaluation results, which show very modest gains due to SHRP. This disparity is likely due to the fact that district and school staff, parents, and students rely almost exclusively on observational data in forming opinions of reading performance as there is no quantitative assessment on reading carried out by teachers at the school level. The top factors perceived by respondents as impeding reading performance are lack of parental involvement and student absenteeism – both of which could have been addressed by earlier and more strategic community mobilization efforts – followed by teacher motivation and transfers. While respondents perceive SHRP’s cascade training model as effective, many were quick to cite typical shortcomings of cascade models and the pending need to supplement such training with robust mentoring and coaching at the school level, and also address issues in the instructional materials. We have noted these same issues in the mid-term evaluation and in continuous evaluation monthly feedback. On sustainability, SHRP’s approach from the outset has been very oriented to working with and through national structures, building country ownership and capability, which can be consolidated through a clearly articulated and implemented exit strategy in the extension years. Health The impact evaluation on SHRP’s HIV/AIDS and health interventions, which primarily consists of the School Family Initiative (SFI), showed positive but moderate gains in general knowledge; insignificant FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 3 gains in prevention knowledge, attitudes, and behavior, except a notable drop in sexual activity; and a positive increase in HIV/AIDS discussions at school. Respondents with knowledge of SHRP’s health and HIV/AIDS activities perceive that SFI programming, where it is working, creates a much more open and supportive school environment that better supports learning and reading outcomes. Respondents praise improvements in SFI program materials, and, noting that schools are a very effective way to reach youth with health information, suggest that there should be some way to address its current optional stature, which would also reinforce mainstreaming of sexual and reproductive health into the thematic curriculum. Adaptive management SHRP has used P&IE continuous performance feedback data to practice adaptive management and improve performance across the gamut of its activities. Recommendation highlights (with lead shown in parentheses)  Address the need for school administrators, teachers, students, and parents to be have concrete evidence in the form of data regarding how well their learners are reading (SHRP)  Acknowledge the dilution and absorption limits of cascade training and move quickly to shore up school-based mentoring and coaching (strong recommendation in mid-term evaluation also) (SHRP)  A dominant focus of SHRP’s extension should be building PTC capacity for in-service and pre￾service EGR (SHRP)  Address issues in the instructional materials – confusing cross-referencing, errors, inconsistencies – and use that opportunity to better integrate health topics (SHRP)  Develop a comprehensive exit strategy and sustainability plan (SHRP)  Mobilize influential voices including foundational bodies to galvanize parents around the value of education and get their kids to schools (SHRP)  Further study SFI effects to inform scale-up and possible adoption of SFI as an official part of the co-curriculum (MoES), in order to take advantage of perceived positive contributions to learning by rolling it out to other SHRP districts (USAID for funding)  Organize donors to advocate for increased GoU education funding (USAID) FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 4 SECTION I. EVALUATION PURPOSE AND EVALUATION QUESTIONS Purpose. The purpose of the final evaluation is to assess RTI’s achievement of SHRP performance goals with regards to improving school reading and health outcomes. Given SHRP’s extension to August 2019, this final performance evaluation will also provide recommendations to help maximize results in SHRP’s extension years. It will also inform scale-up efforts via LARA and the Global Partnership for Education as well as similar efforts in other countries. Evaluation Questions. The final evaluation will focus on the following five key questions:  To what extent did SHRP improve reading and health outcomes? What factors contributed to or inhibited improvements in desired outcomes?  How was SHRP’s capacity building approach for teachers perceived to facilitate improvements in reading and health instruction?  How was SHRP’s community mobilization approach perceived to facilitate improvements in reading and health outcomes?  To what extent are the reading and health activities on track to continue without USAID assistance?  To what extent did SHRP utilize the continuous evaluation data for adaptive management? These questions were selected and refined in consultation with stakeholders and reflect the core purpose of the evaluation while also informing features of the program that are germane to results achievement for both the extension and scale-up efforts. Question 1 will assess SHRP’s success in achieving targets and results as stated in the Performance Management Plan and other program documents, as well as its achievement of impacts as measured through P&IE’s impact evaluation. It will identify program and contextual factors that facilitated or hindered its success, and flag actions that can be taken in the next three years of implementation to maximize achievements. Questions 2 and 3 delve into two key design elements of SHRP – capacity building/training and community mobilization – and will surface insights of value to stakeholders and other interested parties in Uganda and elsewhere seeking to implement similar programs. Question 2 will examine SHRP’s capacity building approach which includes:  Cascade training involving master trainers, training of trainers, teacher training, and school￾based peer training  Monitoring and support supervision with classroom observation and teacher feedback and coaching  Continuous professional development Question 3 will examine the extent to which SHRP has been able to mobilize parents and communities in support of early grade reading. Question 4 will examine whether the basic capacity and systems have been built and institutionalized within the GoU, whether additional capacity and systems are necessary to fully institutionalize the FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 5 program; and whether the GoU is making the necessary financial commitments to carry the program forward. Question 5 will examine the degree to which the process of continuous evaluation – third party observation with monthly feedback to RTI – has led to learning and management adaptations to improve performance. Interview guides used to collect information on these questions are included in Annex C, specifically for national stakeholders such as central government officials, USAID, and RTI and its implementing partners; district officials, head and trained teachers (reading and health); classroom observation; and focus group discussions with adolescent learners as well as school management committees and parent teacher associations. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 6 SECTION II. PROGRAM BACKGROUND The School Health and Reading Program (SHRP) is a seven-year USAID-funded cooperative agreement implemented by RTI International in collaboration with SIL Language Education and Development (SIL LEAD) which supports local language reading, World Education Inc. (WEI) which supports HIV/AIDS and health programming, and Perkins International which supports Special Needs Education; all of whose funding from SHRP concludes by the end of FY2016. Other partners include Voluntary Services Overseas (VSO) which supports continuous professional development in five of SHRP’s 31 districts through May 2017; and the International Book Bank and Books for Africa which provide supplementary reading materials for cost share. The Center for Social Research supported EGRA implementation through May 2016. The overall objective of the program, “Increasing Literacy and Health Seeking Behaviors,” mirrors USAID/Uganda’s Health Development Objective 3, through achievement of sub-objectives 3.1.1 “Health￾Seeking Behaviors Increased” and 3.1.1.1 “Improved Literacy.” Underlying this is USAID’s goal to support the Government of Uganda (GoU) in developing, implementing, assessing, and bringing to scale a successful approach to reading instruction and to deliver the goal of the Ministry of Education and Sports (MoES) toward achieving a Ugandan-led “Literacy Policy.” To this end, the program will build institutional capacity, support policy development, and help institutionalize the training, support structures, and policies necessary for sustainability. To achieve the stated objectives, the program has two key results: 1. Improved Early Grade Reading and Transition to English 2. Improved HIV/AIDS Knowledge, Attitudes, and Practices The development hypotheses in each result are: Result 1 Reading: By focusing interventions on the nexus of language, pedagogy, and instructional materials, USAID can significantly improve students’ early grade reading and P3 literacy scores within targeted schools and districts. Result 2 Health: By strengthening cross-sector coordination between USAID’s health and education partners, USAID can significantly improve teachers’ and students’ HIV/AIDS knowledge and skills within targeted schools and districts. Over the life of the program, May 2012-August 2019, reading improvements will be directly supported in 3,476 schools by working through the established MoES systems to train Primary Teacher College (PTC) tutors, coordinating center tutors (CCTs), district and areas inspectors and head teachers to train and support teachers to effectively teach reading. Under Result 1, the program aims to train over 20,000 teachers in early grade reading and provide effective reading instruction to approximately 2 million learners in P1-P4. Indirect beneficiaries from the scale up of the reading program through the MoES will add another 1.4 million learners. Another important component of the program is supporting the National Curriculum Development Centre (NCDC) to develop language boards, reading instructional materials, and pupil primers in both English and 12 local languages – over 2 million reading primers will be in the hands of learners by the end of the program. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 7 Under result 2, over 8,000 teachers will be trained and 500,000 learners across nearly 1,700 primary schools, secondary schools and BTVET institutions reached through learning activities designed to help them understand and practice healthy behaviors aimed at preventing HIV/AIDS. Result 2 will conclude in September 2016. In total, the program will work in 4,317 schools – 810 schools will have both result 1 and result 2 activities. The program will cover a total of 12 languages in 31 districts for result 1. Result 2 activities will take place in 17 districts. The languages and districts were selected by the MoES. The Performance and Impact Evaluation contract is a five-year effort implemented by prime contractor NORC and subcontractor Panagora Group. The primary purpose of P&IE is to evaluate SHRP program implementation and effectiveness, and the extent to which the program’s literacy and HIV/AIDS prevention activities result in measurable impact at the level of student learning. NORC leads overall contract management and implementation, annual impact evaluations, and workshops to disseminate results; and Panagora leads continuous evaluation (ongoing performance monitoring and feedback to the RTI program team to promote adaptive management and continuous improvement), performance evaluations, and supports dissemination. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 8 SECTION III. EVALUATION DESIGN, METHODS, AND LIMITATIONS A. USE OF BEST PRACTICES The Final performance evaluation uses best practices in evaluation, namely:  Evaluation team with subject matter specialists in literacy and health, including national specialists to ensure country knowledge and context and to build local capacity  Participatory processes for development of the performance evaluation methodology and questions that included vetting our evaluation methodology in startup, ongoing consultation on evaluation questions and sampling, and stakeholder consultations during the evaluation (see “Stakeholder Consultations” under Section D below)  Mixed methods to collect data from multiple sources followed by rigorous triangulation between a wide array of national and district stakeholders, and other data, including continuous evaluation data, impact evaluation data, classroom observation, and program documents  Transparency in evaluation design, implementation, and dissemination of findings B. EVALUATION DESIGN This will be a cross-sectional descriptive and analytical performance evaluation, primarily based on qualitative data. We will use multiple sources of data for the performance evaluation, including:  Information and data gathered throughout the P&IE contract: o Continuous evaluation data including 28 months of activity observation reports and performance feedback memos o Quantitative data and analyses from annual impact evaluations of EGRA results and final impact evaluation of KAP results  Program and other documents  Semi-structured interviews with key informants  Classroom observation  Focus group discussions (FGDs) C. DISTRICT AND SCHOOLS SAMPLING METHODOLOGY We used purposeful sampling to select districts and schools, aiming at a representative blend of:  Regions  Local languages  Urban and rural areas  High-to-low performing schools based on EGRA data  SHRP clusters (staggered district implementation) Further, we:  Prioritized schools receiving both R1 Reading and R2 Health interventions  Were alert to possibilities for including schools that share LARA languages FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 9  Sought to include some districts visited during the mid-term evaluation1  Consulted with SHRP to glean additional considerations, e.g., districts of particular interest to the MoES such as Moroto. Table 1 shows SHRP languages, regions, districts, and interventions. Table 1. SHRP Program Districts Cluster Local language Region Districts Result area 1 Luganda Central Wakiso, Gomba Result 1 and 2 1 Runyankore/Rukiga South West Kiruhura, Bushenyi, Kabale 1 Ateso Eastern Kumi, Katakwi, Serere 1 Leblango Northern Apac, Lira, Kole 2 Runyoro/Rutooro Mid-Western Masindi, Kyenjojo, Kabarole 2 Acholi Mid-Northern Gulu, Pader, Kitgum 2 Lugbarati West Nile Arua Result 1 only 2 Lumasaaba Mid-Eastern Mbale, Sironko, Manafwa 3 Lugwere Mid -Eastern Budaka, Pallisa, Kibuku 3 Ngakarimojong North East Nakapiripirit, Napak, Moroto, Kaabong 3 Lukhonzo Mid-Western Kasese 3 Lusoga East Central Iganga and Kamuli Source: USAID/Uganda SHRP PMP. Version September 19, 2013 (Bold = sites visited during mid-term performance evaluation) Table 2 shows the districts selected for data collection during the final performance evaluation. Our sample of seven districts, 23% of SHRP’s 31 districts, is geographically and linguistically diverse: it includes six regions; a blend of urban, semi-urban, and rural areas; agricultural, farming, and pastoral communities; and range of languages with some of the most established orthographies, e.g., Luganda, as well as some of the newest, e.g., Ngakarimojong. It includes all three of SHRP’s clusters, and all but one district (in Cluster 3) has benefited from both R1 Reading and R2 Health interventions. Table 2. Districts Selected Region District Result Language Cluster Urban/Rural Northern Lira R1/R2 Leblango C1 Semi-urban/Rural North-Eastern Moroto R1 Ngakarimojong C3 Rural Eastern Katakwi R1* Ateso C1 Rural Central Wakiso R1/R2 Luganda C1 Urban Central Gomba R1/R2 Luganda C1 Semi-urban/Rural South-Western Kiruhura R1* Runyankore-Rukiga C1 Semi-urban/Rural Mid-Western Kabarole R1* Runyoro-Rutooro C2 Rural 1 This sheds light on several evaluation questions, e.g., factors contributing to or inhibiting the achievement of health and reading outcomes by probing how well planned school turnarounds (Lira) have actually evolved (Q1); changes in perceptions on SHRP’s capacity building and community mobilization efforts in facilitating reading and health outcome (Q2 and Q3); and the degree to which the reading and health methodologies have been further institutionalized (Q4). FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 10 * District had R2 programming previously, before R2 funding cuts required a reduction from 17 to 5 districts. Table 3 shows the schools selected within districts. Our sample included:  Six low-performing and eight high-performing schools based on EGRA data  Six districts that either currently receive both R1 Reading and R2 Health interventions (three districts) or had both interventions (three districts)  Six languages with three overlapping LARA languages  All three SHRP clusters  11 primary schools and three secondary schools, which was appropriate in size for a descriptive and analytical qualitative evaluation in which depth through semi-structured interviews, focus group discussions, and observation is key to gaining the rich contextual data needed to shed light on highly representative impact evaluation findings Table 3. Schools Selected District School Perf. Result Language Cluster PS SS Lira P/S School 1 Low R1/R2 Leblango C1 1 S/S School 2 Low 1 Moroto P/S School 3 High R1 Ngakarimojong C3 1 P/S School 4 Low 1 Katakwi P/S School 5 High R1* Ateso C1 1 P/S School 6 Low 1 Wakiso P/S School 7 High R1/R2 Luganda C1 1 S/S School 8 High 1 Gomba P/S School 9 High R1/R2 Luganda C1 1 S/S School 10 High 1 Kiruhura P/S School 11 High R1* Runyankore￾Rukiga C1 1 P/S School 12 Low 1 Kabarole P/S School 13 Low R1* Runyoro￾Rutooro C2 1 P/S School 14 High 1 TOTAL 14 6L 8H 6 (3 LARA) All C’s 11 3 D. DATA COLLECTION AND ANALYSIS Methods. Our approach to each data collection method is described below.  Stakeholder consultations – In order to approximate as closely as possible, the features of a utilization focused evaluation (which recognizes that use of evaluation results is most likely when users are involved in design and conduct of an evaluation), we held a joint consultation with all stakeholders to review the evaluation purpose, questions, and methodology; and separate FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 11 consultations with each major stakeholder to review initial findings. We offered the opportunity for participation in data collection to USAID and the MoES; USAID joined visits in one district.  Document review – We reviewed the Performance Management Plan, quarterly, annual, and ad hoc reports, work plans and presentations, and minutes; monitoring data and detailed program reports; EGRA reports and KAP survey data; cooperate agreement modifications four through nine; SHRP internal and external information and knowledge sharing materials (including virtual) P&IE’s annual impact evaluations for years two and three; Republic of Uganda budget information; SHRP newsletters and brochures; SHRP teacher training guides, learner textbooks, and support supervision monitoring tools; district and school level monitoring and supervision registers and report; and teacher lesson plans and registers. We also reviewed P&IE Continuous Evaluation reports, including monthly observed events, activities reports, and performance feedback memos. We included only those documents issued since the Mid-Term Performance Evaluation.  Key informant interviews (KIIs) – KIIs were conducted to respond to the key evaluation questions, specifically to assess the extent to which: SHRP improved reading and health outcomes (including factors contributing to or inhibiting improvements); SHRP’s capacity building approach for teachers and community mobilization were perceived to facilitate improvements in reading and health instruction; and SHRP activities are on track to continue without USAID assistance. We conducted KIIs with different stakeholders at national, district, and school/community levels. We interviewed a total of 93 stakeholders: 26 national stakeholders, from MoES, USAID, RTI/SHRP, and SHRP implementing partners; and 67 district stakeholders, including the District Education Officer (DEO), District Inspector of Schools (DIS), Primary Teacher College (PTC), Coordinating Center Tutor (CCT), Local Language Board, and Teachers (head teachers, trained R1 and R2 teachers) from each district.  Focus Group Discussions (FGDs) – We conducted a total of 14 FGDs: three SMC FGDs and 11 adolescent learner FGDs. The FGDs with SMCs allowed us to assess parent and community engagement and perceptions, while the adolescent learner FGDs provided valuable insight into the effectiveness of SHRP school level R2 Health activities. FGD participation included 11 SMC members and 29 adolescent learners. Participation among SMC FGDs was 31 percent women, and 69 percent men. Learner FGDs were quite lively, and ranged from 2 – 22 participants per FGD, and almost equal participation of girls and boys.  Classroom observations – We visited 11 primary schools in seven districts to examine the extent to which teachers are applying the reading methodologies learned during SRHP training sessions. In each primary school, we observed two classes, for a total of 22 classes and a combination of P1-P4, local language, reading, and English. The team also assessed the conditions of the classroom, use of SHRP teacher and learner materials, learner participation in reading activities, and the interaction between teachers and learners for Result 1. See Annex F for a full listing of all documents reviewed, KIIs, FGDS, and classrooms observed. Tools. Based on the key evaluation questions, we developed interview guides tailored to each audience, including:  Key Informant Interview (KII) Guides: o National Stakeholders: MoES, USAID, RTI/SHRP, SHRP Sub-awardees o District Stakeholder: DEO, DIS, PTC, CCT, LLB, FA o Head Teacher FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 12 o Literacy-Trained Teacher o Health-Trained Teacher  Classroom Observation Protocol  Focus Group Discussion Guides o SMC and PTA o Adolescent Learners Each evaluation question was addressed through a triangulation of data and information gathered from multiple sources (e.g., GoU, USAID, RTI, partners) and stakeholders. The documents and information collected through KIIs, FGDs, and site visits/observations were analyzed by the performance evaluation team through a process of identifying key themes by evaluation question. The following table summarizes data collection methods and sources aligned with each evaluation question. Table 4. SHRP Final Performance Evaluation: Key Questions Correlated to Data Sources and Methods Key Evaluation Questions Data Sources Document Review KIIs, FGDs, Classroom Observation Key Question #1 To what extent did SHRP improve reading and health outcomes? What factors contributed to or inhibited improvements in desired outcomes?  SHRP reports: PMP, monitoring data, detailed program reports  SHRP EGRA and KAP data  P&IE Annual Impact Evaluation Reports KII:  GoU: MoES counterparts, department heads, and district officials  USAID: SHRP AOR, P&IE COR, HIV/AIDS POC  RTI: SHRP leadership, M&E, result and initiative leads; and sub-awardees  School staff: head teachers, trained teachers FGD:  SMC/PTA Key Question #2 How was SHRP’s capacity building approach for teachers perceived to facilitate improvements in reading and health instruction?  SHRP reports: PMP, quarterly, annual, ad hoc reports, work plans and presentations, minutes  P&IE Continuous Evaluation reports (observed events, monthly reports, performance feedback memos) KII:  GoU: MoES counterparts, department heads, and district officials  USAID: SHRP AOR, P&IE COR, HIV/AIDS POC  RTI: SHRP leadership, M&E, result and initiative leads; and sub-awardees  School staff: head teachers, trained teachers Classroom observation  Primary and secondary schools Key Question #3 How was SHRP’s community mobilization approach perceived to facilitate improvements in reading and health outcomes?  SHRP reports: PMP, quarterly, annual, ad hoc reports, work plans and presentations, minutes  P&IE Continuous Evaluation reports (observed events, monthly reports, performance feedback memos) KII:  GoU: MoES counterparts, department heads, and district officials  USAID: SHRP AOR, P&IE COR, HIV/AIDS POC  RTI: SHRP leadership, M&E, result and initiative leads; and sub-awardees  School staff: head teachers, trained teachers FGD  SMC/PTA FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 13 Table 4. SHRP Final Performance Evaluation: Key Questions Correlated to Data Sources and Methods Key Evaluation Questions Data Sources Document Review KIIs, FGDs, Classroom Observation Key Question #4 To what extent are the reading and health activities on track to continue without USAID assistance?  Government policies, systems, and tools reflecting/supporting the methodology (including exit strategy)  Government budget information  Instructional materials  District and school level monitoring and supervision registers and report  P&IE Continuous Evaluation reports (observed events, monthly reports, performance feedback memos) KII:  GoU: MoES counterparts, department heads, and district officials  USAID: SHRP AOR, P&IE COR, HIV/AIDS POC  RTI: SHRP leadership, M&E, result and initiative leads; and sub-awardees  School staff: head teachers FGD  SMC/PTA Key Question #5 To what extent did SHRP utilize the continuous evaluation (CE) data for adaptive management?  SHRP reports: quarterly and annual reports, work plans, PMP  SHRP internal and external information and knowledge sharing materials (including virtual)  P&IE Continuous Evaluation reports (observed events, monthly reports, performance feedback memos) KII:  GoU: MoES counterparts, department heads, and district officials  USAID: SHRP AOR, P&IE COR, HIV/AIDS POC  RTI: SHRP leadership, M&E, result and initiative leads; and sub-awardees See Annex C for full guides. Analysis Data collected from semi-structured interviews, FGDs and classroom observations were recorded and analyzed independently for Results 1 and 2. To identify key findings corresponding with each evaluation question, we used the following process. For semi-structured interviews, we entered the data into spreadsheets by stakeholder group, including national officials, district ministry officials, head teachers, literacy trained teachers, and health trained teachers. We determined frequency distributions for each stakeholder group at the national, district, and school level, i.e., the number of times a particular response was provided to a question within each stakeholder group. We then consolidated responses across stakeholder groups. The qualitative data from classroom observations, FGDs, and key informant interviews were analyzed using thematic content analysis2 with findings in order of commonality of responses within themes associated with the evaluation questions. Findings from the performance evaluation were then triangulated with project-related M&E data (SHRP Performance Management Plan, EGRA and KAP survey data, P&IE impact evaluations for R1 Reading and for R2 Health, and P&IE continuous evaluation data) to arrive at the final findings, conclusions and recommendations. In this process, we consider not just the times a response is given but also the additional content provided in the interview process. For example, sometimes respondents will bring up something the evaluators did not think of; sometimes 2 Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). London: Sage Publications FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 14 people will offer different types of explanations that will help us understand the full complexity of the question. When we group different answers together and synthesize them, we are noting agreement but also differences as well; what we write is the product of that process. This includes selecting quotes that represent the general trends being expressed, or in some cases an unusual perspective. E. ETHICAL CLEARANCE We submitted our evaluation protocol to the Uganda National Council for Science and Technology (NCST) for ethical clearance. NCST granted approval (SS 3487) with the restriction that learners be age 18 and above. We informed all participants of the purpose of the evaluation and obtained verbal consent from key informants and written informed consent from all learners who participated in FGDs. F. DESIGN AND METHODOLOGY LIMITATIONS We recognize the inherent limitations and challenges associated with a non-experimental evaluation design and with collecting and analyzing information for a program that is both national and local in its scope. Fortunately, we have the benefit of collecting a large amount of information over a long period of time covering a great deal of the activities underway. We also have a strong understanding of the work, and did not have to spend a large amount of time simply orienting ourselves to the basic fundamentals of the program. Panagora and NORC took advantage of the time in advance of the final evaluation to analyze available data, including continuous evaluation data, and the SHRP work plan and Performance Management Plan against the progress as stated in the quarterly reports, which increased the foundation of our understanding and knowledge coming into the evaluation. To minimize the degree of bias in the evaluation and increase the validity of the findings, we rigorously triangulated results from the document review with other data sources from interviews, classroom observations, focus groups, and stakeholder consultations. While recognizing the limits of this evaluation regarding generalizability, we assessed a cross-section of the population that cuts across both project components, high and low-performing schools, and across multiple districts and regions in order to provide a snapshot into the results and lessons learned that may inform future projects with similar conditions and target populations. Specifically, we visited 11 primary schools and three secondary schools, which was appropriate in size for a descriptive and analytical qualitative evaluation in which depth through semi-structured interviews, focus group discussions, and observation is key to gaining the rich contextual data needed to shed light on highly representative impact evaluation findings. This is the most cost-effective approach for achieving the results within the timeframe and resources available. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 15 Highlights of the Impact Evaluation Results The impact analyses suggest that the program has had some positive impact but this impact, when it exists, is quite modest. Cluster 1: At the end of P3 learners that received SHRP were on average able to identify only 2.4 more letters sounds in English than identical learners that did not receive SHRP. In Luganda, the advantage is 5 letter sounds and in Leblango, it is 2.5. SHRP did not improve the ability to identify letter sounds of Ateso and Runyakore/Rukiga learners. SHRP’s effect on oral reading fluency is zero for all languages with the exception of Ateso (4 words) an RR which is only borderline significantly different from zero. SHRP’s effect on reading comprehension is zero for all languages with the exception of Ateso. Cluster 2: At the end of P2, SHRP has not improved the ability of learners to identify letter sounds, their oral reading fluency may be improving but the effects are very small if they exist. Cluster 3: After one year of the intervention we found that as a result of SHRP, Lusoga and Ngakarimojong learners can identify 1 additional letter sound. The learners that received SHRP show a small advantage relative to the control group when asked to identify the direction in which text should be read (print orientation). SECTION IV. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION QUESTION 1 To what extent did SHRP improve reading and health outcomes? What factors contributed or inhibited improvements in desired outcomes? A. RESULT 1 READING SHRP has created local language early grade reading materials in twelve local languages and English for P1 – P4, totaling 52 pupil primers and 52 teachers’ guides with scripted lesson plans which are now being scaled up beyond SHRP’s 31 districts into 55 other districts and the Global Partnership for Education, covering 86 of Uganda’s 112 districts, almost 80% of the country. SHRP has worked to build up a cadre of master trainers and district level trainers who in turn train and provide ongoing support to teachers. SHRP has trained 14,210 teachers, 9,277 head teachers, and approximately 250 CCTs in early grade reading methodologies and leadership. In total, the program will work in 4,317 schools. Starting in 2013 in Cluster 1, SHRP is reaching 11 districts using Ateso, Leblango, Luganda and Rukiga as languages of instruction. In 2014, SHRP added Cluster 2 with 10 additional districts using Rutoro, Acholi, Lugbarati and Lumasaba as languages of instruction. In 2015, SHRP started work in Cluster 3, adding 10 districts and Lugwere, Nakarimojong, Lukhonzo and Lusoga as languages of instruction. A1. Findings First, we provide results from the impact evaluation as a backdrop to the performance evaluation data which seeks to shed light on the modest reading results achieved by SHRP as revealed by the impact evaluation. Results from the Impact Evaluation NORC has conducted impact evaluations of SHRP for Clusters 1, 2 and 3. These clusters have received the intervention for Years Three, Two and One respectively. Overall, the analyses suggest that SHRP has had some positive impact but this impact, when it exists, is quite modest. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 16 Table 5 shows the additional letter sounds, number of words, and reading comprehension questions that on average students in the treatment group were able to correctly identify compared to the control group. 3 Table 5. Estimated effect of SHRP on Cluster 1 Learners Performance (End of P3) Language Additional # of correct letter sounds Additional reduction in letter sound zero scores Additional # of correct words Additional # of correct reading comprehension questions English 2.4** 0.07 3.4 0.2 Luganda 5.0* 0.06 3.0 0.3 Leblango 2.5* 0.01** 0.1 -0.0 Ateso 1.8 0.03 4.4** 0.3* Runyakore/Rukiga 1.9 0.01 6.3* 0.3 *** p<0.01, ** p<0.05, * p<0.1. All regressions include school fixed effects, individual controls, and a constant term. Individual controls include age and sex of learner, dummy for age missing, score on household assets index, dummies for student living with both parents, someone at home reading to the student, and language spoken at home. Regressions for English language include control for local language. Among learners in Cluster 1, the estimates are, in general, positive, although, in most cases, they are not statistically significant, i.e., we cannot say with a high degree of confidence that those differences are real and not simply a result of random chance due to sampling. Even if we assume the positive impact to be real, it is, in general, small. For example, in the letter sound recognition subtask, the advantage of the treatment group over the control group is around 2 letter sounds, with the exception of Luganda where it is 5. Similarly, SHRP’s effect on oral reading fluency ranges from zero words (Leblango) to a maximum of 6 words (Runyankore/Rukiga), and in all but one case (Ateso) the differences between the treatment and the control group are not statistically significant. The only case where we are able to detect a positive effect on reading comprehension is Ateso, where treatment learners show an average advantage of a third of a question. Table 6 shows the same estimates for Cluster 2 at the end of P2; and also do not indicate significant effects of SHRP on the letter sound identification, fluency, or reading comprehension subtasks of EGRA. Table 6. Estimated effect of SHRP on Cluster 2 Learners Performance (end of P2) Language Additional # of correct letter sounds Additional reduction in letter sound zero scores Additional # of correct words Additional # of correct reading comprehension questions English 0.4 0.03 1.3** 0.0 Runyoro-Rutooro 0.8 0.05 2.6* 0.2 Acoli 1.4 0.01 1.6* 0.1 Lugbarati -1.0 0.10 0.8* 0.0 Lumasaaba 0.9 0.12 -0.0 -0.0 *** p<0.01, ** p<0.05, * p<0.1. All regressions include school fixed effects, individual controls and a constant term. Individual controls include age and sex of learner, dummy for age missing, score on household assets index, dummies for student living with both parents, someone at home reading to the student and language spoken at home. Regressions for English language include control for local language 3 NORC. (2016). Impact Evaluation Report Year 3. Performance & Impact Evaluation (P&IE) of the USAID/Uganda School Health and Reading Program: Result 1 School Level Interventions. Prepared for USAID/Uganda. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 17 Cluster 3 learners were seen only at the end of P1. After one year of the intervention, we found some positive and significant effects in letter sound identification, as a result of SHRP, in Lusoga, and Ngakarimojong of around 1 letter. The number of learners unable to identify at least one letter sound was also reduced in Lusoga, Lugwere and English. The treatment group shows a small advantage relative to the control group when asked to identify the direction in which text should be read (print orientation). There is no clear trend in the effects on more advance literacy skill subtasks. Finally, no differences in the impact of SHRP by gender were identified for any of the clusters. Results in comparison to targets According to the SHRP Performance Management Plan, reading improvements are demonstrated if learners can read 20 words per minute of an oral reading fluency passage in the local language after two years of schooling (end of P2). Table 7. Cluster 1 and 2: Oral reading fluency results after two and three years in program Cluster 1 Language After 2 years in program (P2) (wpm) After 3 years in program (P3) (wpm) Cluster 2 Language After 2 years in program (P2) (wpm) Luganda 7.5 17.5 Lebacoli 1.5 Leblango 2.3 4.2 Lugabarati 1.6 Ateso 1.5 7.6 Lumasaba 0.4 Runyakore/Rukiga 7.7 20.2 Runyoro/Rutooro 5.7 Source: NORC calculations using EGRA data collected by RTI Table 7 shows the average oral reading fluency scores for clusters 1 and 2 by language. Oral reading fluency score averages are below 20 wpm for all languages at the end of P2. Average fluency of 20 wpm was reached only by one language, Runyakore/Rukiga, and at the end of P3. In Table 8 we show the oral reading fluency targets to be reached at the end of P2 for each of the languages in clusters 1 and 2 as presented in SHRP PMP of March 2016. The rows shaded in gray display the levels actually reached by the learners at the end of P2. The targets were reached by the Luganda learners and, in less extent, in Runyakore/Rukiga, where the target was met by the girls but not by the boys, and girls outperformed boys in five of the seven districts. In all the other languages, the percentage of students that read 20 wpm is below the desired threshold. Table 8. Cluster 1 and 2. Oral reading fluency targets and results at the end of P2 Cluster 1 Luganda Leblango Ateso Runyakore/Rukiga M F M F M F M F Baseline 0.0% 0.0% 0.2% 0.0% 0.0% 0.0% 0.0% 0.2% Target (% reading 20 wpm) 15.0% 15.0% 5.0% 5.0% 5.0% 5.0% 15.0% 15.0% End of P2 14.7% 20.2% 3.5% 3.8% 1.5% 1.8% 11.7% 17.2% Cluster 2 Lebacoli Lugbarati Lumasaba Runyoro/Rutooro M F M F M F M F Baseline 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.2% 0.4% Target (% reading 20 wpm) 10.0% 10.0% 10.0% 10.0% 10.0% 10.0% 15.0% 15.0% End of P2 4.8% 2.0% 4.1% 2.4% 0.5% 0.5% 9.0% 11.6% Source: NORC calculations using EGRA data collected by RTI FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 18 Stakeholder perceptions of reading performance. Stakeholder perceptions do not conform to the quantitative findings from the impact evaluation, which show little impact of SHRP on reading performance. The majority of key informants (94%, or 78 out of 83) believe that reading performance has improved since the inception of the SHRP project. We asked respondents, as a result of SHRP, “to what extent has reading improved: significantly, somewhat or a little?” District and national stakeholders stated performance has increased significantly based on classroom observations during regular Monitoring and Support Supervision visits. More than half (54%, 26 people) believe reading has improved “significantly.” Another 37% (18 people) believe reading improved “somewhat.” Seven percent (8 people) believe reading has improved a “little.” Overall, more national respondents, who are more cognizant of the EGRA results, answered “somewhat” or a “little” as compared to district respondents: 62% of national versus 33% of district respondents rated the extent of reading improvement at the two lower levels. Of the 22 literacy teachers interviewed, more than 90% (20) think reading performance of their learners has improved. Fifty percent (11 out of 22) of teachers base this perception on use of the Continuous Assessment Monitoring (CAM) form. They assert that there are more students able to read though not fluently. However, none of the teachers could accurately describe how the CAM form is used (e.g., stating that the triangle represents meeting the competency rather than exceeding it, as described in the teacher guide). The remaining 50% of teachers believe reading has improved based on observation of class participation (30%) or informal assessments (20%) via weekly assessments recommended in the teacher’s guide or marks in exercise books. While some stakeholders, primarily national, have seen the impact evaluation and/or the actual EGRA results (SHRP staff), most stakeholders are basing their opinions on reading improvement on classroom observation, not quantitative data related to the learners observed. While the SHRP Teacher Guide includes a discussion and guidance on carrying out an end-of-term assessment, there is no evidence that end-of-term assessments are a routine part of the program. End-of-term assessment is not emphasized in the training, and there is no actual tool provided for carrying them out. No respondent was able to cite end-of-term assessment data to substantiate their perception of reading performance. Without such data, teachers, school administrators, students, parents, and district/national officials do not have a quantitative basis for judging specific learner reading performance. They can only base their opinion on observational data, which is an insufficient basis. The Learner Check has helped surface the reality that while children appear to be reading given the high level of participation in the classroom, when they are tested individually via the Learner Check, they are much less able. This implies that students are following the teacher and repeating what they hear in class (e.g., choral reading), but are unable to decode letter sounds and words in printed text on their own. District and national stakeholders report that when they conduct the learner checks in classrooms and share the results with teachers, it is often an "eye awakening experience" Factors contributing to improved reading achievement. The three key factors respondents associated with perceived improvements in reading achievement are, in order of most frequently cited:  Teachers trained in EGR methodology and local language (63%, or 61 respondents)  Availability of teaching and learning materials (44%, or 43 respondents)  Increased Monitoring and Support Supervision per the SHRP methodology (34%, or 33 respondents) FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 19 Respondent perception of factors inhibiting reading outcomes Top factors  Lack of parental support (60%, 58 respondents)  Student absenteeism (47%, 46 respondents) Next ranked factors  Teacher transfers (32%, 31 respondents)  Low teacher motivation/attitudes (32%, 31 respondents)  Large class sizes (27%, 26 respondents)  Poor head teacher support (26%, 25 respondents)  Insufficient coaching/MSS (25%, 24 respondents) Notable factors  Insufficient EGR training (22%, 21 respondents)  Teacher absenteeism (22%, 21 respondents) See SFI discussion below as it appears to have a positive effect on school environment & learning Respondents state that the training, pupil books, and scripted lessons have helped teachers to improve reading instruction; and that increased and improved Monitoring and Support Supervision by the CCT, District Inspector of Schools and SHRP increases accountability of teachers and head teachers to regularly adhere to the program. On the other hand, few respondents cited Continuous Professional Development, teacher commitment, and supportive head teachers as positively contributing to reading outcomes. When probed, respondents stated that Continuous Professional Development is infrequent and of varying quality; low morale and poor attitudes is prevalent among teachers; and head teachers are not as well trained in the methodology as teachers and CCTs. Factors inhibiting improvements in reading outcomes. Key informants, in particular national and district stakeholders as well as head teachers and literacy teachers, perceive lack of parental support and student absenteeism to be the most significant factors contributing to low reading achievement. This finding is consistent with student survey data, which found that about 50% of learners were absent at least one day in the week prior to the assessment. The two are obviously linked as a lack of parental support fuels student absenteeism. The next ranked cluster of factors perceived as important barriers to reading improvements included teacher transfers, low teacher motivation/attitudes, large class sizes, unsupportive head teachers (do not monitor regularly, support teachers with teaching aids, or provide constructive feedback), and insufficient coaching and monitoring and support supervision. Low teacher morale and poor teacher attitudes were consistently cited as the key factors affecting teacher absenteeism and teaching quality. Key informants also perceived the low value placed on education by the public reinforcing these factors among teachers, as do difficult job conditions such as overwhelming class sizes, lack of interest among supervisors, or overall lacking school environment including basic supplies. The degree of orthography establishment and complexity appears to be a key factor inhibiting reading outcomes. For example, Cluster 1, which has existing and well-established orthographies experienced greater reading improvement. Cluster 2 languages, however, were newly developed orthographies and therefore more challenging for teachers. On complexity of the orthography: some USAID and SHRP stakeholders said that Luganda, Runyakore-Rukiga and Runyoro/Rutooro, all of which met PMP reading targets, have simpler orthographies and are more transparent than the other languages. A2. Conclusions  Among learners in Cluster 1 and 2, SHRP Result 1 interventions generally had a positive effect on reading outcomes, although, in most cases, it is not statistically significant. In terms of real impact, the program showed effects higher than control groups in five languages with no effect on three languages (Leblango, Lugbarati, and Lumasaaba).  While some stakeholders are aware of the EGRA results, most key informants base their opinions on reading improvement on classroom observation and not quantitative data. The end-of-term assessment as discussed in SHRP EGR materials for week 12 doesn’t currently yield measures that FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 20 MoES Recommendations  Stress the need for improved and quantitative assessment at the school level  Stem teacher transfers in intervention schools  Address school level deficiencies, e.g., large class sizes, insufficient desks, lack of materials and supplies. Help districts access the Universal Primary Education capitation grants to cover some of these costs.  Help engage influential voices in parent/community outreach USAID Recommendations:  Coordinate CHC support to SHRP with participatory community mobilization strategy development and messaging  Advocate for MoES attention and resources to address school-level deficiencies. allow teachers, school administrators, students, parents, and MoES officials at the district and national level a firm basis for judging learner reading performance. This creates a wide disparity between perceived reading progress and the reality identified in the impact evaluation.  Better results would be achieved if SHRP ramped up its efforts to increase parental support and involvement which would in turn help to reduce student absenteeism, with special attention needed to encourage parents regarding education of girls.  Teacher transfers continue to thwart program success and additional ways to overcome this at both the decision-making and school levels are needed.  The many factors affecting low teacher morale and poor teacher attitudes that contribute to reduced reading outcomes merit attention. According to key informants, these include building respect for the teaching profession at the primary and secondary level among parents, the community, and teachers themselves. This needs to part of the parent and community outreach discussed above, as well as through appreciation and recognition initiatives that can be launched at national, district, local, and school levels. While this might also positively impact head teacher engagement, additional effort is needed in training head teachers in coaching, mentoring, monitoring, and support supervision. School level deficiencies – large class sizes, insufficient desks, lack of materials and supplies -- need MoES attention. (See also the conclusions and recommendations that follow on SFI and its ability to improve the school learning environment.) A3. Recommendations To accelerate reading outcomes, SHRP should address factors associated with low performance and should enhance the use of assessment for tracking results. Specific recommendations are as follows: 1. Assessment. Simplify assessment protocols to increase the use of assessment and disseminate findings to stakeholders at all levels. Suggested revisions include: Continuous Assessment Monitoring. Simplify the instrument to include key competencies that are measured on a periodic basis. Clarify what the three strokes represent so there is common interpretation on how to use them and consistency in use and action indicated. For example: Level 1, attaining the competency; Level 2, is almost attaining the competency; Level 3, needs remediation. Train teachers to differentiate instruction based on assessment results, for example, Level 1: provide enrichment activities and engage in peer support; Level 2: provide review; Level 3: provide remediation to address weaknesses. Learner Check. Train teachers in how to use the tool to monitor reading fluency and disseminate results periodically to head teachers, parents, DIS, and CCTs. End-of-term assessment generating meaningful data for all stakeholders. Although an end-of-term assessment is included in the teacher’s guide, it is not FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 21 widely in use, and doesn’t generate reading performance data that is meaningful to schools, parents, and learners, which is needed to reduce the disparity between perception and reality as revealed by the impact evaluation results. While SHRP doesn’t want to reinforce the “exam culture” that prevails in Uganda, it needs to find a solution so that schools, students, and parents are clear on early grade reading performance. 2. Parent/community mobilization. Develop a strategy for sensitizing parents and communities on the value of education in addition to the value of EGR in local languages. Stress the need for regular attendance and the role of parents in ensuring children go to school equipped to learn – with sufficient sleep, nutrition, and supplies. Highlight the benefits of girl’s education. Promote reading￾related activities in the home and community. Use a participatory and engaging methodology for community mobilization (versus the current focus on reading talking points without engagement and discussion); draw upon the community mobilization expertise of the USAID-funded Communication for Health Communities. (See also discussion under Q3.) 3. Teacher morale and attitudes. As part of the community mobilization activities, build respect for the teaching profession among parents, the community, and teachers via the community mobilization activities. Help the MoES develop district-level appreciation and recognition initiatives that can be undertaken at district, local, and school levels. Provide training for head teachers in coaching, mentoring, monitoring and supportive supervision. Advocate for MoES attention and resources to address school-level deficiencies. 4. Teacher transfers. Continue advocacy to reduce teacher transfers. B. RESULT 2 HEALTH B1. Findings SHRP KAP results. Per the SHRP KAP baseline and endline assessments in June 2013 and June 2015, SHRP reported the following improvements in HIV/AIDS knowledge, attitudes, and practices over a two-year period: Table 9. SHRP Achievements in HIV/AIDS Knowledge, Attitudes, and Practices Indicator Baseline 2013 Target Endline 2015 Proportion of learners with comprehensive knowledge about HIV/AIDs 22% 35% 27% Proportion of teachers with comprehensive knowledge about HIV/AIDs 64% 70% 65% Percentage of learners who know about the 3 main HIV prevention methods 58% 75% 56% Proportion of learners aged 15-19 years that have never had sex (primary abstinence) 75% 82% 77% Proportion of primary and post primary learners who correctly answer at least 75% of HIV and AIDS knowledge questions on KAP survey 65% 80% 70% Bold = emphasis added None of the targets were met, with some significantly short of goal, most notably the three related to learner knowledge on HIV/AIDS, which are at 76%, 75%, and 88% achievement. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 22 P2 Learner, Kasimeri Primary School, Moroto Impact evaluation results. The impact evaluation of R2 Health activities carried out by NORC under the P&IE contract only focused on interventions in primary schools 4, and concluded that the impact of SHRP in treatment schools as compared to control schools was:  Positive but moderate on general and comprehensive knowledge indicators, with an increase of 5-7 percentage points among SHRP schools as compared to control schools, and with boys and older learners scoring somewhat higher than girls and younger learners  Insignificant in treatment schools as compared to control schools on: o Prevention knowledge o Attitudes, stigma, and discrimination o Behavior, except for sex among learners reporting to be sexually active over the prior 12 months; in this case, SHRP treatment schools were almost 15 percentage points lower than in control schools  Positive on HIV/AIDS discussions at school with teachers, which was 17.6 percentage points higher in SHRP schools than in control schools Stakeholder perceptions. When asked the question whether there have been improvements in learner HIV/AIDS prevention knowledge, attitudes, and practices, the ability of key respondents to answer drops noticeably. Out of 81 respondents interviewed at a national level or in districts that have received R2 Health interventions, less than half (46%, or 37 respondents) believe there have been improvements in learner HIV/AIDS prevention, knowledge, attitudes, and practices; and the rest (54%) could not offer an opinion. Some stakeholders commented that they wish they had the information to answer the question. Stakeholders with knowledge of R2 Health are, like the District Inspector of Schools in Lira, generally quite enthusiastic about its perceived positive and even transformational impact on school health and overall school environment, in particular via the School Family Initiative (SFI) programming. The top concrete examples of improvements cited by stakeholders in schools with R2 Health SFI programs were:  Better overall school environment: There is more open communication on health and personal issues, and students are comfortable speaking more freely about topics that formerly were perceived as taboo and not discussed either at school or at home. Through the SFI programming, learners have come to have higher self-esteem and greater confidence to speak up and seek guidance from both adults and peers.  Improved HIV/AIDS knowledge: Respondents perceive that there is increased HIV/AIDS knowledge made possible through the PIACSY training, the SFI handbook with scripted lesson plans, and the after school sessions with students and “teacher-parents.” 4 Sample data for post primary schools has problems and, therefore, was not used by NORC in the IE due to data collection issued discussed with USAID. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 23 “SFI is a very cost effective way to create HIV/AIDS and health awareness, if taken seriously, and can yield very good results. A positive HT can also disseminate the same knowledge throughout the community, as can the child, as ambassadors to the community. SFI is the best approach to get the most reach. Teachers get knowledge from the PIASCY and SFI materials, and translate it to local situations. The Secondary Education Department is now using this approach elsewhere, e.g., in sciences, mathematics, training in PTCs.” -- MoES Official  Attitudes: Respondents cite as an example of reduced stigma the changed perceptions around HIV+ persons and increased willingness to be tested and disclose status.  Practices: Respondents state that learners engage in healthier behaviors, most notably abstinence. Two secondary schools reported a steep reduction in teen pregnancy, e.g., from approximately 8 cases per term to it being a rare occurrence. Similar to the disparity with the EGRA data and stakeholder perception, the KAP data and respondent perception are also at variance in some areas, e.g., the effect on HIV/AIDS knowledge is less supported by the quantitative data. Factors contributing to improved health outcomes. Respondents repeatedly cited the positive impact of SFI in creating perceived improved outcomes. Many respondents noted the improvement in the quality and impact of SFI now that there is a handbook with scripted sessions for “teacher-parents” to use in leading the ten weekly sessions (versus former dependence on teacher preparation and knowledge). A USAID respondent noted that “Scripted lesson plans are important so that sessions are not dependent on teacher knowledge [and] Health Centers are now using the scripts for their own purposes.” Because the program is co-curricular and not a required part of the school timetable, respondents also say that head teacher leadership is pivotal in realizing the perceived benefits of SFI in school health and overall positive climate by making sure weekly sessions are held. Respondents also cite their belief that SFI has led to increased awareness of HIV causes and prevention methods, leading to positive health outcomes such as increased abstinence and testing. The use of peer educators in SFI programming is also cited as a factor leading to positive health outcomes: learners respond well to peer messaging, and use of peer educators expands the pool of people in a school who take quite seriously their responsibility for promoting healthy practices and creating a safer and more positive school environment. Learners and teachers alike mention the willingness of learners to speak up and/or take action to disrupt risky behaviors. Another factor is the school venue itself: as a USAID respondent said, “To reach a young population [on health], it’s much easier and cheaper to do so in schools…you can reach youth easily in the structured environment of schools, it’s low-hanging fruit.” This was echoed in the comments in the adjacent box. The perception that SFI is a low-cost way to achieve positive outcomes in the school environment is corroborated by the small amount of funds used by SHRP for SFI. On a per school annual basis, it represents just under $570. Factors inhibiting desired outcomes. The most commonly cited factors undermining positive potential outcomes of R2 Health activities are:  Low teacher morale and/or poor teacher attitudes which mean fewer or less engaged parent￾teachers  Lack of head teacher leadership which tends to be a death knell for SFI programming because as a co-curricular program it can and is easily displaced by other activities, most commonly it appears by athletic events  “Timetable constraints" which again refers to the co-curricular, optional nature of SFI programming  Persistent parent and child beliefs in myths that contradict PIASCY and SFI content FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 24 B2. Conclusions The conclusions we draw from these findings are:  Again, there is a disparity between the quantitative findings -- KAP and impact evaluation findings -- and the perception of benefits from SHRP programming.  However, data and perception do align in several important ways: o Both the quantitative findings and respondent perception indicate that SHRP has had positive if moderate impact on HIV/AIDS knowledge. o Both the quantitative findings and respondent observations align around the reduction in sex among learners in SHRP schools -- almost 18 percentage points lower than in control schools, which tends to corroborate respondent comments on reductions in teen pregnancy where SFI programming is strong. o SHRP’s positive impact on increased HIV/AIDS discussions in schools with teachers (almost 18 percentage points higher in SHRP primary schools than in control schools) which aligns with respondent perception that SFI programming has created an improved and more open and safe environment in schools.  SFI programming is a low-cost way of reaching way of reaching youth, teachers, and to some extent communities, with accurate information on HIV/AIDS and health; and for creating a more open school environment which encourages attendance and learning. There is a need to make the program less dependent on situational leadership and less subject to postponement  There were too few references to the role of SHRP support to school-based Guidance and Counseling or referral information to lead to any conclusion on their contribution to health outcomes. B3. Recommendations Co-curricular activities can be made a required part of the MoES thematic curriculum, and thereby a required element of a school timetable. Given the common perception among those familiar with SFI that it has a positive impact on health outcomes and the school environment, it merits further study to determine its real effects. This can inform scale-up efforts under LARA (as Uganda Kids Unite) and shed light on whether and how to adopt SFI as an official part of the co-curriculum. Should SFI become a required part of the curriculum, SHRP could assist the MoES in developing an SFI policy including the:  Training requirements for pre- and in-service teachers via PTCs  Sustained provision of SFI materials (handbooks and supplementary materials)  Integration into the support supervision system FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 25 EVALUATION QUESTION 2 How was SHRP’s capacity building approach for teachers perceived to facilitate improvements in reading instruction? A. RESULT 1 READING Capacity building model. SHRP’s capacity building approach for improved early grade reading and transition to English is based on a cascade model that includes the development of teaching content and pedagogy, initial and refresher training, and ongoing support for classroom teaching of reading through continuous professional development trainings and monitoring and support supervision. The project has developed: a scope and sequence of reading in 12 local languages and English; 52 pupil primers and 52 teachers’ guides with scripted lesson plans; supplementary materials in English; and a cadre of master trainers and district level trainers to train and provide ongoing support to teachers. As of April 2016, SHRP has trained 14,210 teachers, 9,277 head teachers and 659 CCTs in EGR methodologies and leadership. A1. Findings Stakeholder perceptions. Among stakeholders asked, “How effective was SHRP’s capacity building approach in improving reading instruction,” 70% (55) of respondents stressed it was effective, 21% (17) stated it was very effective, and 9% (7) thought it was not very effective. Many stakeholders noted the inherent challenges associated with cascade capacity building model, particularly when implemented on a large scale. The majority of stakeholders believe the approach has improved teacher competencies in creating learner-centered, participatory reading classes. As evidence, national, district and school-level stakeholders cited observing a greater percentage of teachers follow the scripted lesson plans; use the pupil books as a teaching aid; apply active learning methods; develop teaching aids and lesson plans in advance of classes; and model examples through application of the “I do, we do, you do” methodology. The evaluators also observed during classroom observations improvements in teachers and learners' ability to utilize the textbooks and scripted lessons compared to during the mid-term evaluation. SHRP PMP data PMP data shows an increase in trained teacher performance per SHRP-set standards5 for the years with sufficient samples to yield reliable results: Per Table 9, 27% of teachers conducted reading lessons in accordance with set standards in 2014; 52% in 2015; and SHRP anticipates 64% in in 2016. Table 10. Trained Teachers’ Classroom Performance 2014 2015 Number of teachers trained 4,197 3,652 % of observed teachers conducting reading lessons in accordance with the set standards 27% 52% Source: SHRP PMP data Challenges: Stakeholders at all levels reported numerous challenges with implementing the SHRP EGR cascade capacity building model, to wit: 5 SHRP’s PMP defines set standards as: teacher conducts 30-minute reading lesson; teacher has appropriate language teaching guide; all learners are reading from textbook or printed material; teacher taught the same lesson as indicated in the lesson plan or script; and, lesson was conducted in the local language. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 26 Respondent perception of challenges with the SHRP Capacity Building Model Top factors  Irregular CCT visits: 30% (29 respondents)  Difficulty with instructional materials: 25% (24 respondents); and 59% of EGR trained teachers (13 of 22)  Insufficient training: 25% (24 respondents); and 27% of EGR trained teachers (6 of 22) Next ranked factors  Lack of Head Teacher support: 19%, 18 respondents  Insufficient CPDs: 18%, 17 respondents  Difficulty preparing lesson plans:16%, 15 respondents; and 32% of EGR trained teachers (7 of 22)  Quality control with Cascade model: 16%, 15 respondents  Using CAM: 27%, 26 respondents; and 32% of EGR trained teachers Top factors per EGR trained teachers  Difficulty with instructional materials  Difficulty preparing lesson plans  Using CAM  Insufficient training  Teacher mentoring and coaching. CCTs rarely complete their intended two visits per term due to demanding workloads and extensive coverage areas. Similarly, head teachers and deputy head teachers do not regularly provide supervisory coaching and mentoring at the school level. Teachers reported that head teachers either cannot or do not provide adequate follow-up support, e.g., much-needed assistance with lesson planning and scheming, understanding and interpreting steps in the teachers’ guide, and role modeling sessions. The “Blue Book,” which is used by head teachers to record feedback, was often blank, very recently filled in (i.e., done for our visit), or filled in with off￾target and non-substantive comments. The District Inspector of Schools tries to visit schools in their districts once per term.  SHRP instructional and pupil materials. o Most teachers said that SHRP instructional materials require a confusing level of cross￾referencing (versus a sequential presentation of requirements) within the teachers’ guide and between it and the pupil books. We observed teachers and SHRP Field Assistants struggling to locate the correct page during classroom visits. Principals and district officials cited cross-referencing as a challenge, especially given the shortcomings in the training sessions and follow-up support. One CCT said the complex cross￾referencing to follow the steps increases the difficulty of training untrained teachers. If trained teachers are confused by the materials, this negatively affects their ability to train peers, which is part of the cascade model. o Teachers also noted difficulties and inconsistencies in the learner books between the theme, story, illustration, and vocabulary words. We noted these as well. MoES officials say there are errors in local language in some of the books, and one pointed out a particular error in the Ngakarimojong pupil book. These issues exist in a variety of languages and grades. o There are three issues to address in the English teacher’s guide for all grades: 1) English teachers said that it would facilitate cross-referencing if the guide listed “the term, the week, and the day” of the lesson similar to the local language books, rather than merely the “term and the week.” 2) Days and steps should be clearly separated for each day of the week. Currently, the steps are not separated by day since Day 3 is a repeat of Day 1 and Day 4 is a repeat of Day 2. Separating the lessons by day and steps for each day would help teachers understand that they should cover all steps on each day and not divide the steps up between Day 1 and 3, or Day 2 and 4. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 27 3) English teachers state that there are too many steps to teach in one day. When the issue was raised during training in one district, the trainer suggested breaking up the steps in Day 1 and Day 3 rather than trying to teach all in one day. That would mean that some teachers would fall behind. PTC principals and CCTs state that teachers should not break up the steps since they are timed and may be completed in one day, and recommend more coaching or revising the teachers’ guide. To avoid confusion, the steps for each day should be separated for each day of the week, the number of steps should be minimized, and more coaching is needed to help teachers transition more quickly between steps.  Teacher training. Teachers said that district training sessions were overcrowded and overloaded with too many topics in too short a timeframe for the teachers to grasp the material much less master it. There was not enough time to cover the orthography needed for teachers to feel confident in teaching local language lessons; or sufficient time to other elements of the methodology such as continuous assessment (administration, scoring), transitions, and preparing lesson plans or schemes of work.  Continuous Professional Development. Following SHRP-sponsored Monitoring and Support Supervision visits (resource-intensive teams with MoES officials and SHRP staff who review a teacher’s lesson plans and schemes, observe a class, and then provide feedback), CCTs are supposed to use the feedback to develop Continuous Professional Development sessions to a cluster of teachers. Respondents reported that this happens irregularly and, as a result, teachers rarely benefit from hands-on coaching or demonstration sessions. As a result of the insufficient training, Continuous Professional Development Trainings and follow-up support, teachers have faced difficulties with preparing lesson plans, following steps in the teacher’s guides, and conducting continuous assessment. Teachers reported that lesson planning is taxing due to difficulty with understanding the steps, cross-referencing and the time required to translate the English scripts into the local language. Non-native speakers face the most difficulties. In terms of assessment, about 70% of teachers observed during classroom observations did not use the Continuous Assessment Monitoring form during instruction and none of them could accurately describe how to assess their students. Teachers claim continuous assessment procedures were not adequately covered during the district trainings.  Dilution. Stakeholders report that training of trainers (TOTs) and training of teachers are not as effective as the master trainer sessions due to the dilution of understanding and competence at subsequent levels. Availability of quality trainers also affects the quality of training as well as the follow-up Continuous Professional Development, which becomes more challenging as EGR is scaled up under LARA and Global Partnership for Education/Uganda Teacher and School Effectiveness Project. Classroom observations. Our classroom observations revealed the following:  70% of teachers observed generally follow the scripted lessons, apply the "I do, we do" methodology, and adhere to time allocations for reading/writing (Literacy 1/2) lessons.  Nearly all teachers, however, skipped the “you do” exercises which create opportunities for pupils to practice in pairs, groups, or individually.  Teachers in low-performing schools placed greater emphasis on vocabulary and reading fluency skills compared to teachers in high-performing schools who spent more time on phonics and reading FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 28 comprehension, as well the other key reading skills (1 of 9 taught phonics in low-performing schools compared to 5 out of 13 in high-performing schools)  Nearly all teachers skipped exercises allowing practice in pairs, groups, or individual practice Teacher recommendations. During interviews, teachers were asked what additional support was needed to help them improve children’s reading performance. Teachers requested additional refresher training and coaching in the following areas:  Basic early grade reading methods  Continuous Assessment Monitoring administration and scoring procedures  Letter sounds/phonics  Orthography, supporting understanding of letter sounds and teaching of writing lessons  Lesson planning, scheming and support with developing teaching aids Teachers also said that their supervisors and mentors (head teachers and CCTs) need more supportive supervision as well as EGR training to provide the support trained teaches need. A2. Conclusions  Most (92% or 72) stakeholders perceive that SHRP’s cascade capacity building model is “effective” or “very effective” in improving teaching competencies in SHRP/EGR methods. Stakeholders were able to provide evidence of observed improvements in trained teachers' reading instruction. Evaluators also note improvements in teachers and learners' ability to utilize the textbooks and teacher guides compared to the midterm evaluation.  Nevertheless, both SHRP’s PMP data and stakeholder comments on challenges within the cascade capacity building model indicate many issues still to be addressed.  Key areas of the capacity building model that need to be addressed include: o Reducing some of the inevitable dilution within the cascade model by better acknowledging absorptive limits of trainers and teachers and giving a laser focus on key topics with more use of demonstration (see also the discussion in Q5 on this topic, on p. 48), in particular with orthography, continuous assessment, transitions, and preparing lessons plans and schemes. o Identifying concrete ways to achieve a critical link in the cascade model, namely providing trained teachers with school-based mentoring, coaching, and support supervision via CCTs and head/deputy head teachers. Both CCTs and head/deputy head teachers need to be better equipped to effectively provide such support, and need to be held accountable to implementing the support. o Similarly, CCTs need to be held accountable for providing Continuous Professional Development sessions, based on areas identified through Monitoring and Support Supervision visits to improve the quality of instruction and fidelity to scripted lessons in the teachers’ guide.  SHRP instructional and pupil materials are impressive but have issues related to user-friendliness (confusing cross-referencing), inconsistency within lessons (theme, story, vocabulary, and illustration), and actual error. This exacerbates the challenges teachers still face in acquiring basic EGR skills such as lesson planning and scheming, using Continuous Assessment Monitoring forms FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 29 during instruction, teaching phonics, handling transitions, and mastering "I do, we do, you do." This is especially concerning given the rush to have NCDC officially approve materials. A3. Recommendations  Better acknowledge absorptive limits of trainers and teachers by trimming down teacher training sessions to truly key topics with more use of demonstration to promote better understanding, in particular with orthography, continuous assessment, transitions, and preparing lessons plans and schemes.  Identify concrete ways to provide trained teachers with school-based mentoring, coaching, and support supervision via CCTs and head/deputy head teachers. This must include (1) better equipping both CCTs and head/deputy head teachers with the mentoring, coaching, and support supervision skills needed to perform these roles and (2) holding them accountable to implementing the support.  Hold CCTs accountable for providing Continuous Professional Development sessions among school clusters, based on areas identified through Monitoring and Support Supervision visits to improve the quality of instruction and fidelity to scripted lessons in the teachers’ guide. Continuous Professional Development should include a variety of EGR methods (scaffolding, multi-sensory, special needs), letter sounds, orthography, lesson planning/scheming, and making teaching aids from local resources, etc. o Increase engagement of PTCs in ensuring CCTs carry out their functions, including school mentoring visits and Continuous Professional Development sessions. o Have teachers who demonstrate completion of required competencies serve as peer mentors/trainers within their school clusters.  Respond to concerns among stakeholders, especially teachers, regarding SHRP instructional and pupil materials user-friendliness (confusing cross-referencing), inconsistency within lessons (theme, story, vocabulary, and illustration), and actual error. Recognize that this exacerbates the challenges teachers still face in acquiring basic EGR skills; and is critical to resolve prior to full scale-up by LARA and the Global Partnership in Education/Uganda Teacher and School Effectiveness Project and the uptake of the method among PTCs for preservice training (this would greatly reduce the amounts of coaching and refresher training needed. Refer to USAID/Kenya Tusome and USAID/Kenya PRIMR6 project for examples of well-organized scripted lesson plans that are user￾friendly. Support development of teaching aids (e.g., audio recording of letter sounds, alphabet charts, orthography guides); and promote use of local materials for teaching aids. B. RESULT 2 HEALTH Background. Over the four-year performance period under review, SHRP trained 10,202 teachers in a total of 1,651 schools (1,557 primary schools and 94 secondary schools) in 17 districts at sessions of 5 days’ duration. This represents 72% of the cumulative target over the four years. Training included: 6 See https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=498 for sample scripted lesson plans from the USAID/Kenya PRIMR Initiative. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 30 Sampling of SFI Handbook Scripted Lessons:  STI, HIV and AIDS  How HIV and AIDS affects children  Dangerous situations that lead to HIV infection  VCT  Teenage pregnancy  Cross-generational and transactional sex  Effects of alcohol and drug abuse  Homosexuality  Life Skills; Self-awareness; Self-esteem  Helping children choose to abstain  Body changes in girls and boys  Caring for our private parts  Responsible and irresponsible sexual behavior  Cultural beliefs and practices  Friendship formation  Resisting negative peer pressure  An initial training on Enhanced PIASCY for the 1st cohort of schools in 2013 with the head teacher and four teachers, with a total of 4,055 participants representing 11 districts and 822 schools (772 primary and 50 secondary).  A second training on Enhanced PIASCY for the 2nd cohort of schools in 2014 with the head teacher and three teachers selected by the head teacher with a total of 3,876 participants representing 17 districts and 829 schools (785 primary and 44 secondary).  A sustainability-oriented training in 2016 covering five district’s 588 schools (551 primary and 37 secondary), and 2,000 participants: 89head teachers designated as “Zonal head teachers” (in effect, peer leaders among the 5-6 schools in their zone) and the District Inspector of Schools in each of the five districts were trained as trainers in a training of trainers session beforehand and then served as the lead trainers in districts other than their own. In the training of trainers, Zonal head teachers developed work plans for implementing and supporting SFI in their zones. The trainings used a new September 2015 “Enhanced PIASCY Teacher’s Reference Manual for Primary Schools” with updated age-appropriate information on topics similar to those in the scripted lesson list, which more proactively “aids the teacher in delivering child-centered, participatory activities that can be easily integrated into subject matter lessons.” (p. 10). The training sessions used a broad mix of participatory methodologies so teachers would better integrate the information and also be better equipped to use them. Based on a recommendation from the mid-term evaluation, SHRP created an SFI handbook with scripted lesson plans covering a full school year. The handbook also contains a registry of learners in each “family,” weekly attendance forms, and work plan templates for “parent teachers” to plan weekly sessions for each term. Scripted lessons cover the topics in the adjacent box. SFI handbooks were distributed to each teacher in program schools, along with Guidance Counseling Registers for each school. B1. Findings KAP data. Per Table 9 above on the results of the baseline and endline KAP surveys, there has been a modest positive increase in HIV/AIDS knowledge among teachers and students. Respondent data. Respondents perceived R2 Health capacity building activities positively. Again, fewer respondents have experience with or knowledge of R2 Health activities: of the 60 respondents that replied to the question whether there have been improvements in teacher HIV/AIDS prevention knowledge, 52% (31 people) believe there has been, 5% (3 people) believe there has not been, and 43% (26 people) could not offer an opinion. Evidence cited to substantiate improved teacher HIV/AIDS prevention knowledge included:  Regular use of SFI handbook/register to plan and record “family” sessions  Integration of health topics into regular classes  Integration of health themes into co-curricular activities (music, dance, drama, sports)  Greater confidence and comfort teaching and discussing sensitive health topics  Greater willingness to be tested for HIV and to disclose status FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 31 Head teacher, Agweng Secondary School, proudly standing in front of his school time table that shows PIASCY as a fixed element. Respondents answering the R2 Health questions tended to include all the health trained teachers, about half the head teachers, one-third of the district officials, and about half the national stakeholders interviewed. Respondents with firsthand knowledge of the SFI program (such as those quoted below) believe that, where there are well-functioning SFI programs, it has (1) led to improved HIV/AIDS and health-related knowledge, attitudes, and practices; and (2) the participatory processes used have had a transformational and positive effect on the overall school environment and thereby helped meet reading and health outcomes. Stakeholders said that where the SFI program is working, it creates an openness and better overall school climate/environment where kids feel safer, more supported, and more confident; that this in turn translates into better attendance and a more conducive environment for learning; which in turn leads to better reading and health outcomes. The District Inspector of Schools in Lira has come to this conclusion and as a result (1) created a consensus to make SFI required in the 93 schools in his district and (2) incorporated SFI into the district inspection tool. He regrets that R2 Health did not get more attention and was not integrated from the start with reading. Similarly, the Senior Education Officer for Secondary Education in the MoES believes SFI is very effective, and, and should be institutionalized notwithstanding the mainstreaming of SRH into the thematic curriculum, because that won’t match SFI’s “benefits of opening up discussion on sensitive topics, empowering new behaviors, and developing leadership skills… especially for girls.” The head teachers, trained teachers, and learners in schools with well-functioning SFI echoed these perceptions in KIIs and FGDs. More than half the respondents (52% or 31 of 60 respondents) to the question regarding improvement of teacher health and HIV/AIDS knowledge responded “yes.” Examples of ways teachers apply their knowledge include integration of health topics into the curriculum or into co-curricular activities, e.g., music, drama, dance, sports. Respondents also noted that PIASCY training and the SFI materials have increased the confidence, knowledge, and skills of teachers with teaching and discussing HIV/AIDS topics and also increased their willingness to test and disclose their status. For example, a Head Teacher in Katakwi said that as “teacher knowledge has improved… they go for counseling and testing… and are helping learners cope and prevent HIV.” Key issues affecting implementation of the PIASCY/SFI training include 1) lack of materials for “family” sessions other than the scripted lessons, 2) repetition for learners using the same scripted lesson from year to year, and 3) the need for head teacher leadership for SFI to gain traction, bring reluctant teachers along, and for the “family” sessions to take place. As with R1 Reading training, the R2 Health training agenda is over-packed and overwhelms absorptive capacity. One respondent particularly close to the training estimates about 50% absorption of the material, and “if they don’t really grasp it, trained teachers can’t really train other teachers.” The case study of SFI in Lira illustrates the potential of SPI to create positive outcomes. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 32 SFI Case Study: Agweng Secondary School, Lira, A High Prevalence HIV/AIDS Area with Strong District and School Leadership We revisited Agweng Secondary School which at the mid-term evaluation had strong SFI program. Two years later, the program was ever stronger. The HT is now providing SFI guidance to HTs at the cluster level, “mentions SFI at every opportunity” to parents and the community, and is proud to show the school timetable which includes SFI as a required element. We also met with the same two senior head female and male teachers. They have attended every SHRP PIASCY training, and were equally proud of showing the register attesting to the school’s weekly sessions. The DIS attended the last PIASCY training and came away so enthusiastic that he met with all the HTs to gain their commitment to SFI as a required program, and integrated it into the district monitoring and supervision tool. Learners are so engaged that we had 22 attend an FGD intended for 8-10, even after a day of exams. SFI success factors  HT is very supportive, always mentions SFI in meetings, and has trained teachers speak  SFI is an established part of the school program as a co-curricular activity  Team spirit is strong among the staff and learners  Peer educator approach where learners provide support and coaching  Support from district officials who motivate teachers as part of supervisory visits  SMC is very supportive and open on the issues  HIV/AIDS is a major issue in the area Challenges  Getting all teachers on board – ten of 14 teachers participate consistently  Day scholars participate less than boarders. With the integration of sexual and reproductive health into the curriculum, all learners get the information but only the ones attending SFI sessions get the empowerment, support, and leadership benefits  The program gets repetitive and it’s a challenge to “change it up” – peer educators/leaders are given the task of creating new approaches/ideas  Lack of supporting supplementary materials to use in weekly sessions and dilapidated and outdated school signage  SFI takes place after class lessons and teachers complain about longer hours Benefits  Large drop in teen pregnancy and girl dropouts: there used to be about 8 pregnancies per term, now it is rare  Builds life skills, self-awareness, ability to speak up and make better life decisions  Twenty girl learners are going on to UCE (Uganda Certificate of Education) compared to maybe 4-5 in the past  B2. Conclusions The SFI handbook, in particular the scripted lesson plans, has been a critically important adjunct to PIASCY/SFI training, helping ensure that teachers impart accurate information, more comprehensively cover content on each topic, and use structured and participatory processes. The new Enhanced PIASCY Teacher’s Reference Manual is aligned with the SFI Handbook which also makes it easier for teachers to use them following the training. Based on key informant opinions and perceptions, we conclude the following:  Teacher and learner HIV/AIDS knowledge has improved as a result of the PIASCY/SFI training and the SFI weekly sessions.  Teachers appear better able to integrate PIASCY HIV/AIDS and health information into classroom subjects, and more comfortable with treating these sensitive topics in the school setting, which some respondents noted helps prepare them to handle mainstreaming of SRH into the thematic curriculum.  The PIASCY training sessions have used innovative tactics to prepare for sustainability in five of SHRP’s 31 districts (three of which will continue to be supported under DREAMS). This includes training head teachers in PIASCY/SFI training and helping Zonal head teachers develop concrete work plans for providing continued support to other head teachers in their zones of responsibility. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 33 Adolescent Learner Focus Group, Agweng Secondary School, Lira  SFI appears to create a more conducive school and learning environment, and promote sustained student attendance, perhaps especially among girls. Therefore, SHRP’s other 28 districts should also receive SFI capacity building with sustainability features such as used in the 2016 PIASCY/SFI training, e.g., training and follow￾up mentoring and coaching by school-based and therefore accessible head teachers who have been trained as trainers; and monitoring and supervision via district visits and inspection tools.  Many teachers who have attended PIASCY training have been trained two to three times and have had up to three years of experience serving as “parent-teachers.” These trained teachers represent a cadre of knowledgeable teachers that can be used to replicate and/or strengthen SFI elsewhere.  Similarly, learners who have been peer educators can help initiate peer educator initiatives in other schools.  Future SFI capacity building should: o Better address absorptive capacity limits, e.g., with more time allocated to the training, better treatment of fewer topics, more practice time, etc. o Identify whether training head teachers to be PIASCY trainers in fact solves the follow￾up training of teachers who do not attend the training and/or the follow-up mentoring and coaching requirements of trained teachers; or if additional support is needed  Schools with SFI programs need supplementary learning materials for use in weekly SFI sessions.  Schools need posters and other signage with updated messages that reinforce the concepts and themes in the PIASCY training materials and SFI Handbook. Current “talking compound” signage is often a hazardous low-lying mass of bent steel with messages that are either no longer able to be read or highly outdated. These hazards should be replaced with safer signage and current messaging.  SFI capacity building in the long run as well as the need for annually provided SFI Handbooks can only be addressed if the MoES adopts SFI as a formal part of the co-curriculum. B3. Recommendations  To capture SFI’s perceived positive impact on creating a more conducive school and learning environment for R1 Reading objectives, further study of the program’s concrete effects is FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 34 warranted to shed light on scale-up under LARA and potentially in SHRP and GPE districts. This needs to be  Any scale-up effort should: o Include sustainability features such as used in the 2016 PIASCY/SFI training (use of head teachers as trainers and subsequent mentoring and coaching, district follow-up using tools in which SFI is integrated). o Consider use of experienced PIASCY trained teachers and peer educators o Build on SHRP lessons learned regarding absorptive capacity and need for follow-up training of other teachers as well as mentoring and supervision that is as close to the school as possible (so it is frequent and sustainably affordable)  SHRP should proactively better address the need for supplementary learning materials for use in weekly SFI sessions.  SHRP or other relevant USAID projects, such as perhaps CHC (Communications for Healthy Communities) should address the need for new signage at schools with updated messages that reinforce the concepts and themes in the PIASCY training materials and SFI Handbook. The hazardous and illegible “talking compound” signage should be removed. A competition involving schools to coin new messaging would reinforce all elements of SHRP. SHRP notes that CHC is finalizing guidelines for talking environments.  MoES should adopt SFI as a formal and required element of the co-curriculum, and commit to the capacity building and annual materials development needed. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 35 EVALUATION QUESTION 3 How was SHRP’s community mobilization approach perceived to facilitate improvements in reading and health outcomes? 1. Findings SHRP’s community mobilization approach. Per the SHPR Cooperative Agreement, under IR 1.3, SHRP is to “design and implement strategies at the national, district, and school levels for effective parental and community mobilization and engagement in promoting early grade learning.” According to the PMP, SHRP works to achieve this sub-IR in four main ways: 1) Supporting local PTAs/SMCs to organize and participate more fully in education and activities that support early grade reading 2) Promoting national reading activities, such as national literacy week and national reading competitions 3) Increasing the proportion of schools participating in community activities that support reading 4) Engaging Local Language Boards in efforts to advocate and support initiatives to improve early grade reading. Table 11 shows targets for each of these indicators and actual results to date. The results are summarized below and juxtaposed against the final performance evaluation findings to determine the extent to which community mobilization efforts facilitated improvements in reading outcomes. Table 11. Advocacy and Support for Reading Increased Indicators Indicators Year Targets Actuals Number of PTAs or similar school structure supported 2013 2014 2015 410 389 840 0 60 1,045 Number of activities to promote reading at the national level 2014 2015 2 2 2 2 Proportion of schools participating in community activities to support reading (reading competitions, reading awareness days, literacy week) 2013 2014 2015 50% 20% 40% 0% 0% 51% LLBs supporting efforts to strengthen early grade reading in local languages 2013-2015 12 12 Numbers of PTAs/SMCs supported. While SHRP fell far short of its 2013 and 2014 targets in this area, it exceeded them considerably in 2015. The actual number of PTAs/SMCs reached in 2013 was zero since the program had not yet put in place mechanisms to facilitate community activity. Outreach to PTAs and SMCs began in late 2014, so only 15% (60) of the targeted PTA/SMCs were reached that year. In 2015 SHRP began to achieve traction in this area, reaching 1,045 PTAs/SMCs, representing 125% of the target. According to SHRP’s Performance Management Plan, activities under this indicator include supporting PTAs/SMCs to organize, meet regularly, and participate more fully in monitoring school quality/governance. Once trained, PTAs/SMCs are expected to increase community participation, advocacy and support for early grade reading interventions in schools. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 36 We conducted focus group discussions with SMC/PTA committees in 11 primary schools. The engagement with SHRP described by these committees largely consisted of brief dialogues with SHRP and district stakeholders during Monitoring and Support Supervision visits regarding the contributions of the SHRP project and parental roles and responsibilities in supporting their children's education and health. Half of the SMC/PTAs with whom we met recalled hearing about stakeholder roles; materials received from SHRP; and the value of early grade education in local language. In two of our SMC/PTA meetings we heard that these meetings also covered how parents are supposed to support their children’s education. Only three of the 11 SMC/PTAs have taken actions as a result of the meeting. They cited the following examples: they conduct more frequent school visits; they fundraise, and they speak to children and teachers about discipline. None of these activities directly relate to improving EGR. Reading promotional activities at the national level. According to the Performance Management Plan, SHRP has conducted two national reading promotional activities in 2014 and two in 2015. In 2014, SHRP provided technical support to the Global Partnership for Education/Uganda School and Teacher Effectiveness Project to incorporate early grade reading into their program, and supported the Ministry planning unit with integrating EGR achievements into the education sector review report. In 2015, SHRP promoted mother tongue day and international literacy day. Key to SHRP community mobilization is its dialogues with the SMC/PTA members during joint Monitoring and Support Supervision visits and Drop Everything and Read (DEAR) Day. Schools participating in community mobilization activities to support reading. According to SHRP Performance Management Plan data, 51% of schools in 2015 participated in community activities to support reading (e.g., reading competitions, reading awareness days). However, less than 10% (6 people) of all stakeholders interviewed were aware of community-wide events. Of the 11 primary schools visited, only two were involved in reading competitions, according to head teachers interviewed. Eight schools received brief talks from SHRP and district officials during joint Monitoring and Support Supervision visits on the contributions of the program and parental responsibilities. These talks involved a small proportion of parents and community members and thus were not considered by respondents as a community sensitization meeting. Language Boards supported. SHRP has supported 12 local language boards, in each of the local languages. local language boards are expected to be the key local champions to advocate for mother tongue instruction among all local audiences, including schools, parents, and the community. Community outreach to generate support for local language is part of the mandate of the local language boards. Local language boards were involved in the development of the orthographies and were expected to work with communities to increase awareness and understanding of the new orthographies. However, their outreach role has been compromised because their current structure does not give them access to the facilitation funds required to do this, either from SHRP or the MoES. Performance evaluation findings. As noted above Question 1, lack of parental engagement is perceived to be the top inhibitor of early grade reading outcomes. Parental involvement in reading activities has been minimal due to lack of engagement with the majority of parents and community members. Only a small proportion of SMC/PTA members have been reached, and the majority of the members we met with had not conducted sensitization meetings with other parents because they themselves had not been fully sensitized or trained to perform their roles. Head Teachers also appear to not have been facilitated or motivated to conduct reading competitions and community-wide events to promote early grade reading. Seventy percent (52 out of 75) of respondents report there are still challenges with obtaining parental support in SHRP reading activities. Although it has improved, there is still resistance to teaching in the FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 37 local language and the new orthography. Parents prefer that their children learn English. In a few cases, parents withdrew their children from SHRP schools and enrolled them in private schools when the local language was introduced. Once they began to witness improvements in children’s reading skills, they re￾enrolled them. Parents in many pastoralist communities (such as in the North) are not literate themselves and often do not place a strong value on education. Therefore, they do not monitor and/or promote their children’s school attendance or attend school events often. Frequently, parents prioritize household labor over school, especially in the planting and harvesting seasons. The declaration of “free” Universal Primary Education has also affected parental contributions to schools. Parents believe they should not have to contribute anything, even when in reality this means their children will be at school without basic scholastic supplies and food. Schools grapple with raising needed funds, typically not allowing learners to sit for exams until the termly fee is paid. In schools where SMC/PTAs have been sensitized, some of the members who are parents have begun to show signs of greater participation in educational activities. For instance, there is increased attendance at school events, monitoring of student attendance/performance, provision of scholastic materials, and acceptance of teaching in the local language. This, however, applies to a small proportion of parents who were reached. 2. Conclusions As a result of SHRP's limited engagement with parents and community members, parental support and participation continues to be a serious challenge inhibiting reading and health outcomes. 3. Recommendations  Per the midterm evaluation, and even more needed now given the deleterious effect of student absenteeism on achievement of reading outcomes, SHRP should quickly develop an effective community mobilization strategy for sensitizing communities on the importance of education in general, as well as early grade reading, and local language learning. SHRP needs to develop and implement a community mobilization strategy that conforms to best practice in this area, and be much more than a recitation of talking points.  SHRP should work with USAID to organize support from the USAID-funded Communication for Healthy Communities whose mandate includes developing community mobilization strategies and supporting communications. We interviewed the Chief of Party who confirmed that this is within the project’s mandate and capabilities. Community mobilization efforts should encompass key influential voices, including the local language board members, foundational bodies, cultural leaders, and so on.  Specifically, SHRP should promote, in a new, highly participatory manner, with the SMCs and PTAs, as well as beyond among parents and the larger community, the value of education and the need for parents to commit to and fully implement their roles and responsibilities in addressing the educational needs of learners, e.g., parents should provide basic food and school supplies to their children to ensure they are ready to learn, support/enforce school attendance, monitor academic progress, etc. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 38  Ramp up reading promotional activities (e.g., reading competitions) and engage parents and community members. Share good practices from districts that have initiated such activities, e.g., the Gomba PTC holds reading competitions and similar activities among district schools. The Gomba PTC principal said, “When they compete against other schools, teachers and students work hard to make sure they are prepared. These activities were initiated by the PTC, but should be recommended for all schools in order to improve literacy.” The principal asked each head teacher to conduct school-based reading promotion activities such as Drop Everything and Read (DEAR) days, morning spelling drills, and debates.  Advocate for facilitation support for local language boards so they can exercise their community engagement functions (see discussion on this in Q4, p. 40). FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 39 EVALUATION QUESTION 4 To what extent are the reading and health activities on track to continue without USAID assistance? 1. Findings Personnel and systems. All district and national stakeholders within the MoES, USAID, SHRP, and SHRP partner organizations were asked, “Does the MOES have the personnel and systems in place to continue the SHRP program without USAID assistance?” Two-thirds (43 people) of stakeholders replied, “yes” and one-third (21 people) responded, “no” for R1 Reading. The answer “yes” included 50% (13) national stakeholders, 61% of district respondents, and 50% (11) of head teachers; generally a group with information on MoES personnel and systems at both the national and district levels. For R2 Health, only 27% of respondents believe SFI will be sustained. According to respondents, there is already evidence of scale-up through replication of SHRP reading activities and a variation on the SFI program under LARA in 25 districts; and of the reading methodology under Global Partnership for Education/Uganda Teacher and School Effectiveness Project. SHRP is coordinating closely with LARA, in fact sharing some functions, and supporting the MOES with expanding the early grade reading intervention to 27 additional districts across 2,644 primary schools through a 3-year Global Partnership for Education grant to the MOES managed by the World Bank7. Since the inception of the project, SHRP has worked with and through all relevant departments of the MOES and, by engaging particular departments, has strengthened the capacity of ministry personnel in materials development (NCDC), teacher training (TIET), inspection (Directorate of Education Standards) and assessment (UNEB). Overall, SHRP has focused more on building capacity by working with and through national systems and structures, and less on systems strengthening per se. As is clear from our descriptions below on the approximate status of capacity building on a department-by￾department basis, this varies considerably by department. UNEB is an example of deliberate capacity building in which initially the systems were defined, followed by training of UNEB personnel, then involving UNEB personnel in EGRA collection, and then putting UNEB in the driver’s seat on EGRA, i.e., responsible for carrying out this function. In the process, UNEB also shaped the process and systems as they deemed appropriate for Uganda. On the other hand, the PTCs have been used as venues for training and have supplied experts for a wide range of SHRP activities, but remain on the cusp of being able to carry out these activities independently for in-service and, in particular, pre-service teachers. SHRP has launched such capability by training a range of PTC personnel in the methodology and working with TIET and Kyambogo University to include EGR into the PTC curriculum. An important overarching vehicle for sustainability of SHRP, LARA, and Global Partnership for Education/Uganda Teacher and School Effectiveness Project is the formation of an EGR and Retention Task Force underway in the MoES, which will be charged with: “…sustaining current investments made by SHRP, Global Partnership for Education, and LARA… house and coordinate EGR information on behalf of the MoES, monitor progress and strive to ensure sustainability of gains in EGR. Deep ownership will be displayed in increasing 7 USAID/Uganda School Health and Reading Program. (2016). 2nd Quarterly Report, January 1-March 31, 2016. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 40 contributions of MoES staff time, finance, materials and other resources to leverage donor￾funded interventions, expansion into the districts not yet covered, and maintaining the momentum and capacity for implementing EGR after these projects are complete especially scaling up to other grades (P5-P7).”8 In-service training. By engaging PTC personnel in its activities and using PTCs as training venues, SHRP has developed the capacity of principals, tutors, and CCTs to deliver EGR and PIASCY teacher training using participatory, learner-centered methodologies, as well as support follow-up Continuous Professional Development and Monitoring and Support Supervision. PTC venues and staff have conducted initial training and refresher workshops for P1 and P2 teachers, and initial trainings for P3 and P4 teachers. Refreshers for P1, P3 and P4 teachers are scheduled for May 2016. To institutionalize the capacity for EGR and PIASCY training into the PTCs, the following actions are needed:  MoES must adopt a policy making the EGR methodology a requirement at the primary school level as part of the thematic curriculum. While this policy is in process, at this point the EGR methodology is an optional support to the thematic curriculum.  SHRP needs to build the capacity of the PTC at the national level (Kyambogo University) in, as one PTC Deputy for Outreach stated it, “how to really carry forward the different elements, with mentoring and coaching on the nuts and bolts of implementation.” Or as a tutor who has been involved in all aspects of SHRP activities described it, “This needs to be a process, like a mother weaning a child.” In other words, besides the policy adopting the EGR methodology, and ideally PIASCY and SFI, there needs to be a plan for how PTCs will carry out the various elements at the national and district level policy and how SHRP will help throughout the process so that PTCs are fully in the driver’s seat well before SHRP’s close in August 2019. This includes most specifically development and implementation of an EGR curriculum at PTCs for pre-service teachers, and assuming responsibility for support to in-service teachers. As noted, this is a necessary next step following SHRP’s efforts with Kyambogo University and TIET to integrate the EGR methodology into the PTC curriculum. As part of the institutionalization process, SHRP and the MoES need to address various key issues affecting quality, specifically:  Ensuring quality trainers for all districts on-boarding the EGR methodology. Many respondents noted this as an issue given the high demand for trainers with SHRP, LARA, and Global Partnership for Education/Uganda Teacher and School Effectiveness Project underway.  Fully articulating and implementing a more effective and school-based approach for Continuous Professional Development, coaching, and mentoring. SHRP recognizes what we heard from many respondents and what we have provided repeatedly as continuous evaluation feedback regarding the congestion and crowding of its training sessions. However, SHRP has found it very difficult to modify and still reach the numbers required. Therefore, CCT follow-up at the school level with teacher Continuous Professional Development is all the more critical for effective assumption of the new teaching methods. However, CCT ability to provide two visits with quality Continuous Professional Development Trainings per term varies considerably, and, for the most part, is not a reality. SHRP has made Continuous Professional Development a part of Monitoring and Support Supervision, but this is very resource intensive and doesn’t obtain the 8 Terms of Reference for MoES EGR and Retention Task Force: FY2015/2016, Draft provided by MoES SHRP focal point. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 41 reach and frequency needed to support teacher mentoring and coaching needs. We identified the need at the mid-term evaluation for a more resource-effective and school-based approach for teaching mentoring and coaching that could approximate the level of support teachers actually need to become proficient in the EGR methodology. LARA is addressing this critical weakness by hiring qualified teachers as FAs who will accompany CCTs on school visits, helping to ensure the quality and frequency of school monitoring and coaching visits needed and at the same time building the support supervision capacity of CCTs. LARA is also placing more emphasis on hands-on demonstration for teachers. SHRP continues to grapple with this and is working toward a more school-based model but it is not yet fully articulated or in place.  Better equipping trained teachers to train and mentor others. Another critical part of SHRP’s plan for capacity building of in-service teachers is to have trained teachers, including head teachers and deputy head teachers, train other teachers – with peer training and mentoring in the case of trained teachers, and support supervision in the case of head teachers and deputy head teachers. This part of the cascade training model is important for training all in-service teachers in the EGR methodology, and helps mitigate the effect of teacher transfers. Again, for a variety of reasons, primarily competence, but also lack of compensation or other rewards for trained teachers to take on this additional task, reality has fallen short of expectations. Trained teachers have not achieved the competencies required to become trainers within their schools, they expect incentives in order to carry out trainings or orientations, and it may also be that the additional time required might compromise their primary responsibilities. Pre-service training. SHRP has trained all principals, deputy principals, language tutors, and Early Childhood Development (ECD) tutors in all 45 public PTCs (core and non-core) in the EGR methodology. Some PTCs use all or parts of the EGR methodology in their pre-service classes. Most PTCs visited believe they have the capacity to fully integrate the EGR methodology into their pre￾service training, but note that institutionalization requires formal adoption of the EGR methodology as part of the thematic curriculum along with a corresponding addendum to the existing teacher-training curriculum. SHRP and TIET are currently working on a concept note to achieve this, which Kyambogo University must approve. This is expected in the next few months. PTCs will then need some assistance in implementing this policy, e.g., possibly additional training to transition from a five-day approach to a course syllabus, support with teaching use of particularly thorny elements of the methodology such as the Continuous Assessment Monitoring and learner checks, and teacher guides and pupil books for their students. Best practices that have already sprung up in some PTCs should be identified and integrated into this transition, e.g., the Kabulasoke PTC in Gomba has successfully integrated Continuous Assessment Monitoring into its practicums. A similar trajectory could also be implemented for PIASCY/SFI, i.e., formal adoption into the co-curriculum, addendum to the existing teacher-training curriculum, assistance in transitioning the short-duration training into a course, and distribution of supporting instructional materials. Materials development. According to NCDC, SHRP has received official approval of 54 of 104 EGR pupil books and teacher guide titles in 12 local languages and English. This is a massive accomplishment given development of these books was an intensive 3-year process that involved developing orthographies, drafting textbooks, printing, pilot-testing and revising. Moreover, SHRP has distributed 2.2 million books since program inception. When asked whether NCDC has the capacity to continue developing, replenishing and distributing materials independent of SHRP, stakeholders were optimistic. Respondents believe that NCDC has the capacity and knowledge to develop materials since they were integrally involved in the materials development process. They have access to the writers, CCTs, teachers, language boards and others involved in the development of the orthographies and materials; FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 42 and therefore can access the talent to draft materials and manage the process, should new orthographies be included or for redrafting of existing materials. NCDC will need support and funding for future printing and distribution of materials. Regarding R2 Health, the learner primers represented an easy opportunity to integrate age-appropriate health and HIV/AIDS concepts. This would reinforce SFI programming and will complement and support the MoES’ effort underway to integrate SRH into the thematic curriculum. The opportunity to integrate health and HIV/AIDSs into the EGR materials was a missed opportunity, according to the USAID HIV/AIDS Prevention Specialist, who said, “If there had been a mindset oriented to integration, it might have worked because SHRP works through teachers and therefore had the foot soldiers needed for integration.” Some health topics were ultimately included in the P1-P3 primers, particularly on accidents (one-third of such entries), and a more dedicated effort was made to do this in the P4 English primer which includes topics on HIV/AIDS, oral hygiene, sanitation, and malaria. As an indication of this as a missed opportunity overall, no primer discusses handwashing, the single most important health behavior (when soap is mentioned, it refers to clothes washing). When materials are updated, for example to correct errors, this missed opportunity can be addressed. NCDC can access the same experts that were used to craft the Enhanced PIACSY materials and the SFI Handbook. When materials are updated, it will also be important to review and ensure alignment with the theme, story, illustration, and vocabulary words. In the haste to create the materials, there are many cases where these elements are not aligned. Community mobilization. As the former SHRP Agreement Officer’s Representative explained to the evaluation team, SHRP originally planned a major community mobilization campaign. However, priorities shifted during implementation, reducing the scope of community mobilization for SHRP and eventually shifting responsibility for a national campaign to LARA. A Social Mobilization Officer only joined the SHRP team two years into the project, and only began community mobilization efforts in earnest in the last year, working closely with the DCOP. During that time, a number of activities have been initiated, including four regional dissemination meetings, three local language board training sessions, event participation (Drop Everything and Read (DEAR) Day, International Mother Tongue meeting,), six reading competitions, and a radio program targeting four districts over four months. Community mobilization around R2 Health issues is intended to be part of the monitoring and support supervision visit with SMCs/PTAs but in reality this has been quite limited. There does not appear to be an overarching strategy guiding the selection of events. The goal, per the Social Mobilization Officer, is to create appreciation for EGR in local language and motivate parents and communities to support school attendance and material needs. In-roads have been made in creating appreciation for local language instruction, especially as parents and the community begin to see P2 learners reading. Many respondents mention with a certain amount of awe the excitement felt at reading events where a P2 learner who has been in a SHRP treatment school can read words that a P7 learner cannot. However, respondents very much agree that more needs to be done, particularly to address high learner absenteeism, through a more deliberate mobilization strategy, that brings in influential community voices other than educators, and helps the community, parents, and children understand the value of education itself. Mobilization also needs to address the issues around school feeding so that parents can understand that government will not fill this gap and also understand the importance of good nutrition on learning. This will impact R1 Reading outcomes positively, and themes from R2 Health can easily be woven in. Although LARA has responsibility for strengthening communications and community mobilization at a national level and in its districts, SHRP needs to provide the communications and mobilization needed in FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 43 its districts. We met with the Communication for Healthy Communities (CHC) COP, who said CHC can provide the technical assistance needed to develop a community mobilization strategy for SHRP, including audiences, messages, and tools. This would include advocacy with foundational bodies, cultural leaders, local politicians, and successful local individuals and celebrities that would inspire youth and parents alike to see new possibilities for the future. It should also include local language board members who are highly respected yet insufficiently tapped as advocates and resource on orthography questions, particularly newer orthographies. SHRP has been challenged in mobilizing local language board members because, although willing and enthusiastic to be EGR and local language advocates, the law governing local language boards does not attach them to a specific part of government for facilitation support. This needs to be addressed. NCDC is the most logical home, given the commonality of their functions. SHRP points out that the language boards have been trained in mobilizing resources to support their work but we did not see evidence of this working. Support supervision: The majority of stakeholders reported that Monitoring and Support Supervision would be sustained because district teams have been oriented on the process and tools. CCTs have been equipped with fuel allowances to ensure visits at least twice per term to SHRP schools. However, due to a shortage of CCTs and district inspectors and their extensive coverage areas, along with other factors, they have failed to visit all schools. Additionally, while some districts (e.g., Moroto,) have integrated EGR elements into their district inspection monitoring tools, this has not been done on a national scale. All central and district inspectors have also not benefitted from SHRP’s Monitoring and Support Supervision training. For instance, district inspectors in Moroto have not been trained due to schedule conflicts, and those responsible for monitoring control schools were excluded. Thus, additional capacity building and integration of EGR elements into Directorate of Education Standards monitoring tools are needed to sustain this component. The Directorate of Education Standards has developed a robust monitoring and training system for secondary education that could be replicated for monitoring EGR standards in primary schools (see text box). For R2 Health, the support supervision tool modified by the District Inspector of Schools in Lira to include PIASCY, SFI, and referrals can be a model. Stakeholders strongly recommended increased coordination among PTCs, District officials and SHRP on support supervision planning, tools, and implementation. This will help ensure the level of visits needed, possibly identify ways to divide up and use limited resources more efficiently, and help prioritize school most in need (e.g., remote schools). There are currently many different tools being used; while each organization may need tools to respond to their specific reporting requirements, this should be examined as a resource and sustainability issued. Key for sustainability is simply having the number of CCTs necessary for the minimum two visits per term and with the supporting facilitation budget. Monitoring and evaluation. SHRP has built the capacity of the UNEB to conduct EGRA, involving UNEB staff in the process from developing and pilot-testing instruments through training assessors and conducting quality assurance of fieldwork. UNEB is now responsible for leading the EGRA under Global Partnership for Education with technical assistance from SHRP. This is a clear example of the Ministry’s capacity being built up over a period of time to take ownership of a SHRP-initiated activity. SHRP has also tried to build the capacity of supervisors to conduct learner checks (individual reading assessments) during Monitoring and Support Supervision visits, and of teachers to utilize Continuous Assessment Monitoring on a daily basis to inform instruction. These efforts are less formalized and are far from being institutionalized. The Continuous Assessment Monitoring tool is not user-friendly; and assessment in general has not been emphasized during trainings, including end-of-term assessment. Given UNEB’s expertise, they could assist districts in developing learner assessments, including end-of-term quantitative assessments, and training teachers in assessment skills, even though this is not their central mandate. The District Inspector of Schools and CCTs could also assist with monitoring the use of FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 44 Continuous Assessment Monitoring and increasing the perceived value and usefulness to improve instruction. HIV/AIDS and health. R2 Health’s counterpart is the Guidance and Counseling Department in the MoES, but it has not been very engaged, as noted in the mid-term evaluation and confirmed again during this evaluation. The HIV/AIDS Unit, which would appear the obvious counterpart, is very donor￾dependent, not staffed by civil servants but rather contract employees, and not regarded as an intrinsic part of the Ministry. It is the secretariat to the HIV/AIDS Working Group, which therefore meets infrequently. Its last meeting was November 2015. HIV/AIDS in the MoES draws on a very small budget, Ush 30M (less than $10,000), which covers three other cross-cutting areas. SHRP’s increased school￾based R2 Health targets, lack of engagement by the Guidance and Counseling Department, and the weak HIV/AIDS Unit, shifted SHRP’s focus away from MoES support and capacity-building. This was noted in the mid-term evaluation, which included a strong recommendation to recommit SHRP to integrating HIV/AIDS indicator data into the Ministry’s Education Management Information System. While this has not yet been achieved, and is particularly challenging due to proprietary EMIS software, it appears to be on track, and many respondents noted the importance of achieving this for accountability and sustainability purposes – “what gets measured, gets done.” Senior MoES officials said that health as a topic needs a more permanent home in the Ministry, in a department staffed by a senior civil servant; and that this can be tackled as part of the EGR and Retention Task Force. This will be an important adjunct to making PIASCY/SFI a compulsory element of the thematic curriculum. Funding commitments. While there is evidence of increased capacity and systems within the MOES, all stakeholders emphasized the sustainability of SHRP activities largely depends on GOU funding commitments. There is currently no evidence of GOU budget commitments to SHRP program activities. Nearly 90% of stakeholders (39 people) stated that GOU budget commitments either did not exist or they were not aware of any budget allocations. Education, overall, is largely donor dependent. According to MoES officials, for the GoU to qualify for the next round of Global Partnership for Education funding in 2019, the government must dedicate 15-20% of public expenditures to education. Unfortunately, the opposite seems to be occurring. According to the World Bank, public expenditure on education as a share of total public expenditure decreased from 13.9% in 2009 to 9.4% in 2010, before leveling off at 11.8% in 2013; as share of GDP, public expenditure has decreased from 3.3% in 2009 to 2.2% in 2013, the latest year for which data is available. (Source: World Development Indicators, 2016). Many respondents noted the need for concerted donor advocacy at the highest levels to counter this trend, at the Ambassadorial level by individual donors, and collectively by donors via the Local Development Partners Group attended by Ambassadors and Chiefs of Missions. Given the timing, there is urgency to act before the next round of government budget decisions in November 2016 for the 2017/2018 budget. One idea that gained some traction among respondents is for SHRP to quickly provide consultant assistance in developing a brief policy support via an expert who can create a convincing argument based on loss of GDP of an illiterate workforce and return on investment/GDP of a literate workforce, along with projections of actual budget requirements on a line item and per capita basis. LARA is providing support along these lines but on a much longer timeframe based on a more comprehensive organizational development process; SHRP assistance could be swift and timely in terms of the budget process, help create the longer-term receptivity of use to LARA’s efforts, and form part of a sustainability and exit strategy. [SHRP notes that GPE funding depends more on effective use of funds and results than the amount of funds per se. This needs to be sorted out.] FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 45 2. Conclusions  Exit strategy for sustainability. SHRP has worked with and through all relevant departments of the MoES, and as a result has strengthened the capacity of ministry personnel in a number of departments, e.g., NCDC, TIET, and DES. In some cases, SHRP has strengthened the internal systems and procedures and trained personnel in the systems and procedures in a learning-by￾doing approach that allowed departments to take on specific functions over time in a phased and orderly manner, e.g., UNEB with EGRA. With the extra time available to SHRP through the extension, it has the opportunity to develop and implement a comprehensive exit strategy aimed at systematically building the systems, procedures, and capability across different departments in the Ministry, so that by project end the Ministry is able to fully carry the EGR program forward. The EGR and Retention Task Force in the MoES would be a natural home for this effort.  Community mobilization. Positive experiences with early grade reading and, to a lesser extent, community outreach have helped create support for local language reading in primary schools. Student absenteeism however and overall support to schools requires community mobilization within the context of a strategic communication effort. SHRP needs a community mobilization strategy for its own districts as soon as possible to galvanize influential voices that can effectively convince parents and communities on the value and potential of education. USAID’s CHC project has the capacity and mandate to help SHRP develop a community mobilization strategy, including audiences, messages, and tools.  Monitoring and Support Supervision. Increased coordination among PTCs, District officials and SHRP on support supervision planning, tools, and implementation is needed to help ensure the number of visits needed for each school, use limited resources most efficiently, and best prioritize among schools. While recognizing organizational needs behind the different tools, ensuring complementarity and achievement of common EGR goals among tools is an important resource and sustainability issue.  Monitoring and evaluation. SHRP has institutionalized EGRA in UNEB and this effort represents a good example to guide SHRP’s overall exit strategy and sustainability planning. Less progress has been made in related areas, in particular monitoring and assessment at the school level by teachers and as a part of MSS, in some cases due to the tool being difficult to use (Continuous Assessment Monitoring), in other cases because the tool is relatively new (learner check), and/or less emphasis to date (end-of-term assessment). In the case of end-of-term assessment, there has been some reticence given the test culture that prevails in Uganda. However, the result is that teachers, children, and parents do not have a realistic understanding of reading progress, based on the impact evaluation results.  HIV/AIDS and health. R2 Health does not have a sufficient counterpart in the MoES due to various issues with both the Guidance and Counseling Department and the HIV/AIDS Unit. Given this reality, and with PEPFAR’s focus on school-based targets, SHRP’s MoES efforts concentrate on integrating HIV/AIDS indicator data into the Ministry’s Education Management Information System, which is very important for sustainability of HIV/AIDS activities at the school level, but quite challenging given the proprietary nature of the software on which the EMIS is based. To reap the benefits SFI has on school environment and learning outcomes, and make it a compulsory element of the co-curriculum, health and related topics need to be lodged in a section of the Ministry staffed by a senior civil servant. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 46 Implications for USAID and MoES USAID  Require an explicit, consensus-based sustainability strategy that cuts across EGR activities  Conduct high-level advocacy on a multi-donor basis for increased GoU education funding  Leverage other options for school feeding MoES  Jointly develop and implement an EGR sustainability strategy with SHRP anchored in a high-level, cross-cutting, and empowered task force  Adopt SFI as an official part of the co-curriculum to sustain its positive impact on school environment and learning, with appropriate leadership  Support funding advocacy including appropriate way to quantify school reading progress  GoU education funding. Sustainability of SHRP activities largely depends on GOU funding commitments, which has fallen to less than 12% of public expenditures. To qualify for the next round of Global Partnership for Education funding in 2019, the government must dedicate 15- 20% of public expenditures to education. This could be a basis for rallying coordinated donor advocacy at the highest levels. A convincing argument is needed to buttress such efforts focusing on returns on an investment in education and specifically literacy, and summarizing budget requirements on a line item and per capita basis. This needs to be done swiftly to impact the next round of GoU budget decisions. At the same time, there is a need for focus on efficiency and efficacy of education funding. 3. Recommendations  Develop a comprehensive exit strategy and sustainability plan aimed at systematically building the systems, procedures, and capability across different departments in the Ministry, so that by project end the Ministry is as fully able as possible to carry the early grade reading program forward. o Use the Early Grade Reading and Retention Task Force as the MoES counterpart for this effort. o Provide specific assistance to PTCs for both in-service and pre-service teacher training following the adoption of EGR into the PTC curriculum by Kyambogo University. To address elements of the SHRP model that have thwarted achievement of reading outcomes, the sustainable strategy needs to incorporate viable responses to the need for 1) quality trainers; 2) more effective approaches to teacher continuous professional development, coaching, and mentoring that is close enough to schools to happen at the frequency required; and 3) equipping trained teachers to train and mentor others. New approaches being piloted by LARA and Global Partnership for Education should be factored into this. o Ensure NCDC has the systems and procedures in place to support future materials development, printing, and distribution, including connecting NCDC to the network of experts who created the PIASCY and SFI materials so that when materials are updated, they can integrate health topics. o Examine needs and incorporate TIET, UNEB, Directorate of Education Standards, and other relevant MoES departments into SHRP’s exit strategy. Work with Directorate of Education Standards to develop an EGR monitoring tool and assessment-training program to ensure that all District Inspectors of Schools are fully trained and equipped to carry out monitoring activities. o Develop a structural solution within the MoES for health and HIV/AIDS that supports SFI as a compulsory part of the co-curriculum and provides the expertise and support needed to foster PIASCY/SFI programming within all other MoES departments, e.g., PTCs for in-service and pre-service training, NCDC for materials development, Directorate of Education Standards and other monitoring and support supervision systems, etc.) FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 47 o Ensure HIV/AIDS indicators are integrated into the Education Management Information System of the MoES. o Help resolve local language board access to facilitation support by connecting them to a specific department in the Ministry, recognizing the synergies with NCDC and their potential to support schools and community advocacy.  Mobilize influential voices to galvanize parents around the value of education and get their kids to schools. Develop an overarching strategy for community mobilization that brings influential voices to bear and inspires parents to promote schools’ attendance and communities to support school needs. Achieve this via technical assistance from the USAID-funded CHC project. Donor advocacy to increase GoU education funding for sustainability. USAID and other donors should advocate for GoU funding for education in the range of 15-20% which is required to qualify for the next round of Global Partnership for Education (GPE) funding. The mid-term evaluation of GPE planned for September 2016 will clarify the overall situation with the program. USAID and SHRP should closely monitor GPE developments for near-term advocacy opportunities. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 48 “A project like this cannot be static but must be a learning organization. We have changed many things based on the mid-term evaluation and we like the CE feedback. Examples of change include: Result 1 and 2 work more closely together, sharing resources; teacher training changes; and integrating health into the materials. Collaborating, Learning, and Adapting needs to be a major component, especially as we move into the extension years.” --Senior SHRP Staffer EVALUATION QUESTION 5 To what extent did SHRP utilize the continuous evaluation (CE) data for adaptive management? 1. Findings Respondent Data. Although this evaluation question primarily relies on the continuous evaluation data collected throughout SHRP implementation and consolidated into monthly performance feedback to SHRP, we did include in our key informant interviews two related questions: 1. In your opinion, to what extent does SHRP integrate feedback from its stakeholders (into planning and implementing project activities)? 2. In your opinion, to what extent did SHRP disseminate information, lessons learned, and best practices to its stakeholders? Of the 43 stakeholders that responded to the first question, 88% (38) said that SHRP is responsive in integrating feedback from stakeholders into project planning and implementation. In response to the second question, respondents mentioned the following key ways used by SHRP for dissemination:  Stakeholder meetings and workshops (national and regional)  Brochures, briefs, and newsletters  Debriefings during MSS visits  Progress reports  Work planning and coordination meetings Continuous Evaluation Data. From May 2013 – Jan. 2016, P&IE has provided SHRP with:  254 appreciative comments, defined as actions we have observed that were done well  211 constructive comments, defined as actions we have observed that could be improved From the 211 constructive comments, SHRP identified 166 follow up actions, i.e., provided comments on feedback memos on 166 actions they could undertake to address the comments, representing 79% follow-up intentions, and, per Table 11, implemented 38% of these. Below we provide summary samples of continuous evaluation feedback data and responses and/or actions taken by SHRP, selected for their relevance to the evaluation questions, namely surveys (EGRA and KAP), capacity building, community mobilization, and sustainability. These examples illustrate that SHRP has made significant use of the continuous feedback data, deepening the results achieved. Table 12. Feedback Memo Analysis (May 2013 – January 2016) Feedback Total Number Average per month Percentage of follow-up by SHRP Appreciative Comments from P&IE to SHRP 254 9.1 NA Constructive Comments from P&IE to SHRP 211 7.5 NA Action Points Suggested by SHRP 166 5.9 79% Action Follow-up by SHRP 63 2.3 38% FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 49 a. Surveys: EGRA and KAP EGRA. P&IE observed and provided feedback on training for three EGRA rounds and on five rounds of EGRA data collection. Approximately two-thirds (30) of the 45 recommendations were implemented, a high percentage, and one-third (15) were not yet implemented. Of those not yet implemented, SHRP stated that it planned to implement 60% (9) of the recommendations in the next round of training or assessments. SHRP did not feel change was warranted or possible for the remaining 6 unimplemented recommendations. In short, the IP incorporated more than 85% of the EGRA-related recommendations.  EGRA preparation and training. We provided comments on class size, methods, materials, and field practice. SHRP considered the feedback on over-large class sizes, but chose to keep classes large so that trainees for each language received consistent training. P&IE identified problems in ensuring inter-rater reliability (IRR), and encouraged use of videotaped (versus staged) demonstrations, seating arrangements to maintain rater independence, and shorter lag between rating and discussion of inconsistencies. As a result, SHRP introduced video demonstrations in its 2015 EGRA assessor trainings. It did not change discussion timeframes. SHRP incorporated or committed to incorporating suggestions for improved adult learning methods in its training sessions. We observed a number of problems with translation of the instrument into languages with newer orthographies. The majority of these were addressed. We noted the absence in any training materials of detailed instructions for administering each question (a QXQ). SHRP responded that it relies on general instructions in the translated instruments.  EGRA field data collection. Our comments related to inconsistent assessor performance of some sub-tasks, the choice of assessment venue, team sizes, and instrument issues. For a number of observations regarding assessor performance on sub-tasks, SHRP noted “we have not seen this before” and modified the training materials accordingly. SHRP attempted to alter the team size to better accommodate data collection needs. Over time, SHRP has worked to resolve our observations regarding inadequate local language translations, issues with response options, and missing instructions with the instrument in the program used for data collection on tablets. KAP. P&IE provided feedback on three KAP trainings and pretests and two surveys. P&IE suggested ten ways to improve training, fieldwork, and the KAP questionnaire. SHRP acted on 70% of these, a high percentage.  KAP preparation and training. P&IE observers identified errors in the questionnaire and response options; these were addressed in the endline KAP survey. We recommended additional supervisory training but this was not provided. We recommended that KAP training materials include a detailed question-by-question section; instead, SHRP relies on a general paragraph explaining requirements in the questionnaire. The midterm KAP survey could not be used due to anomalies in following Institutional Review Board requirements. SHRP acted on our recommendation that survey leadership receive ethics training, and as a result the endline KAP survey avoided the problems encountered with the midterm KAP.  KAP data collection. Our team observed some failure to follow good interviewing practices (e.g., making the respondent comfortable) or adhere to the instructions (e.g., allowing learners time to think through a response). SHRP commented that it would address these issues in future trainings. We identified a number of logistical issues that impeded smooth conduct of the first survey that had been addressed by the endline. We noted parental misconceptions during the FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 50 baseline survey that reduced participation. SHRP did not follow our recommendation to use SMCs and PTAs to provide correct information to parents; rather head teachers directly informed parents and used radio. Our recommendation to formalize the role of interns, who helped mobilize schools, was implemented via the FA position. b. Capacity Building P&IE provided numerous appreciative and constructive comments to SHRP on issues related to capacity building, specifically on training and monitoring and support supervision. R1 Reading Training. Of the comments, 29 specifically provided recommendations for SHRP action to improve EGR training. SHRP acted on 63% of these recommendations, a high percentage. Our recommendations targeted improvements in: training approach (12), training materials (5), trainer preparation (4), time management (4), registration and payment (2), uniformity in training delivery (1), and adequate number of trainers (1). Examples of specific constructive feedback and actions taken by SHRP follow. Training approach  Observation: During the initial EGR training, some trainers did not clarify the link between Literacy 1 and Literacy 2 that make up the literacy hour. Action: In the refresher EGR training, SHRP specifically discussed literacy hour teaching procedures showing the linkages between Literacy 1 and 2.  Observation: Trainers read content directly from training materials, and did not explain, elaborate, or substantiate concepts. SHRP action: In subsequent sessions, trainers were discouraged from reading directly from the materials.  Observation: Training sessions were over-crowded with 100 participants per room, and small group work of 15-18 trainees. Action: SHRP recruited more trainers to create smaller classes. However, classes remained too large with approximately 70 trainees. SHRP reported difficulty recruiting trainers with local language capability, a continuing problem.  Observation: Training sessions did not provide enough preliminary material and trainees could not properly prepare for the sessions. Action: SHRP encouraged trainers to focus on core content.  Observation: Training incorporated audio-visual materials with very valuable demonstrations of the methodology, but did not ensure the time needed to explain or discuss them. Action: SHRP “right-sized” the training to fit the time available and include time to reflect on the videos. Training materials  Observation: Audio-visual materials recorded in one local language were used in different local language region training, which trainees might not know and/or model. SHRP action: SHRP responded that it was not feasible to recorded materials in all 12 local languages. Trainer preparation  Observation: Trainers were not given sufficient time to prepare lessons between trainings of trainers and teachers training sessions. This forced trainers to prepare lessons when they were supposed to be team training. Action: While SHRP did not indicate an action per se to this feedback, SHRP noted that it reminds trainers not to prepare future sessions during trainings. SHRP also notes that program realities dictate a one-day refresher but it is now streamlined with enough time for trainers to plan and travel. From our observations, this is not yet fully resolved. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 51 Example of SHRP Response to Feedback “When SHRP was told that Ministry staff is left out of R2 programs, e.g., MSS and training, which will negatively affect taking the program over later, SHRP started to include Ministry staff in MSS trips and PIASCY training. The Lira district officials were so enthusiastic after the training that they changed their own monitoring and supervision form to include checking on the SFI program.” -- MoES Official  Observation: Trainers were not confident in delivering training sessions on English. Action: SHRP brought in mentors to help trainers. This noticeably improved trainer confidence and capabilities. Time management  Observation: Trainers did not follow the training schedule rigorously. This led to cascading delays, running past the end time, and short-changing some critical material. Action: SHRP reviewed the training guides and included clear guidance on time allocations.  Observation: Late meals caused delays. Action: SHRP asked PTCs to serve meals on time and trainers were asked to continue teaching until meals were ready to be served. Registration and payment  Observation: Registration and payment caused serious delays and disruptions in training. Action: SHRP created participant lists that only require a signature, and switched to mobile payments. R2 Health Training. We provided 12 constructive feedback points and SHRP responded to nine (75%) of them, a high percentage  Observation: PIASCY training of trainers had only one day to allocate tasks, prepare lessons, and travel to training venues. Action: The problem was included on a list of issues to be addressed before the next activity, but not yet addressed.  Observation: PIASCY teacher trainings used the same two trainers for each class and trainees disengaged. Action: SHRP included more experts in subsequent training of trainers to create more trainers for teacher trainings, and used rotating trainers at PIASCY trainings to create more engaging sessions.  Observation: Trainers were unprepared on delivery methods, particularly participatory approaches, due to a lack of time. Action: SHRP used participatory methodologies in its 2016 PIASCY training so teacher would experience as well as receive instruction in them.  Observation: PIASCY training used too many different trainers for each subtopic, which led to disjointed content delivery and scheduling delays. Action: In subsequent PIASCY trainings, there were fewer trainers who covered a large scope of the training. Monitoring and Support Supervision (MSS): R1 Reading P&IE provided 12 constructive comments related to MSS and SHRP acted on all but one, a very high percentage.  Observation: Supervisors provided uncoordinated and disjointed feedback on classroom observations to teachers. Action: SHRP created a reporting template for supervisors to use in organizing their feedback and then follow when sharing feedback with teachers.  Observation: Supervisors had difficulty determining which documents to use for various parts of the process. Action: SHRP reviewed the materials and created one user-friendly supervision document.  Observation: Supervisors weren’t able to carry out the process properly at two schools per day. Action: MSS is now limited to one school per day. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 52  Observation: Supervisors were not using SHRP instructional materials as a reference during lesson observations. Action: SHRP clarified the importance of these materials for lesson observation.  Observation: After-action field debriefs lacked structure and did not result in conclusions and action points. Action: SHRP noted that “MSS activities have been streamlined but it is a long week…Debriefs take place on Thursday afternoon following school visits and lunch. We are always learning – for example, before MSS starts up this next term (June, 2016) all staff who support this will attend a half day re-training to ensure best practices are followed.”  Observation: School administrators were not informed on lesson observation cycle procedures and therefore could not support monitoring and support supervision adequately. Action: head teachers were provided a copy of the SHRP lesson observation tool. However, training is required for head teachers to use the tool. This is a real oversight given the importance of the head teacher in daily teacher support.  Observation: monitoring and support supervision sometimes included personnel without the relevant local language capability whose utility was compromised. Action: SHRP involves officials with local language capability. Monitoring and Support Supervision: R2 Health P&IE provided 12 constructive comments related to monitoring and support supervision and SHRP responded to nine (75%) of them, again, a very high percentage  Observation: Schools were not following record-keeping requirements for R2 Health. Action: A SHRP reinforced the importance of record-keeping in SFI and Guidance and Counseling registers which has improved somewhat. SHRP suggested providing teachers a journal for record-keeping but this has not happened.  Observation: monitoring and support supervision teams for R2 Health varied. Those without MoES or SHRP Kampala representation could not always respond to questions on the program. Action: SHRP said this happened when R2 Health began using a monitoring and support supervision checklist that feeds into the SHRP M&E system and required program knowledge. SHRP is taking steps to ensure teams able to answer teacher questions.  Observation: Monitoring and support supervision follow up visits did not bring knowledge of gaps already identified which created redundancy and time inefficiencies for teachers. SHRP needs to have existing information available per school for subsequent visits. Action: SHRP notes that MSS is typically most intensive at the beginning of the school year which is also when teacher transfers are discovered, which negatively impacts the availability of prior MSS data. However, this does not address the lack of MSS data in general within MSS visits.  Observation: The refresher training of trainers on Enhanced PIASCY Instructional Materials was very detailed, as if for an initial training. Action: SHRP said this was necessary due to the length of time between initial and refresher trainings; and that almost two-thirds of the teachers were new to the training. c. Community Mobilization Examples of specific constructive feedback and actions taken by SHRP follow.  Observation: The meeting to prepare for the radio talk show (to sensitize the public on early grade reading in local languages) did not include a local language board representative; P&IE recommended local language boards be a part of all social mobilization activities. Action: SHRP better engaged local language boards in subsequent mobilization activities. FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 53  Observation: SHRP did not have notes from a prior local language board meeting on Luganda orthography available for local language board members at the subsequent meeting, which reduced productivity. We recommended notes of the prior meeting be made available for reference. Action: SHRP said it is the local language board’s responsibility to document meetings and make them available. [Given local language boards have no funding, this seem impractical.]  Observation: At the same meeting of the Luganda local language board, there was low participation due to dissatisfaction with the terms of service. local language boards wanted appointment letters clarifying their roles. Action: SHRP consulted with different authorities and clarified that local language board fall under the jurisdiction of the local government. However, local language boards continue to have concerns on these issues.  Observation: At the orientation meeting for the Runyoro-Rutooro LLB there were significant communications barriers because local language board members were comfortable communicating in the local language but SHRP facilitators were insufficiently fluent to do so. We advised to provide facilitators with local language fluency. Action: SHRP said it could not hire facilitators with local language capability for all 12 program languages, and ask local language board chairs to serve as translators when necessary.  Observation: The orientation meeting with the Runyoro-Rutooro local language board on the local language board constitution was a very dry review/reading of the clauses and participants became visibly disengaged (sleeping). Action: SHRP increased the time allocated for these workshops to allow time for a more interactive agenda.  Observation: local language board orientations did not include an explanation of follow-on assignments. We recommended written guidance be provided on localizing the local language board Constitution and Work Plan, along with verbal guidance at the orientation. Action: SHRP designed and distributed a template with explanatory details, which local language boards now use to prepare work plans to share with SHRP.  Observation: Community mobilization activities conducted as part of monitoring and support supervision have low participation, exacerbated by late/poor communication from SHRP regarding support supervision visits. Action: FAs now facilitate such communications. These comments corroborate findings presented under evaluation question #3 above. Community meetings held as part of the monitoring and support supervision visit, which target the SMC and PTA, follow the same methodology as described above in the original local language board meetings, i.e., talking points are read out by the facilitators, usually educators, and then the meetings are closed. There is not an interactive element to really engage parents and the community and help them own the concepts. While SHRP has generally been responsive to the specific requests such as those listed above, it has not been responsive to the overall recommendation to embrace a more effective community mobilization approach, which was a major recommendation of the mid-term evaluation. As discussed above, this has negatively affected achievement of results. d. Sustainability There is little data related to sustainability in the continuous evaluation data. While SHRP works through existing institutions, achieving a high degree of country ownership and capacity, sustainability per se is not a stated objective. One entry relates to maintaining traction on integration of HIV/AIDS indicators into the Education Management Information System at the MoES. This echoes a recommendation from the mid-term FINAL PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 54 evaluation to re-commit to this activity,9 and is discussed above in evaluation question #4. Based on this recommendation, SHRP did conclude that the effort to integrate HIV/AIDS indicators into the Education Management Information System should be included again in its work plan. While there have been considerable obstacles in achieving this outcome, SHRP has continued to pursue it, and anticipates achieving it in the near future, before R2 Health activities conclude, notwithstanding the software challenges. The second entry recommends including the MoES Planning Unit and Statistics Department into joint planning meetings. SHRP responded that it engages with the Planning Unit via the MoES’ M&E Work Group and finds that sufficient; and that R2 Health is coordinating with the Statistic Department on integration of the HIV/AIDS indicators into the Education Management Information System. 2. Conclusions  Outside observation and feedback identifies and focuses attention on some important issues that otherwise an IP, operating under time constraints, would not necessarily notice or, even if noticed, get the attention needed to articulate the problem and identify/implement a solution.  The back and forth between the third party observer and the IP creates additional accountability, with the IP considering and responding systematically to observations and recommendation and explaining its view and possible response. This does not supplant internal debriefs and/or after-action reviews but rather adds a third party/external evaluator lens as well as the need to respond to an external entity.  SHRP has followed up on 38% of the constructive feedback provided, implementing a variety of actions, some quite significant, and has spurred a process of continuous improvement, complementing and feeding into other internal systems for performance improvement.  Skype calls following the feedback memos impact both relevance and understanding of each other’s comments. 3. Recommendations  We should prioritize recommendations so that issues more critical to success get more attention and thereby have a higher likelihood of resolution.  The uniqueness of this continuous evaluation process merits reflection by all parties to articulate its value as a model and ways to improve it as a pilot effort.  Monthly Skype or in-person meetings following the feedback memo should be held to help ensure understanding and relevance of each other’s comments. This summary did not comment on the 254 appreciative comments provided to SHRP. While appreciative comments would not generate changes, we note that projects, like people, appreciate positive feedback and such feedback reinforces and helps sustain good performance. 9 SHRP notes that they did recommit to this activity but discovered that the implementing partner supporting EMIS development did not leave the source code, making integration too expensive. The alternative is a stand-alone database. SHRP developed and tested data collection tools in 3 districts with the plan that MoES would integrate the 8 HIV indicators into the annual school census. This wasn’t possible due to resources. Instead, there will be a separate survey whose results can inform the planning process for the new Education sector HIV prevention plans due next year.