1 Revised, May12, 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Panagora Group with NORC at the University of Chicago. Mid-Term Performance Evaluation: School Health and Reading Program MID-TERM PERFORMANCE EVALUATION OF THE USAID/UGANDA SCHOOL HEALTH AND READING PROGRAM CARRIED OUT UNDER THE PERFORMANCE AND IMPACT EVALUATION CONTRACT Revised 12 May 2015 PN 7384; USAID Contract N0: AID-617-C-12-00006 PRESENTED TO: USAID/Uganda Joseph Mwangi, Contracting Officer Representative PRESENTED BY: NORC at the University of Chicago 4350 East-West Highway, 8th Floor Bethesda, MD 20814 Telephone: (301) 634-9413 Fax: (301) 634-9301 Cover photo: P1 students, Nakiwogo Primary School, Waskiso District, Uganda Photo credits: Betsy Bassan DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms i Executive Summary 1 Section I: Evaluation Purpose and Evaluation Questions 3 Section II: Program Background 5 Section III: Evaluation Design, Methods, and Limitations 7 Section IV: Findings, Conclusions, and Recommendations 13 Evaluation Q1: Design 13 A. Findings 13 B. Analysis and Conclusions 14 C. Recommendations 15 Evaluation Q2: Implementation 16 A. Respondent Comments 16 B. R1 Reading 16 B1. Findings 16 B2. Analysis and Conclusions 21 B3. Recommendations 22 C. R2 Health 22 C1. Findings 22 C2. Analysis and Conclusions 29 C3. Recommendations 30 D. Status of Five Year Program Results 30 Evaluation Q3: Results 32 A. R1 Reading 32 A1. Findings 32 A2. Analysis and Conclusions 35 A3. Recommendations 36 B. R2 Health 36 B1. Findings 36 B2. Analysis and Conclusions 38 B3. Recommendations 38 Evaluation Q4: Sustainability 39 A. Findings 39 B. Analysis and Conclusions 40 C. Recommendations 40 Evaluation Q5: Effective Use of Funds 41 A. Findings 41 B. Analysis and Conclusions 43 C. Recommendations 44 Evaluation Q6: Management/Coordination/Lessons Learned 45 A. Findings 45 B. Analysis and Conclusions 45 C. Recommendations 46 Annexes Annex A: Evaluation Statement of Work Annex B: Evaluation Timeline Annex C: Data Collection Instruments Annex D: Program Description Annex E: Matrix Summary of Monthly Performance Feedback Memos Annex F: Sources of Information o National and District Interview Contact List o Bibliography of Documents Reviewed Annex G: Summary of SHRP Budget Projections and Actual Expenditures, May 2012 - March 2014 Annex H: Disclosure of Any Conflicts of Interest MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM i ACRONYMS AIDS Acquired Immunodeficiency Syndrome CA Cooperative Agreement CAM Continuous Assessment Monitoring CCT Coordinating Center Tutor CLA Collaborating, Learning, and Adapting DEO District Education Officer DES Directorate of Education Standards DIS District Inspector of Schools EGRA Early Grade Reading Assessment EMIS Education Management Information System ESS Education and Sports Sector FA Field Assistant FGD Focus Group Discussion G&C Guidance and Counseling GoU Government of Uganda HIV Human Immunodeficiency Virus IR Intermediate Result IRB Institutional Review Board KAP Knowledge, Attitude, and Practice survey KII Key Informant Interview LLB Local Language Board M&E Monitoring and Evaluation MEEPP Monitoring and Evaluation of Emergency Plan Progress MEO Municipal Education Officer MoES Ministry of Education and Sports MIS Municipal Inspector of Schools NAPE National Assessment of Progress in Education NCDC National Curriculum Development Centre NORC National Opinion Research Center PEPFAR President’s Emergency Plan for AIDS Relief PIASCY Presidential Initiative on AIDS Strategy for Communication to Youth, a predecessor program to SHRP P&IE Performance and Impact Evaluation PMP Performance Management Plan PTA Parent Teacher Association PTC Primary Teacher College RTI Research Triangle Institute SFI School Family Initiative SHRP School Health and Reading Program SMC School Management Committee SNE Special Needs Education ToT Training of Trainers TWG Technical Working Group UNITY Ugandan Initiative for Teacher Development and Management System, a predecessor program to SHRP USAID United States Agency for International Development WEI World Education Incorporated MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 1 EXECUTIVE SUMMARY Introduction. The Mid-Term Performance Evaluation of USAID/Uganda’s School Health and Reading Program (SHRP) assesses the effectiveness of activities to date (May 2012 – May 2014, two years into implementation), identifies progress in achieving planned five-year results, and provides recommendations to maintain or improve progress in achieving results. Given that this evaluation was carried out as part of the Performance & Impact Evaluation Contract (P&IE), it also sheds light on the findings of the first annual SHRP impact evaluation, and provides context for P&IE’s collaboration, learning, and adapting (CLA) activities. In the spirit of CLA, we provided and discussed a summary of mid-term evaluation findings and recommendations with SHRP in July 2014. As a result, RTI was able to take the evaluation recommendations into account when preparing the SHRP’s Year 3 work plan. Key evaluation questions. The SHRP Mid-Term Evaluation included six evaluation questions covering design, implementation, results, sustainability, effective use of funds, and management and learning. The evaluation report is organized accordingly. Global significance. SHRP is a program of considerable interest to USAID worldwide, in three key ways: • It represents a flagship effort in promoting literacy through use of mother tongue and therefore its experience will inform efforts elsewhere • Given increasing Agency interest in achieving greater impact through multi-sectoral programming, SHRP sheds light on both the challenges and opportunities in the day-to-day of programming in more than one sector, in this case the health and education sectors • The design includes an innovative application of USAID’s Evaluation Policy, namely a parallel contract, P&IE (providing continuous third party monitoring and monthly feedback, annual impact evaluations, and two performance evaluations, mid-term and final), which creates learning opportunities on the value and structuring of this aspect of the Evaluation Policy Key Evaluation Takeaways Overall. SHRP is viewed positively, with respect and appreciation for weathering a challenging starting relationship with government, and ultimately developing a collaborative and supportive relationship. Working closely with government and within governments structures and systems, SHRP has succeeded in carrying out a very complex implementation and rollout exercise of a mother tongue reading methodology, and achieving what is widely perceived as real progress, with government committing to a national scale-up of the literacy program. SHRP’s health and HIV/AIDS activities are widely viewed as very important but are much less well known than its literacy activities and actual progress against targets is mixed. Reading. SHRP in-service teacher training is also generally well perceived as building key skills and motivation among teachers to integrate the methodology in their schools. The level of follow-on support teachers need for school-based mentoring and coaching to successfully implement the new methodology remains a challenge, with the Ministry of Education and Sports (MoES) structures and personnel unable to provide sufficient on-site monitoring and supervision visits and head teachers at the same level or less as classroom teachers. Leadership is a key force behind transformation at the school level. Where we encountered particularly strong leaders at the district level, typically the District Education Officer (DEO), the vision and plan for leading the integration of SHRP and installing and supporting head teachers with proven leadership and school turnaround track records was palpable and exciting. Transfers of SHRP trained teachers undermine MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 2 the efforts of even the best and most determined head teachers. Ultimately, the way to address this is to enlarge the cadre of trained teachers, which can most effectively be done by accelerating pre-service teacher training where large numbers can be reached at a relatively low additional cost using existing structures and trained staff, and where the training is setting new behaviors and practices, not displacing old ones. Health and HIV/AIDS. Efforts in health and HIV/AIDS have moved forward less quickly and in general independently of reading, due to a variety of factors including different funding streams, changes in PEPFAR policy and decreased PEPFAR funding, management structures, and staffing challenges. Still, SHRP is reaching the substantially increased target number of learners, training the target number of teachers using an updated health and HIV/AIDS manual, and exceeding the counseling and guidance targets for learners. Rollout of the minimum HIV/AIDS package of interventions to schools is quite behind and efforts to include HIV/AIDS indicator data into the education information system stalled. While widely appreciated, the centerpiece of the HIV/AIDS and health activities at the school level, the School Family Initiative (learners meet after school in group with a teacher to discuss health and HIV/AIDS topics) is implemented with varying commitment and frequency, and lacks structured step-by-step content for leaders (teachers) to use in SFI sessions. We make a number of recommendations to address these findings and conclusions. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 3 SECTION I: EVALUATION PURPOSE AND EVALUATION QUESTIONS A. EVALUATION PURPOSE The purpose of the Mid-Term Performance Evaluation of the USAID/Uganda School Health and Reading Program (SHRP) is to assess the effectiveness of activities to date using qualitative and quantitative data and identify progress in achieving its planned five-year results against RTI’s planned achievements for this stage of the work. As a mid-term evaluation, it also provides recommendations to maintain or improve progress in achieving results. Importantly, given the mid-term evaluation is undertaken within the context of the parallel USAID-funded P&IE Contract, it sheds additional light on the findings of the first annual SHRP impact evaluation and provides very useful context for P&IE’s CLA activities (continuous monitoring and feedback). See Annex A for the evaluation statement of work. The performance timeframe examined is May 2012 – May 2014. Fieldwork took place over a three week period in June and July of 2014; see Annex B for a timeline of this work. Cumulative performance data showing SHRP progress against targets covers the period May 2012 - September 2013, drawing on the PMP and one annual report within this timeframe. B. EVALUATION QUESTIONS The key evaluation questions for the SHRP Mid-Term Performance Evaluation are: • Design. Does the program design and structure adequately support and facilitate achievement of the desired results? • Implementation. Has the program been implemented according to plan and is it on track to achieve its overall objectives and results? • Results. What are the key factors for differences in performance (reading skills acquisition and HIV/AIDS knowledge and skills) in schools receiving the same interventions? • Sustainability. To what extent will the program’s components and subcomponents continue without USAID assistance? • Effective use of funds. What are the implications and recommendations for potential scale-up1 of program interventions? In what ways can the programs be more cost effective? • Management/Coordination/Lessons Learned. How can program design, management, and execution become more efficient in achieving program goals? Sub-questions for each key evaluation question are included in the interview guides in Annex C, Data Collection Instruments. We tailored interview guides to each audience: central government officials, USAID, RTI, district officials, language boards, schools, and implementing partners; focus group discussions (FGDs) with adolescent learners and School Management Committees (SMCs); and class observation. The report structure follows the evaluation questions and sub-questions (see Exhibit 1), which track closely with those in the “Plan for Conducting the Performance Evaluations of Uganda SHRP” prepared and vetted in meetings with USAID and RTI as part of P&IE start-up in October 2012 in order to ensure transparency in the evaluation process. 1 “Scale up” refers to efforts to expand the program’s reach to more people over a wider geographical area. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 4 Exhibit 1: SHRP Key Evaluation Questions and Sub-Questions 1. Design: Does the program design and structure adequately support and facilitate achievement of the desired results? a. Y ____ N ___ b. Supporting evidence 2. Implementation: Has the project been implemented according to plan and is it on track to achieve its overall goals and objectives? a. Overall opinion Y____ N ___ b. Supporting evidence (key activities undertaken to date): c. Implementation challenges and how addressed d. Remaining implementation challenges and proposed solutions 3. Results: What are the key results and contributing factors for differences in performance (reading skills acquisition and HIV/AIDS knowledge and skills) in schools receiving the same intervention? Result 1: Improved early grade reading and transition to English a. Observed improvements among P1 and P2 teachers in teaching literacy in local languages and English? i. Y ____ N ____ ii. Supporting evidence b. Observed improvements in literacy among P1-P3 students? i. Y ___ N ____ ii. Supporting evidence c. Reading interventions/strategies with greatest impact on reading skills acquisition? d. Factors contributing to high achievement e. Factors contributing to low achievement Result 2: Improved attitudes, knowledge, and practices a. Observed improvement in health seeking behaviors among students? i. Y __N __ ii. Supporting evidence b. Observed improvements in providing HIV/AIDS related information among teachers? i. Y__ N __ ii. Supporting evidence c. Observed improvements in parent and community engagement in HIV/AIDS activities? i. Y ___ N___ ii. Supporting evidence d. Health interventions with greatest impact on improved HIV/AIDS knowledge and practices? e. Factors contributing to high and low achievement? 4. Sustainability: To what extent will the program’s components and subcomponents continue without USAID assistance? a. Will programs continue without USAID assistance? Y____ N ____ b. Does funding exist to sustain programs? c. Does capacity exist to sustain programs? d. Recommendations for strengthening program sustainability 5. Cost-effectiveness: What are the implications and recommendations for potential scale-up of program interventions? In what ways can the programs be more cost effective? a. Is the program run cost effectively? i. Y ___ N____ ii. Supporting evidence b. Recommended strategies for cost-effectiveness c. Is the program scalable? Y ___ N ___ d. Recommended strategies for scaling up the project 6. Management/Coordination/Lessons Learned: How can program design, management, and execution become more efficient toward achieving program goals? a. Comments on management, coordination, capturing of best practices and lessons learned MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 5 SECTION II: PROGRAM BACKGROUND A. SHRP PROGRAM DESCRIPTION Background. In its Country Development Cooperation Strategy (CDCS) of 2011-2015, USAID/Uganda included support to the education sector within a Health Development Objective (DO3). This recognizes the “linkages between education – particularly girls’ retention in school – and many economic and health benefits…”2 SHRP is USAID/Uganda’s flagship five-year project supporting this goal, aimed at achieving “Increased Literacy and Health Seeking Behaviors” through integrated approaches to Intermediate Result (IR) 3.1.1, Health-Seeking Behaviors Increased, and sub-IR 3.1.1.1 “Improved Literacy.” SHRP is a cooperative agreement (CA) implemented by RTI with main subcontractor World Education Inc. (WEI). Results Framework. The results framework below shows SHRP’s Program Objective and its two results and their development hypotheses and IRs, as modified (CA, October 2013, p. 12). There is not a development hypothesis at the Program Objective level providing overall direction and linking the sub￾elements, but is provided descriptively in the original USAID solicitation (RFA) and subsequent CA. Exhibit 2: SHRP Results Framework Hereafter, this report refers to each result as “R1 Reading” and “R2 Health.” District and evidence-based approach. SHRP supports a number of Government of Uganda (GoU) policy initiatives and therefore an overarching goal is achieving and deepening host country ownership and using a district-based approach. In the original solicitation and ultimately the CA, USAID emphasized the value that SHRP’s predecessor activities’ (UNITY and PIASCY) demonstrated from working through the Ministry of Education and Sports (MoES) systems and structures and engaging with district political, educational, and community structures. USAID noted that while it wanted the SHRP implementer to “improve models for teacher training,” it also wanted it to “engage more strategically with districts and communities, and better integrate HIV/AIDS activities into the MoES and the PEPFAR systems.” (CA, p. 7) For both reading and health education, USAID mandated a shift from an input orientation to a data- 2 USAID/Uganda Country Development Cooperation Strategy 2011-2015 p. 18, and SHRP CA, p. 3 b. Recommendations for strengthening management, coordination, and capturing best practices/lessons learned Program Objective: Increased Literacy and Health-Seeking Behaviors Program Result 1: Improved EGR and Transition to English Development Hypothesis: By focusing interventions on the nexus of language, pedagogy, and instructional materials, USAID can significantly improve students' early grade reading and P3 literacy scores within targeted schools and districts. IR 1.1: National Policies to Support Reading Developed IR 1.2: Materials Developed to Support early grade reading IR 1.3: Classroom Teaching of Reading Improved IR 1.4: Advocacy and Support for Reading Increased IR 1.5: Reading Programs and Policies informed by data and research Program Result 2: Improved HIV/AIDS KAP Development Hypothesis: By strengthening cross-sector coordination between USAID's health and education partners, USAID can significantly improve teachers' and students' HIVIAIDS knowledge and skills within targeted schools and districts. IR 2.1: Improved Planning of MoES HIV Prevention Response IR 2.2: School-Level Impact of HIV/AIDS and Health Education Improved IR 2.3: HIV/AIDS Education Programs and Policies Informed by Data and Research MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 6 driven, evidence-based approach to demonstrate measurable impact on early grade reading and teacher and student knowledge of HIV/AIDS and health. SHRP’s five-year program results: 1. National policy framework and thematic curriculum enhanced to strengthen the pedagogical framework in early grade reading and transition to English 2. At least 3.5 million children demonstrating improved reading skills over the baseline levels for those grade levels through direct program support and National Scale Up (bolded language added in CA Mod Two, Oct. 28, 2013) 3. At least 10 percent of P2 students in target schools and districts demonstrating sufficient reading fluency and comprehension to “read to learn” 4. 65 percent or more of students meeting Uganda’s national literacy standards by P3 (National Assessment of Progress in Education (NAPE)) 5. At least 60 percent of children reached through direct program support will demonstrate improved reading skills for those grade levels 6. Equity improved across genders, geographic regions, and languages in early grade reading fluency, and in literacy at the P3 level (NAPE) 7. Language-based, instructional materials developed for teachers and students to support the P1-P4 thematic curriculum and promote a reading culture 8. HIV/AIDS education assessment and reporting integrated into MoES systems 9. Cross-sector health and education coordination on HIV/AIDS and health strengthened at the national, district, and school levels 10. Improved HIV/AIDS and health knowledge demonstrated by teachers and students in target districts over the baseline levels for target group Additional SHRP program description is provided in Annex D, including detail on R1 Reading and R2 Health targets, strategy, initiatives, and interventions as well as CA modifications. B. P&IE CONTRACT The purpose of the P&IE contract is to provide continuous performance monitoring and feedback for adaptive management, mid-term and final performance evaluations, and an annual impact evaluation to identify the extent to which SHRP’s literacy and HIV/AIDS prevention activities result in measurable impact on student learning. P&IE is a five-year effort implemented by prime contractor NORC at the University of Chicago with subcontractor Panagora Group. NORC leads the impact evaluation and Panagora leads ongoing performance monitoring and feedback to RTI, and the Mid-Term and Final Performance Evaluations. Importantly, the impact evaluation, performance monitoring, and performance feedback effort is not co￾located with RTI, which has allowed evaluators to maintain a position as independent and objective third party monitors. Performance and Impact (P&IE) Evaluation Timeline Evaluation Milestones Due Date Design Report for Performance and Impact Evaluation Jan. 31, 2013 Fist Impact Evaluation Report April 30, 2014 Mid-term Performance Evaluation Report Oct. 31, 2014 Second Impact Evaluation Report April 30, 2015 Third Impact Evaluation Report April 30, 2016 Final Performance Evaluation Report April 30, 2016 Fourth (Final) Impact Evaluation Report April 30, 2017 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 7 SECTION III: EVALUATION DESIGN, METHODS, AND LIMITATIONS A. EVALUATION DESIGN The SHRP Mid-Term Evaluation is a formative, cross-sectional, descriptive and analytical performance evaluation employing qualitative methods of data collection including document review; key informant interviews (KIIs) at the national, district, and school level; FGDs; and school and classroom observation. We also drew on the continuous monitoring data and performance feedback from P&IE activities. (See Annex E with a summary of all performance feedback memos and RTI’s comments on use of the feedback provided.) Quantitative data available in project documents was also reviewed and close-ended questions in the tools used for key informants also generated quantitative information. We employed purposeful sampling of schools and districts. Data quality and analysis was validated through triangulation of multiple sources and stakeholders. This performance evaluation complements P&IE’s impact evaluation; while the impact evaluation measures program impact and the degree to which the end results – in particular, reading outcomes -- are being achieved and attributable to SHRP interventions, the performance evaluation looks more closely at implementation and provides rich qualitative data on stakeholder perceptions, beliefs, and thinking, which has been analyzed by a cross￾disciplinary team of health, education, and development experts to develop findings, conclusions, and recommendations. See Exhibit 1 which details the key evaluation questions and sub-questions, which were further adapted for each audience in guides provided in Annex C. Guides were developed by our evaluation team, led by our literacy specialist, and subjected to a high level of review, scrutiny, and refinement prior to travel and in particular during our start-up Team Planning Meeting. To assess program effectiveness, the mid-term performance evaluation: • Assesses the extent to which the program components are achieving goals and objectives as defined in key program documents, i.e., cooperative agreement, results frameworks, work plans, the performance management plan (PMP), and reports • Provides an understanding of progress by program rationale, impact, effective use of funds, and sustainability (engagement and ownership) • Identifies whether there are management, coordination, and implementation practices that need to be maintained, stepped up, modified, or discontinued • Consolidates lessons and best practices to promote scale up • Examines the validity of SHRP’s development hypotheses To assess achievement of results, the performance evaluation examines progress through September 2013 in reaching SHRP’s overarching five-year results (see p. 32) per the annual report available in the performance evaluation timeframe. USE OF EVALUATION BEST PRACTICES The SHRP Mid-Term Performance Evaluation uses evaluation best practices such as: • Using subject matter specialists (literacy, HIV/AIDS, health) on the evaluation team • Obtaining implementing team input on evaluation methodology and questions (part of P&IE startup in October 2012) • Collecting and analyzing qualitative and quantitative information • Reinforcing local capacity with local specialists on the evaluation team • Transparency in the evaluation design and dissemination of findings MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 8 B. DISTRICT AND SCHOOL SAMPLING METHODOLOGY We used purposeful sampling to select districts and schools. Tables 2 and 3 below show the final districts and schools selected, which yielded a blend of regions, new and traditional districts, rural and urban areas, local languages, and a range of high-to-low performing schools. Primary schools sampled have R1 Reading and combined R1 Reading and R2 Health programming; and secondary schools sampled implement R2 Health activities. In seeking a balance between high- and low-performing schools, we used RTI’s classification in which schools are ranked according to observable intermediate outcomes (versus learner outcomes), identified during support supervision visits, such as: R1 Reading: Materials delivered and in use; signage displayed; storage available for early grade reading materials; and classroom display of learners’ and teacher’s work. R2 Health: Schools have “families” with the recommended number of learners per family (25) and each family is under a “parent-teacher”; School Family Initiative (SFI) registers are available, kept in a central place, and well maintained (e.g., members are registered, work plan is updated, attendance is shown for every meeting, and a term summary provided); term work plan is developed with regular meetings per set schedule; Guidance and Counseling (G&C) registers are well maintained, e.g., learners counseled are recorded and issues are recorded by day and month. We used RTIs ranking to select the following districts and schools to visit. Table 2: District Selection from Among SHRP Program Districts (bold = selected for site visit) Cluster Local Area Language Region Districts Result area 1 Luganda Central Wakiso, Gomba Results 1 and 2 1 Runyankore/Rukiga South West Kiruhura, Bushenyi, Kabale 1 Ateso Eastern Kumi, Katakwi, Serere 1 Leblango Northern Apac, Lira, Kole 2 Runyoro/Rutoro Mid-Western Masindi, Kyenjojo, Kabarole 2 Acholi Mid-Northern Gulu, Pader, Kitgum 2 Lugbarati West Nile Arua Result 1 only 2 Lumasaba Mid-Eastern Mbale, Sironko, Manafwa 3 Lugwere Mid -Eastern Budaka, Pallisa, Kibuku 3 Ngakarimojong North East Nakapiripirit, Napak, Abim 3 Lukhonzo Mid-Eastern Kasese 3 Lusoga East Central Iganga and Kamuli Source: USAID/Uganda SHRP PMP. Version: September 19, 2013 Table 3: School Selection from Among SHRP Intervention Schools District School Name R1 Reading R2 Health Rural/ Urban High/Low Performing Wakiso Nakiwogo Primary School R1 Urban High Kabale Ihunga Primary school R1 Rural High Lira Owinyo Primary school R1 and R2 Peri-urban High Wakiso Kitende Secondary school R2 Peri-urban High Kabale Kihorezo Secondary school R2 Rural Low Lira Agweng Secondary school R2 Rural Low C. DATA COLLECTION METHODS Our approach to each data collection method is described below. Document review. We reviewed the original CA and three subsequent modifications, annual work plans, MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 9 PMP, quarterly progress reports, SHRP survey data, and P&IE’s annual impact evaluation; SHRP Early Grade Reading Assessment (EGRA) and Knowledge, Attitudes, and Practices (KAP) survey reports; HIV/AIDS information; thematic primary school curriculum; SHRP teacher training guides, learner textbooks, and support supervision monitoring tools; teacher lesson plans and registers. We also reviewed reports and tracking documents developed and compiled by the P&IE as part of CLA activities, including third party monitoring data, monthly third party reports, and monthly performance feedback memos. The goal of the review was to obtain information to answer the key evaluation questions and identify primary data collection and verification needs for the field. KIIs. KIIs were conducted to assess whether program activities are contributing to the achievement of results, whether management of the program is functioning well, and to identify any implementation challenges and/or bottlenecks. They were conducted with different stakeholders at national, district, and school/community levels. At the central level in Kampala, we interviewed: • Representatives from USAID and PEPFAR • GoU representatives including MoES commissioners overseeing R1 Reading and R2 Health; National Curriculum Development Centre (NCDC); Directorate of Education Standards (DES); Teacher, Instructor Education and Training, Uganda National Examination Board • RTI sub-awardees • RTI/SHRP top and mid-level managers and field assistants (FAs) based in districts At the district level, we interviewed Municipal Education Officers (MEOs) and District Education Officers (DEOs), District/Municipal Inspector of Schools (DIS/MIS), PTC Principals, Coordinating Center Tutors (CCTs), and Local Language Board (LLB) representatives. At the school level, we interviewed head teachers, trained teachers, and School Management Committee (SMC) chairpersons and members. We interviewed 88 key informants, as presented in Table 4 below. Table 4: Key Informants Interviews and Focus Group Discussions Level Type Number of Persons Total KIIs National Level (Kampala) USAID 3 MoES 8 RTI* 16 Implementing Partners 11 District Level Kabale officials 5 Lira officials 6 Wakiso officials 3 School Level Head Teachers 7 Trained Teachers* 29 Total KIIs 88 Focus Group Discussions School Level SMCs (5 FGDs) 17 Learners (3 FGDs) 46 Total FGD participants 63 Grand Total 151 * RTI personnel were interviewed individually, and the R1Reading and R2 Health teams were also interviewed as group. Trained teachers were interviewed in groups. FGDs. We conducted eight FGDs, five with SMCs and three with adolescent learners at secondary schools learners aged 18 to 19 (age approved by the Uganda Institutional Review Board (IRB)). The SMC FGDs were conducted at all but one school (due to a change in our schedule that the SMC could not accommodate); and with adolescent learners in all three secondary schools visited. The FGDs with SMCs MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 10 allowed assessment of community engagement, while the adolescent learner FGDs provided invaluable information on the effectiveness of SHRP school level R2 Health activities. Both types of FGDs yielded recommendations for improving school reach into communities and overall program design. FGD participation included 17 SMC members; and 43. SMC FGD participation varied widely, reflecting the level of community engagement under SHRP as well as an oft￾commented lack of volunteerism countrywide. Participation among SMC FGDs was 30 percent women, and 70 percent men. Learner FGDs were quite lively, with an average of 14 participants per FGD, and almost equal participation of girls and boys. Classroom observations. We visited three primary schools to examine the extent to which teachers are applying the reading methodologies learned during SRHP workshops. In each primary school, we observed two classes, for a total of six classes and a combination of P1/P2 literacy and English. The team also assessed the conditions of the classroom, use of SHRP teacher and learner materials, learner participation in reading activities, and the interaction between teachers and learners for Result 1. D. DATA COLLECTION TOOLS AND ANALYSIS Based on the key evaluation questions, we developed interview guides tailored to each audience, including: 1. KII guide for national SHRP stakeholders (GoU, USAID, RTI, and RTI sub-awardees) 2. KII guide for district SHRP stakeholders (DIS, DEO, MEO, PTC, CTT) 3. KII guide for Head Teachers 4. KII guide for Trained Teachers 5. Classroom Observation Guide (P1/P2 Literacy and English) 6. FGD Guide: Secondary/Adolescent School Learners 7. FGD Guide: SMC See Exhibit 1 above for a list of key evaluation questions and sub-questions that informed all guides; Annex C for full guides; and Annex F for a full listing of all KIIs and FGDs. Each evaluation question was addressed through a triangulation of data and information gathered from multiple sources (e.g., GoU, USAID, RTI, partners) and stakeholders. The documents and information collected through KIIs, FGDs, and site visits/observations were analysed by the performance evaluation team through a process of identifying key themes by evaluation question. School Management Committee (SMC) Referred to as a “board of governors” in Uganda’s Education Act of 2008, an SMC’s 12 members represent local community leaders. They oversee management and represent the local community and parents’ interests in the school. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 11 E. PERFORMANCE EVALUATION FRAMEWORK Table 5 below aggregates under each evaluation area the key question and data sources. Table 5: Key Evaluation Questions and Data Sources Mid-Term Evaluation Questions Data Sources Document Review KIIs, FGDs, and/or classroom observation Design Key question: Does the program design and structure adequately support and facilitate achievement of the desired results? • CA and amendments • Program reports: PMP, quarterly, annual, ad hoc reports, work plans and presentations, minutes • Ministry/USAID/program strategy documents KII: • GoU including MoES, DES, Uganda National Examination Board, NCDC, LLBs • USAID (including PEPFAR POC) • RTI and its sub-awardees Implementation Key question: Has the program been implemented according to plan and is it on track to achieve its overall objectives and results? • PMP, monitoring data, program reports • P&IE reports of observed SHRP events, monthly reports, and performance feedback memos (which include RTI comments after a monthly discussion) All KIIs and FGDs Results Key question: What are the key factors for differences in performance (reading skills acquisition and HIV/AIDS knowledge and skills) in schools receiving the same interventions? • PMP, monitoring data, program reports • P&IE survey data and annual impact evaluations KII: • GoU: DEO, DIS, LLB, Head Teachers, trained teachers, CCT, PTC • KIIs with USAID, RTI staff, and teachers in intervention districts • FGDs with SMCs and secondary school students in intervention districts Sustainability Key question: To what extent will the programs components and subcomponents continue without USAID assistance? • Program documents • Classroom instruction materials • Policy documents and curricula All KIIs and FGDs Classroom observation Effective Use of Funds Key question: What are the implications and recommendations for potential scale￾up of program interventions? How can the program improve the use of funds? • Program documents All KIIs and FGDs Management/Coordination/Lessons Learned Key question: How can program design, management and execution become more efficient in achieving program goals? All KIIs F. ETHICAL CLEARANCE The evaluation protocol was submitted to NORC IRB and the Uganda National Council for Science and Technology (UNCST) for ethical clearance and granted approval (SS 3487) with the restriction that learners be age 18 and above. All participants were informed about the purpose of the evaluation. Verbal MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 12 informed consent was used for the key informants and written informed consent was obtained from learners who participated in the FGDs. G. DESIGN AND METHODOLOGY LIMITATIONS The P&IE team recognizes the inherent limitations and challenges associated with evaluating the performance of a large-scale program that is both national and local in its scope. Through P&IE, we have collected information on SHRP activities starting in May 2013, in particular on events observed as part of third party monitoring carried out by our Resident Evaluation Manager, Evelyn Namiburu, and Senior HIV/AIDS Evaluation Specialist, Dr. Stella Neema, under the supervision and support of Panagora, in particular Evaluation Team Leader Betsy Bassan. As a result, we brought a stronger understanding of program interventions into the evaluation, reducing the amount of time needed for orientation to the basic program fundamentals than typically is required by outside evaluators. Given our access to SHRP documents, we analysed the work plan and PMP against progress as available in cumulative progress reports (through September 2013), which increased our understanding of SHRP coming into the evaluation and, importantly, provided a frame for discussing and assessing progress. We projected less time than is usually allocated for performance evaluations, roughly by half, assuming less time would be needed given our starting base of understanding. A key learning we will apply to the final performance evaluation is that the basic steps in a performance evaluation remain the same, notwithstanding the additional information, and need proper time allocation. While the relatively small sample of six schools in three districts presents some limitations on the degree to which the findings can be generalized, this is offset through purposeful sampling and extensive triangulation of data sources from among district officials (DIS/MIS, DEO/MEO, PCT, and CCT) who have visited a large number of schools and can provide a more holistic view of the district. Overall, this mid-term evaluation provided significant insights on how the program is working, its achievements and challenges, through which we identified a number of opportunities for increasing program efficacy in its remaining out-years, and for enhancing third party monitoring. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 13 SECTION IV: FINDINGS, CONCLUSIONS & RECOMMENDATIONS Findings, conclusions, and recommendations are organized and presented under each evaluation question below. Our findings, conclusions, and recommendations are based on rich qualitative and quantitative data from document reviews, interviews, FGDs, and school/class observations; and represent stakeholder perceptions, beliefs, and thinking as analyzed by our cross-disciplinary evaluation team of health, education, and development experts. In each question, we first present respondent information, including their responses to close-ended questions where applicable, followed by our own analysis, conclusions, and recommendations as evaluators. Evaluation Q1, Design: Does the program design and structure adequately support and facilitate achievement of the desired results? A. Evaluation Q1, Design: Findings Respondent data. Per Table 6, 39 of the 42 key informants interviewed (over 90 percent) believe that SHRP is well designed and the structure adequately supports and facilitates achievement of the desired results. As supporting evidence, respondents frequently cited the strong collaboration and extensive use of existing government education-related structures at the national and district levels. As examples of government education structures integrated into SHRP at the national level, respondents cited the MoES and its associated entities such as the NCDC, Teacher Instructor Education and Training, and DES. At the district level, respondents cited the offices of the DEOs and MEOs, the area, district, and municipal inspection staff, the core PTCs, and the CCTs. The dominant perception is that SHRP is adding value by building on existing structures and programs, e.g., the Thematic Curriculum and PIACSY. Most respondents believe that RTI’s participatory and collaborative approaches are making progress in building ownership and developing national capacity, critical for building toward a consensus to invest national funds in the education sector and specifically the SHRP reading methodology. Evidence of GoU buy-in is the adoption of the SHRP methodology as national policy to be rolled out countrywide. Many respondents also praised the monitoring and support supervision as a robust and satisfying process for district staff, SHRP staff, and school level staff. It is collaborative, participatory, constructive, and allows teachers to get the feedback they need to improve performance. By setting out action steps, it gives the teacher and his/her supervisor a plan for continuous performance improvement. For IR2, respondents cited the value of focusing on teachers for training, with supportive supervision, as channels for reaching learners with health and HIV/AIDS information. There was special enthusiasm about the creation of the Field Assistant position (district-based RTI staff charged with coordinating SHRP activities with local officials and schools), recognizing that this position was not included in the original proposal design. One DEO who was asked how often he meets with the FAs in his district replied that they are like a “right arm, in and out of his office multiple times a day, really facilitating planning and coordination.” Others mentioned the importance of EGRA and KAP surveys to be able to track over time measureable change in teacher and learner knowledge. Table 6: Does the program design and structure adequately support and facilitate achievement of the desired results? District Yes No Kabale 7 0 Kampala 23 3 Lira 7 0 Wakiso 2 0 Total 39 (93%) 3 (7 %) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 14 Several respondents said that SHRP would have achieved even more if RTI: 1. Initiated community mobilization activities in Year 1 as planned 2. Been better able from the start to identify appropriate experts and teachers for teacher training workshops 3. Started advancing public-private partnership formation earlier as mentioned in the RTI proposal to leverage additional resources to enhance impact, especially at the school level (e.g., supplies, uniforms, classroom posters, talking compound signage, etc.) Some respondents believe SHRP would have greater impact if it had been designed with an early childhood education component, given international research showing the positive impact of early childhood education on early literacy and school readiness. (While we understand early childhood education was not part of the SHRP design, we include it as valuable input for future activities aimed at improved early grade reading.) When asked about the design and structure of SHRP’s R2 Health, many respondents, in particular government officials and partners, district officials who would be expected to be familiar with the entire SHRP program, could not comment due to lack of knowledge or awareness about R2 Health activities. The lack of information and involvement in R2 Health became increasingly apparent over the course of the evaluation. Many expressed urgency, given the health and HIV/AIDS issues in Uganda, for better interface between the reading and health activities and more involvement of R1 Reading stakeholders in supporting R2 Health activities. While R2 Health activities are not examinable but rather extra-curricular at the school level, this does not lessen the need for informing and involving all SHRP stakeholders in them. The few respondents especially conversant and involved in R2 Health said the current MoES counterpart for this result, Guidance &Counseling, does not cover the breadth of initiatives encompassed by this cross-cutting initiative. This results in less advocacy and ministry support for this result. To remedy this, the MoES HIV/AIDS Technical Working Group was recommended as a more appropriate counterpart. While learners in FGDs appreciate the opportunity to discuss health and HIV/AIDS related topics, they expressed the need for knowledgeable resource speakers and more reliable information at SFI sessions. In seeking to further understand this comment, we learned that SFI does not benefit from the methodological discipline of the reading program. Rather, a list of topics is provided for meetings; and teachers leading the sessions have resources such as the enhanced PIACSY handbook and facilitation guidelines. Because there are no step-by-step instructions (as exists for the SHRP reading program), teachers must either research the topic in the list provided and outline the session themselves or make do with the knowledge they have in the eight SFI sessions per term. Most teachers choose the latter option. B. Evaluation Q1, Design: Analysis and Conclusions Based on respondent comments and our observations and analysis, we conclude that: • SHRP is well designed and structured to achieve R1 Reading • SHRP is building related capacity among national, district, and local MoES-related entities and officials; and is developing country ownership critical for mobilizing future GoU funding and long-term sustainability around the R1 Reading model. This came through especially clearly in interviews with district staff, head teachers, and trained teachers who feel equipped and motivated to use the SHRP model to transform early grade education • The creation of the Field Assistant position to ensure strong coordination and communication between SHRP in Kampala and district officials and schools was a very innovative and important MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 15 staffing modification that greatly increased program effectiveness • SHRP would have more buy-in from community stakeholders had it initiated community mobilization activities with parents and community leaders earlier, as planned. It was clear from district and school visits that the lack of parent and community engagement translates into fewer resources in schools and less family and home reinforcement of program objectives. This is felt particularly in rural or poor communities where education competes with the need for children to help with agricultural work or other income-generating activities. A SHRP community mobilization specialist began work in June 2014 and it is important that this element is stressed going forward • Initial trainings might have had more of an impact with appropriate participants. However, it seems that difficulty with discerning issues in advance was unavoidable, given the role of schools and districts in nominating participants. With experience, SHRP’s ability to negotiate more appropriate participation increased • The absence of an overarching development hypothesis to guide SHRP and provide a theory of change linking the reading and health-HIV/AIDS components creates over-reliance on explanatory text in the RFA and cooperative agreement in achieving the Program Objective, “Increased Literacy and Health-Seeking Behaviors.” Lack of an overarching hypothesis exacerbates the inherent challenges in managing and implementing multi-sectoral programs, leaves open the possibility that intermediate results become managed more as separate activities, and risks losing synergies and programming opportunities in the original design • A counterpart for R2 Health with broad MoES responsibility would be able to elevate the importance of R2 Health and advocate for its inclusion across the ministry and educational sector C. Evaluation Q1, Design: Recommendations • For RTI: Ensure community mobilization activities - to harness commitment and support for the program – with parents and families, political and religious leaders, local officials and businesses, and health workers get quickly underway and do not get bogged down in lengthy planning and initiation processes • For USAID: Develop a guiding development hypothesis at the Program Objective level that supports “Increased Literacy and Health-Seeking Behaviors” and provides a theory of change that captures the causal linkages between the ability to read and healthy choices. This would provide a framework to guide management of activities supporting the Program Objective. Given USAID’s stated intention to embrace multi-sectoral programming, it would be good to document and share this experience and lesson learned • For USAID and GoU: A ministerial counterpart assignment is outside the manageable interest of an implementer partner, and would need to be addressed by USAID and Government if they deem the matter worthy of attention. Based on our information and analysis, it seems that a MoES counterpart for R2 Health with broader responsibility than G&C (an important but particular aspect of health and HIV/AIDS) that more fully represents the broad and diverse activities of the ESS HIV Prevention Plan could elevate the importance of R2 Health and advocate for and mobilize resources and attention across the ministry. Similarly, the suggestion that the chair of the MoES HIV TWG is a logical counterpart seems plausible, given its breadth and reach across the ministry MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 16 Evaluation Q2, Implementation: Has the program been implemented according to plan and is it on track to achieve its overall objectives and results? A. Evaluation Q2, Implementation: Respondent Comments Per Table 7, respondents overwhelmingly believe SHRP is on track to achieve its goals and objectives by 2017. Many respondents noted that the early issues which delayed start-up and led to late distribution of teacher guides and pupil textbooks were resolved and expressed admiration for RTI’s ability to turn around a delicate situation, and then sprint through material development and other start-up stages to recover lost time and achieve challenging targets on schedule. At the national level, the MoES and SHRP senior management team highlighted the following key accomplishments, which we have also observed directly in our third party monitoring activities: • Developed the process and systems with language boards and writing panels in collaboration with NCDC for language materials development • Established relationships with district officials and PTCs • Trained master trainers from among MoES structures and entities who in turn trained teachers and head teachers • Provided tangible reading materials to teachers and learners accompanied by supportive monitoring and supervision • Accomplished these outcomes with strong participation from MoES structures at the national and local government levels Echoing what we heard from all three districts visited, the Wakiso DIS enthusiastically said that the DEO and SHRP are working together to achieve GoU objectives for education, and affirmed that SHRP is on track. Respondents were similarly supportive of R2 Health objectives, but, per Table 4, typically less informed and less certain that SHRP is on track with these: many fewer respondents were able to answer for R2 Health than to the same question for R1 Reading, 17 for R2 Health versus 38 for R1 Reading). B. Evaluation Q2, Implementation: R1 Reading B1. Evaluation Q2, Implementation: R1 Reading Findings R1 Reading Progress by Immediate Result Table 8 reflects R1 progress achieved through end of FY13, representing 16 months of program implementation. SHRP cumulative results through September 2013 indicate that it has either met or exceeded targets for four of the five IRs (IR 1.1, IR 1.2, IR 1.3 and IR 1.5). The one IR that lags behind is IR.1.4: advocacy and support for reading increased. A description of accomplishments and implementation challenges follows the results table. Table 7: Is the project on track to achieve its overall goals and objectives? District R1: Reading R2: Health Yes No Yes No Kabale 6 0 3 0 Kampala 22 1 4 1 Lira 7 0 7 0 Wakiso 3 0 3 0 Total 38 (97%) 1 (3%) 17 (94%) 1 (6%) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 17 Table 8: Reading Progress by IR Table 8: R1 Reading Progress by Intermediate Result Intermediate Result Indicator Target: 20133 Cum. Actual (2013) On track (Y/N) 1.1: National Policies to Support Literacy Developed Number of laws, policies, regulations, or guidelines developed or modified to improve early grade reading instruction and transition to English 2 Literacy Strategy Benchmarks 2 SNE Lit. Strategy Y 1.2: Materials Developed to Support Early Grade Reading Number of primers and teachers’ guides developed 4 9 Y Number of textbooks and other teaching and learning materials via Ugandan Government support 84,510 350,000 Y 1.3: Classroom Teaching of Reading Improved Number of SHRP teachers/educators who have successfully completed in￾service training trained via U.S. Government support 1025 teachers 2,600 Y 410 HTs 1300 Y 30 CCs 60 Y % of observed teachers that are conducting readings lessons in accordance with set standards. 50% 68% Y 1.4: Advocacy and Support for Reading Increased Number of PTAs or similar ‘school’ governance structures supported 799 0 N # of activities to Promote Reading at the National level 3 1 N Proportion of schools participating in community activities supporting reading (reading competitions, reading awareness days, literacy week). 50% 0 N LLBs supporting efforts to strengthen early grade reading in local languages. 4 4 Y 1.5: Reading Programs and Policies Info EGRA in local language and English conducted in intervention and control schools EGRA 4 LL ENG EGRA 4 LL ENG Y EGRA data used to inform education policy, programming, and implementation 1 = 2013 1 (Basic Ed Action Plan) Y Source: PMP (January 2014) and Annual Report, May 2012–September 2013 IR 1.1: National policies to support literacy developed. The project aimed to support two policy initiatives in Year 1. SHRP is currently supporting the GoU in drafting a Literacy Improvement Education Strategy, and SHRP’s Voluntary Service Overseas partner is assisting with drafting a special needs manual. Therefore, it is on track for achieving this result. IR1.2: Materials developed to support Early Grade Reading. SHRP has developed the following instructional materials: • Produced Early Grade Reading teaching and learning materials for Cluster 1 P1 and P2, and Cluster 2 P1 3 The evaluation team found it difficult to identify cumulative targets to date in the PMP and related documents. For future reporting, we recommend SHRP report on annual and cumulative progress to date. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 18 • Developed and currently editing Early Grade Reading teaching and learning materials for Cluster 1 P3, Cluster 2 P2, and Cluster 3 P1 to be delivered to schools by 2015 • Began revising Cluster 1 P1 and P2 materials to integrate feedback and align with later Cluster 2 and Cluster 3 materials Per SHRP’s 2013 Annual Report, SHRP originally planned to distribute teacher and learner materials to the first cohort of schools by February 2013. Due to significant delays and printing issues, materials arrived in schools in October 2013, during the third term of the school year and five months after P1 teacher training (May 2013). In hindsight, the MoES and SHRP partners recognize that processes for developing orthographies and materials from scratch and working through multiple government structures for review and approval are time-consuming and slow-moving tasks that were under-estimated. Given the challenges in producing and disseminating print materials, as well as the critical need for textbooks in the classrooms, many informants said SHRP’s greatest achievement was “getting books into the hands of the pupils.” To date, SHRP has developed, printed, and distributed 350,000 primers for P1 and P2 and is on track to reach its target of 433,324 for 20144 . IR 1.3: Teachers’ Ability to Teach Reading Improved. Through a cascade-training model, SHRP has trained Master Trainers, district-level teacher trainers, teachers, and CCTs in the SHRP reading methodologies and materials for P1and P2. Training takes place at PTCs or university colleges of education. Because of the delay in instructional materials, SHRP held an initial teacher training in January 2013 on literacy skills and instructional activities for developing phonological awareness and vocabulary, increasing the amount of printed resources in the classroom. After the materials became available, the R1 Reading team reported that training for P1 teachers and head teachers took place in 432 schools for the first cluster of schools in May 2013. This five-day training was followed up with a three-day refresher training in September 2013. In January 2014, a second cohort of teachers received initial training. The master trainers have also conducted a Training of Trainers (ToT) for in-service CCTs, pre-service tutors, and the DIS. The DIS/ MIS were trained in quality assurance and the CCTs in mentoring teachers. In 2013, SHRP exceeded the target for training teachers (1,701 trained out of 1,025 targeted) and head teachers (835 trained out of 410 targeted). One reason targets were exceeded is that some schools had more than one P1 teacher, and the Deputy Head Teachers were invited to ensure the sustainability of the methodologies. The project also exceeded its target for teacher performance by 18 percent with 68 percent of observed teachers conducting reading lessons in accordance with set standards compared to the 50 percent target. The project fell short with training CCTs, reaching 20 of the 30 targeted. One explanation was lack of remuneration and conflicts with other training that offered such incentives. Overall, the project is on track for achieving its training targets for 2014. IR 1.4: Advocacy and Support for Reading Increased. Due to delays in other result areas, IR1.4 activities have lagged behind significantly. Although the target set was 799, none of the Parent Teacher Association (PTA) or SMC members interviewed reported receiving any kind of support and few were familiar with SHRP activities. National and community levels activities to promote reading were placed on hold. Community mobilization activities were delayed for a number of reasons including staff turnover and targets that have since been deemed by SHRP managers as “overly ambitious.” Targets for community-based reading activities have been reduced from 50 to 20 percent of schools. The SHRP Social Mobilization Officer was visiting schools at the time of the evaluation to assess school capacity to mobilize parents and determine the total possible number of meetings per cluster. The RTI Chief of Party 4 RTI notes that, “At the onset of the program, a report of Uganda languages (Assessment of Language Readiness) was conducted which included a comparison of their readiness for use in the classroom. Languages were recommended to be used in the first cohort while further development was done with other languages. Those recommendations were not followed, languages were adopted for Cohort 1 that had unsettled orthographies which contributed to some of the delays in producing those materials.” MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 19 stated that SHRP is planning to build the capacity of SMCs to perform their roles and involve them in SHRP reading and health activities. To date, little has been done in this regard. Meaningfully engaging SMCs to support SHRP activities is critical for augmenting resources, ensuring children attend school, reinforcing reading and healthy practices at home, and promoting program sustainability. Local Language Boards. According to the SHRP team, the LLB is a key strategic partner responsible for adjudicating language issues and carrying out community activities in support of R1 Reading, including sensitizing communities on the importance of local language. SHRP activities include strengthening LLBs. To date, SHRP has helped revive and/or establish LLBs including drafting constitutions and action plans, which are in various stages of completion. According to RTI, some members of Local Language Boards have been involved in settling orthographies, creating EGRA survey instruments, training assessors, and writing or editing pupil books. However, our interviews with LLB members indicated dissatisfaction with the level of engagement and follow-up to date, and eagerness to be involved. Like the SMCs, LLBs and their individual members are key local advocates and leaders to create community and parental support for reading and local language instruction. 1.5: Reading Programs and Policies Informed by Data and Research. The main activity under IR1.5 is the administration of the EGRA, which is on track. Per interviews with Centre for Social Research and our first-hand knowledge of the data collection activities via P&IE, SHRP has completed the EGRA for C1, both baseline in February/March 2013 and follow-up in October/November 2013, and the baseline for C2 in February 2014. The next EGRA will be conducted in October/November 2014 for C1 and C2 to assess reading performance in all eight languages. R1 Reading Implementation Challenges and How They Were Addressed While R1 Reading is largely on track for achieving its results, it has faced a number of challenges due to internal and external factors. Internally, the project has been challenged with timely materials development, consistency and quality of training across schools, and fidelity to implementation of the continuous assessment monitoring form. While RTI was able to overcome these challenges and largely achieve the R1 Reading targets, they are important to consider for future implementation and sustainability of the project. Project delays. Numerous respondents including from USAID, GoU, and partners noted that SHRP’s most significant challenge has been delays at project start-up while RTI and the MoES worked through a number of serious issues. For a period, activities could not be advanced. RTI states in its first annual report that, “As a result, the original project timeline had to be altered a number of times throughout the year. The absence of a structured channel of information sharing within the MoES also meant that information flow was fragmented and decisions were often delayed, delegated, or taken in an ad hoc manner by government officials.” According to the Chief of Party, SHRP rectified the situation by continuing to dialogue with the MoES and including in that dialogue a new Deputy Chief of Party with very strong communication and interpersonal skills; as a former district official, he was able to engage with the MoES and create understanding on USAID requirements related to allowances, and develop commitment to support SHRP as a critical means of realizing the GoU’s education and health policies. This was a significant turnaround. Materials development, quality, and distribution. Delivering materials on time in Year 1 was very challenging. Even without the delays described above, the timeline was very aggressive given what is needed for textbook and training materials development, especially when taking into consideration orthography development and printing in new languages. Ultimately, due to delays related to the MoES relationship, SHRP staff had to develop an entire Teacher Guide and textbook and edit them within three weeks with one facilitator coordinating the writing workshops in four languages. As a result, teacher MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 20 guides were not ready for the first round of training so SHRP used photocopies in the training sessions. SHRP now allows five weeks for materials development with a facilitator for each language. This allows the time needed for writers using still relatively new orthographies to work along with reading specialists to develop decodable books appropriate for P1 children. SHRP has increased its full-time technical staff to support this process. Consultant linguists are used in materials development as they bring new insights and continuity to ensure quality and standardization. While the teacher guides and student textbooks are of good quality, some minor issues were identified with content and organization that could affect consistency of implementation. The evaluators, FAs, and teachers noted that although the book content matches the thematic curriculum, the organization and content of the units are not always aligned. For instance, the stories, vocabulary, and sentence structure in P1 English books are not well linked to the topic (e.g., sentence: “a dog on a log” under the topic “things we make”); and, the organization of the activities in the pupil textbook do not correspond with the sequenced steps in the teacher’s guide. A P1 teacher also remarked that it is difficult to beat the words in English because there are no examples of segmented words. The mismatching of words with the theme has resulted in some teachers improvising content to teach the theme in the thematic curriculum. For instance, teachers noted that they often improvise material when they find that the vocabulary words do not match the theme for the unit. When teachers add new words to the lesson, we observed that they do not know how to segment the new vocabulary word as they rely on the teacher guide for guidance. Additionally, some teachers are still focused on teaching content rather than phonemic awareness and phonics, thus the mismatched content could distract teachers from focusing on key literacy skills. In one case, the evaluators observed that rather than focusing on teaching letter sounds and syllables, a teacher spent a lot of time teaching definitions of vocabulary words associated with the theme and not the letter identified for the unit. Trained teacher continuous development. Nearly all teachers interviewed stated the training was too short and requested additional training. When asked how many days they have been trained, most stated 3-5 days and that it was insufficient to be able to fully grasp and internalize the reading methodologies. Many teachers requested refresher training and school-based continuous professional development, such as peer observation and coaching. Head teachers and teachers noted the lessons are difficult to complete within the 30-minute timespan, especially for English. With large classes, head teachers stated the English lessons are overloaded and teachers often run overtime. The evaluators observed that teachers in the literacy class often spent 45 minutes on reading and 10-15 minutes on actual writing. While this may not affect the quality of implementation, it is important to address this during the trainings so that teachers do not become exhausted and skip critical aspects of the lesson or spend more time on non-essential steps. Support Supervision and Mentoring. While head teachers have been trained in the SHRP methodology, many do not have sufficient knowledge and skills for independently providing on-site observation and mentoring. In some schools, head teachers are proactive, but teachers reported that the CCT are more knowledgeable and better able to support them than the head teacher. Also, because the CCT only visits once per month teachers suggested building the capacity of the head teacher to provide support and to collaborate with the CCT and DIS and teachers for ongoing professional development. The evaluators recognize that some high-performing schools have adopted the SHRP approach and do practice peer observations, mentoring, and joint lesson preparation, but observed that the majority of schools require greater assistance for peer exchanges and reflection to become routine practice. Assessment. Classroom observations and teacher interviews showed that teachers are not consistently using the rubric for continuous assessment (the Continuous Assessment Monitoring (CAM) form), which is for assessing approximately five children per class. Teachers at Nakiwogo School in Wakiso stated the form is difficult to use. Other teachers thought that because the form was only provided for Year 1, it was no longer in use. Teachers are instituting the practice of calling on individual students to read, but admit MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 21 that it is hard to track five students per day in class with as many as 100 learners. One teacher in Wakiso has developed her own innovation, in which she groups learners according to their reading pace, advancing them as they progress in vehicles of increasing complexity (bicycles, cars, buses, planes). However, the majority of teachers do not monitor students’ performance against the five competencies to identify struggling learners and provide differentiated instruction. The DIS in Wakiso believes the CAM is possible to apply, but that teachers need more training and demonstration sessions. External factors beyond the control of the project, but that have hampered or affected implementation, include issues of remuneration, salary delays, teacher transfers and community perceptions. Remuneration. The deep-seated “facilitation culture” in Uganda in which GoU employees expect allowances for participating in donor-funded project activities, affected full MoES participation from the inception of the project. While MoES involvement has improved, respondents reported that GoU staff participation in SHRP activities is still largely limited to those who play a direct role in project implementation and are facilitated by SHRP. Salary delays. Another systemic challenge outside of SHRP’s control has been GoU delays in salary payments to teachers and district education officials, who sometimes go without pay for as long as six to eight months. During Year 1, SHRP operated under the nearly constant threat of teacher strikes over delayed and cancelled pay. Strikes substantially affected activities in Q3 of Year 1 due to the delayed start of Term 3 of the academic year. Many respondents mentioned salary payments and teacher demoralization as a grave challenge. Implementation of SHRP training is perceived as hard work that requires preparation and focus, but motivation is reduced when current efforts are not rewarded. Too often this results in teachers integrating methods that require less preparation (e.g., “I do, we do, you do” and multi-sensory approaches) and only developing teaching and learning aids and classroom displays when assisted by a peer, head teacher, or CCT. Teacher transfers. Teacher transfers to non-intervention schools have also posed a challenge, resulting in a loss of momentum and investment. To mitigate this, SHRP tracks replacement teachers and invites them to refresher training so that they are provided additional support from head teachers and CCTs; and FAs try to intercede with DEOs to minimize transfer and/or keep SHRP-trained teachers within the same cluster. Placing FAs in DEOs has helped track trained teachers. The FA in Kabale noted that the DEO consults with him prior to transferring teachers and aims to retain trained teachers within the SHRP intervention area. Community Perceptions. According to the DES and NCDC, there is resistance to local language instruction among teachers, parents, and education officials who do not buy into the concept of learning to read in the local language. The LLB believes that this is partly due to the perceived value of the language. If the language is not used beyond P3 through the university level, people will not see the use of learning to read in the mother tongue language and will continue to promote English. Therefore, they stress sensitization is the key to changing community and parental perceptions. B2. Evaluation Q2, Implementation: R1 Reading Analysis and Conclusions • Despite a delayed start-up, the project is either exceeding or on track for achieving its targets for national policies developed (IR1.1), materials development (IR1.2), classroom teaching improved (IR1.3), and reading programs and policies informed by data (IR1.5) • Community engagement in advocacy and support for reading (IR1.4) is lagging behind schedule due to project delays and revision of targets. Meaningfully engaging SMCs and LLBs to support SHRP activities is critical for augmenting resources, achieving community buy-in for bilingual MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 22 education, reinforcing reading and healthy practices at home, and promoting program sustainability • While outside of the manageable scope and contract of SHRP, external challenges such as remuneration of government officials, delayed salaries, and resistance to the local language, are external factors hampering the full participation of MoES at all levels and could potentially threaten the sustainability of the project, but these factors have not impeded results • To increase implementation effectiveness and sustainability, SHRP should focus on materials revision and improving teachers’ knowledge and skills for both implementing the early grade reading methods and conducting continuous assessment. As identified by teachers, coaching or peer mentoring could be an effective means to further training • The majority of schools require greater assistance for support supervision, peer exchanges and reflection for literacy instruction to become routine practice. Building the capacity of the head teacher to provide on-site observation and mentoring, peer exchanges, as well as collaborating with the CCT and DIS and teachers to provide in-service refresher training are necessary for reinforcing and sustaining the early grade reading pedagogies B3. Evaluation Q2, Implementation: R1 Reading Recommendations There are four key recommendations; all of which are within the scope of the SHRP project and some envisioned: • Community engagement. As SHRP embarks on ramping up community mobilization activities for IR 1.4, it should re-engage the LLB and initiate communication with the SMC to engage them as reading promoters and advocates of mother tongue instruction. The SMCs and LLBs are eager to participate in the project; thus, SHRP should take advantage of this interest and provide opportunities for consistent follow-up and engagement • Teacher refresher training. SHRP should provide refresher training in the step-by-step lessons to help teachers follow the sequence and timing of the lessons; and, in continuous assessment and differentiated instruction to support teachers with developing a user-friendly assessment tool to identify struggling or advanced learners and provide remedial instruction or more advanced reading text as needed • Materials revision. During the revision of the teacher guide and pupil textbook, SHRP should pay close attention to the coherence of the unit theme and vocabulary words, stories, etc. to promote not only reading, but reading comprehension, which should be taught as an integrated skill alongside phonics, word analysis and vocabulary (see p. 19 above). Pagination in the teacher guides should correspond with the pupil textbooks for ease of reference • Head Teacher training. SHRP should strengthen the capacity of head teachers to coordinate school-based professional development, such as peer observation and coaching, and joint lesson planning among trained teachers as well as planning of school-based, in-service refresher training workshops working with master trainers in the Coordinating Centre or trained teachers at the school level C. Evaluation Q2, Implementation: R2 Health C1. Evaluation Q2, Implementation: R2 Health Findings R2 Health Progress by Intermediate Result Per the Table 6 below and discussion thereafter, as of the end of 2013, SHRP has achieved most of its 2013 targets for IR 2.2 and 2.3, with IR 2.1 still lagging behind. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 23 Table 9: Result 2 Progress by Intermediate Result Result 2: Improved HIV/AIDS Knowledge, Attitudes, and Practices Intermediate Result Indicator Target (2013) Actual (2013) On Track (Y/N) 2.1 Improved planning of MoES HIV prevention response Number of elements of the HIV and AIDS assessment and reporting system operational in the existing EMIS (number of elements operating are 5)* 4 0 N Proportion of MoES departments with demonstrated use of HIV/AIDS information for decisions about HIV/AIDS programming as a result of SHRP supported data and data system 50% (8/16) 0 N 2.2 School level impact of HIV/AIDS and health education improved No of targeted population reached through individual and or small group level HIV prevention interventions that are based on evidence and or meet the minimum standards required 250,000 M = 121,795 F = 128, 205 240,192 M= 117,017 F= 123, 175 Y Total number of targeted population reached with individual and or small group level preventive interventions that are primarily focused on abstinence and /or being faithful (AB only) and are based on evidence and/ or meet minimum standards required 250,000 M = 121,795 F = 128, 205 240,192 M= 117,017 F= 123, 175 Y Proportion of schools implementing minimum package approved by MoES in delivering HIV education 80% (658/822 schools) 0 N Number of teachers trained to deliver HIV education in schools with U.S. Government support 4.010 M = 2691 F = 1319 4,055 M = 2721 F = 1334 Y Proportion of program schools that have HIV/AIDS service directory to enhance service linkages 50% (411/822 schools) 9.1% (75/822 schools visited) N No of learners receiving HIV/AIDS related counseling, care and support per year 8,220 M=3,882 F=4,398 12,823 M=5963 F=6860 Y Number of HIV textbooks and other teaching and learning materials provided with U.S. Government assistance 8,020 8,170 Y 2.3 HIV/AIDS education programs and policies informed by data and research No of research studies conducted on HIV/AIDS in the MoES sector to inform policy, planning and decision￾making 1 1 Y Source: PMP (January 2014) and Annual Report, May 2012–September 2013 * Elements of assessment and reporting include: 1) HIV/AIDS education data collection tools. 2) Data collection at school level. 3) Integration of data into EMIS. 4) Analysis of data. 5) Generation of reports based on data. IR 2.1: Improved planning of MOES HIV prevention response. • HIV indicators integration into the Education Management Information System (EMIS). In collaboration with key stakeholders, the SHRP R2 Health team developed eight HIV indicators to be used in tracking progress in implementing the MoES HIV Prevention Strategic Plan 2011- 2015 and making informed, data-driven decisions. The DES approved integration of HIV indicators into the monitoring and support supervision tool used by school inspectors at the MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 24 school level. Per the Year 2 work plan, the next step was to provide technical support to the Education Planning and Policy Analysis Department to integrate the approved HIV indicators into the EMIS and then develop data collection instruments and data analysis strategies to feed into the HIV assessment and reporting system. This is a critical next step for spurring schools to implement HIV/AIDS activities, creating accountability, measuring progress, and future planning. However, the SHRP R2 Health team reported that this work is not now envisioned due to declined availability of PEPFAR funds. SHRP leadership confirmed that this activity is now on hold • HIV & AIDS TWG support. The R2 Health team has supported planned monthly meetings of HIV TWG, which is chaired by the MoES and brings together key ministry departments and affiliated institutions such as Kyambogo University, NCDC, and Uganda National Examination Board to discuss critical HIV/AIDS issues in the education sector Per the Year 2 work plan, TWG activity support was to focus on: • Finalizing and following up on the “Institutional Gap Analysis to Sustain ESS HIV & AIDS Response” and the study on MoES coordination frameworks • Disseminating KAP survey results within the MoES • Holding a national stakeholders meeting on HIV in the ESS during which KAP survey data and EMIS-generated HIV data could be shared SHRP has supported the TWG meetings. The Institutional Gap Analysis concluded that the ESS HIV & AIDS Prevention Plan is not well known and lacks implementation and coordination guidelines, the MoES has little ability to advocate for education-related HIV/AIDS funds, districts have limited capacity to lead and sustain an education-related HIV/AIDS response, and there is lack of clarity around school leadership roles and responsibilities for plan implementation. The national stakeholders meeting, which will help the MoES and HIV TWG address such issues, is now planned as regional meetings in September 2014. Unfortunately, the PEPFAR funding situation has led RTI and WEI to much more significantly shift out of technical support and systems strengthening at the national and district levels than indicated in the current SHRP Results Framework, i.e., reduce further activities under IR 2.1. IR 2.2 School level impact of HIV/AIDS and health education improved IR 2.2 follows a five-pronged strategy (2013 annual work plan, p. 27), as follows: 1. Train teachers to effectively deliver HIV/AIDS education to young people in schools. • Materials development. SHRP used the PIASCY materials developed under the predecessor UNITY project as a starting point to produce what are referred to as “enhanced PIASCY materials” including teacher reference manuals and a participatory methodology supplement. Like the R1 Reading textbooks, they are being piloted while undergoing a MoES approval process. These materials are: o PIASCY Primary Education and Training Complementary Teacher’s Hand Book 2 o PIASCY Post-Primary Education and Training Complementary Teacher’s Hand Book o Supplement on Participatory Methods: Tools for Actively Engaging Students The enhanced PIASCY materials are comprehensive with basic facts on HIV/AIDS, sexual and reproductive health, life skills, adolescence, relationships, child sexual abuse, teenage pregnancy, and guidelines for implementation of child-friendly HIV/AIDS programs in school. While under review by the MoES, these materials are being used in draft form in training sessions and co-exist at schools along with the older PIACSY materials MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 25 • Strengthen HIV/AIDS counseling, care, and support at school level. In January 2013, four HIV/AIDS G&C training sessions were conducted regionally for school teachers and nurses in primary and post-primary schools and training institutions. A total of 387 teacher counselors (217 men and 170 women) were trained to provide HIV/AIDS counseling to learners in schools across 11 program districts • PIASCY training at primary and post-primary levels: Head teachers and teachers in intervention schools have been trained by skilled facilitators in enhanced PIASCY in May 2013 and January 2014, including basic facts on HIV/AIDS, sexual and reproductive health, child vulnerabilities, and life skills. Training topics included participatory methodologies; designing HIV/AIDS activities; linkages to community service providers; creating a safe and friendly environment for learners; the concept behind SFI and how to use it to reach learners; and the importance of data management and documentation at school level. The first training for Cluster 1 schools comprised 4055 teachers, while 3, 876 teachers from the 1651 supported schools were trained in Cluster 2. Teachers generally indicated they appreciated the training and felt it equipped them to implement activities at the school level. The facilitators included MoES staff (CCTs) and external facilitators 2. Roll out the minimum PIACSY package of HIV interventions SHRP is developing guidelines for implementing the minimum PIACSY package of HIV interventions required by all schools to reduce new HIV infections. A description of what the minimum package includes and SHRP status of work in each element of the package follows: • Integrating HIV/AIDS education into classroom subject lessons -- this is not being done • HIV/AIDS issues integrated into co-curricular activities – See SFI below • Referrals and linkages for HIV and AIDS services – a referral list has been compiled and distributed to some schools but much fewer than planned • Participatory and active HIV/AIDS talking compound (display of HIV/AIDS information in the school environment) – this is mentioned in the Year 2 work plan but not yet begun • G&C of pupils by trained HIV/AIDS counselors – teachers have been trained and this element is underway • SFI implementation – the SFI program is underway but quality and frequency vary greatly 3. Improve support, supervision, and collaboration with teachers to deliver the enhanced PIASCY program • Rapid monitoring and data collection. FAs are responsible for rapid monitoring of all 1,651 schools where they collect data on all learners reached with “abstinence, be faithful” prevention messages in primary and post-primary schools via the registers for SFI and for Guidance & Counseling. The data is entered into the Monitoring and Evaluation of Emergency Plan Progress (MEEPP). FAs reported that school records are uneven in quality and too often activities are not well recorded, which is consistent with our third party monitoring observations regarding record￾keeping at the school level. As of April 2014, data was collected from a total of 1,619 schools; Minimum package of school HIV interventions:  Integrate HIV/AIDS education into classroom subject lessons  Integrate HIV/AIDS issues into co-curricular activities  Provide referrals for HIV and AIDS services  Have participatory and active HIV/AIDS talking compound (HIV/AIDS information on school grounds)  Provide counseling and guidance of pupils by trained HIV/AIDS counselors  Carry out the School Family Initiative (SFI): “SFI uses small learner groups or “School Families” under the care of a teacher who plays a parental role for the group. The purpose…is to provide a mechanism for in-depth information sharing, guidance, care and support to learners in the area of HIV and AIDS, general health and life skills to small groups of 20-25 pupils…happens on a weekly basis during the school term.” (SHRP Year 2 Work Plan, p. 30) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 26 the remaining 32 schools have not yet started programs, reportedly due to conflicts with sports and other activities, non-payment of salaries, or busy schedules. This is consistent with reasons cited for missing sessions at schools where SFI exists. If SFI were part of the core curriculum (versus being co- or extra-curricular) it would be given greater priority in the school calendar and data collection processes5 • Support supervision. Support supervision is conducted by R2 Health staff, FAs, MoES staff, CCTs, and district inspectors. SHRP anticipated that with an average of five teachers per school, about 8,255 teachers could be provided support supervision per term. Due to funding and staff limitations, monitoring has shifted to a biannual basis. Schools are ranked according to performance based on rapid monitoring visits and schools where implementation levels are low are prioritized for support supervision visits with participating MoES staff. Some informants, in particular FAs, cited transport as a challenge in visiting schools due to impassable roads during the rainy season, mountainous terrain, and use of hired motorbikes which are risky and prone to accidents • Referral services. SHRP is providing a list of referral services related to HIV/AIDS and health to schools. The referral lists are provided as laminated sheets bound together; many schools do not yet have these and the information provided is judged by school-level respondents as too oriented to distant providers and lacking in focus on nearby/lower-level service providers • Talking compounds. Talking compounds, primarily signage with positive HIV/AIDS and health messages, were visible on the grounds of most schools visited. The signs were created under the predecessor UNITY project and today some of the messaging is dated and the signs quite worn. Updating these talking compounds at every school is an SFI program opportunity that could be linked with community mobilization efforts. For example, R2 Health staff from SHRP could help schools organize a competition among “families” in each school for new/updated messaging and communities could be engaged to help, e.g., provide sign materials. We understand that in Year 3 SHRP is working with another USAID implementing partner, Communications for Health Communities, to update guidelines on developing health messages for schools in line with Ministry of Health standards 4. Strengthening and supporting the establishment of the SFI SFI is on course to reach its target of 500,000 learners per annum. By June 2014, 460,800 learners had been reached through the SFI. SFI is generally well regarded at the school level. Learners are organized in small groups called “families” (about 25 members) under the care of a teacher who is the “parent” for information sharing, guidance, care, and support to learners. Families are supposed to meet approximately weekly, which translates to eight meetings per term. While popular, SFI programming varies widely among schools. In large part because it is an extra-curricular activity, meetings are often much less frequent than intended, as we saw in examining the SFI registers at schools we visited. As mentioned above, already over-burdened teachers often are not able to meet the requirements of developing topic plans and in our FGDs, many adolescent learners noted that it would be good to have more knowledgeable resources at their family meetings. Notwithstanding these shortcomings, learners agreed in all three FGDs that they have benefited greatly from the sessions which they find empowering and helpful in gaining comfort with speaking openly on a host of otherwise taboo subjects, whether HIV/AIDS, relationships, unwanted pregnancies, family planning, masturbation, abortion, discipline, and school attendance. Learners also mentioned that the SFI 5 USAID notes that by approximately 2017, “…once the government has signed the school health policy the MoES will start implementing Sexual and Reproductive health as part of the curriculum in early post primary schools and the subject will be examinable.” MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 27 has helped them to share problems openly and seek guidance, to be more assertive, and to look out for each other. In Wakiso, the secondary school visited by the NORC team created a peer leader program among adolescents to make up for lack of teacher interest. This has been very successful and inspired the peer leaders to act as role models and others to emulate them. The trained teachers at Wakiso also mentioned the need for additional resources and materials. They noted that the Straight Talk Foundation sends three copies of “Young Talk” each month and it is “the most read item in the school, helping by itself to promote literacy.” In another “low performing” secondary school visited in Kabale, learners are in an “anti-AIDS club” (a term from prior projects) and said in their focus group that the information obtained in the club has helped them to change their behavior in several ways. They said that it has inspired them to serve as role models, practicing what they advise others to do, e.g., abstaining from sexual activities. They said that now they stay close to school and avoid areas where bad behavior is occurring, whereas before they used to leave school early to go to places where they would meet peers who were bad influences. They also said that their club has taken a stance against teasing and bullying which has been very positive for them. Similar to other FGDs, they reported problems with insufficient materials and the need for additional informed resources at their meetings. The teachers reported that they are overloaded and that these activities encroached on their time. 5. Increasing and institutionalizing parental and community involvement in HIV and AIDS education at the school level for sustainability Per all respondents, including RTI, all aspects of community mobilization – whether parental, community, or SMC – involvement, have been slow to advance. During the evaluation field work, we met the newly-hired community mobilization specialist. With her on board, this area of work should progress more rapidly. This is a critical component for building local ownership, mobilizing parental and community reinforcement of health and reading objectives, generating resources, and in general promoting the sustainability of the program. At the schools we visited, parents and school management committees are largely ignorant of the program, and this breeds unfounded concerns that can impede program progress. It is important that RTI move quickly to foster better understanding, commitment, and tangible support among parents, communities, and SMCs. IR 2.3: HIV/AIDS education programs and policies informed by data and research The key activity under this IR is conducting and supporting the use of HIV/AIDS KAP surveys. The baseline KAP survey was conducted in two phases, in June 2013 and October 2013. The data on teachers’ and students’ knowledge and skills are to be used to orient the ESS response to HIV/AIDS. The first phase KAP survey was conducted by local partner DRASPAC in 533 schools in 20 districts comprising 17,284 learners and 2,088 teachers from primary, post-primary and business, technical, vocational, education, and trainings. The results were disseminated to the MoES HIV TWG, which provided constructive feedback. Importantly, R2 Health staff has started using the findings from the KAP survey to further improve the design of health interventions in schools. The KAP survey data will also be used for the NORC impact evaluation of Result 2 activities, which will occur in early 2016 after the first round of follow-up KAP data collection is completed in Oct/Nov 2015. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 28 Implementation Challenges Key implementation challenges for SHRP in R2 Health are: Structural. Separate and unintegrated planning, budgeting, and management functions between R1 Reading and R2 Health, as discussed in previous sections. PEPFAR policy and funding changes. Early on, shifts in PEPFAR policy led to SHRP programming shifts, funding reductions, and an increase in the numbers to be reached in terms of learners, schools, and districts. While SHRP has worked hard to meet the new requirements, and done very well in ratcheting up the numbers reached, the way SHRP is strucutured exacerbates the difficulties of achieving R2 Health goals by segmenting funds that might serve dual purposes in training, montoring, and supervision, etc., while recognizing and respecting funding source parameters. If addressed, economies of scale may be achieved, and resources might be freed up for such critical activities as the integration of HIV indicator data into the EMIS, and resources stretched to finance more MoES participation in program activities covering both results, e.g., support supervision trips. Appropriate MoES counterpart. As described above, the current MoES R2 Health counterpart does not sufficiently cover the breadth of this result’s activities in the ministry, inhibiting the success of its support to the spectrum of activities envisioned in the ESS HIV Prevention Strategy and reducing opportunities to develop broad ministry ownership and buy-in. Staff turnover. SHRP’s R2 Health team has experienced a challenging level of staff turnover, and considerable delays in filling critically important positions such as the HIV Advisor and lead Results 2 Manager. Meanwhile, expatriate short-term technical assistance was provided over long periods to bridge the gap, which is very expensive and does not build local capacity without staff in place to mentor. Ultimately, one Health Advisor was hired to fill both positions, and there is no longer a “manager” level position for R2 Health. The new Health Advisor said that WEI was slow in orienting her to the work which negatively impacted her productivity. RTI and WEI need to address this in the future and ensure current staff have the information needed for success. Salary non-payment in a co/extra-curricular scenario. As noted, this is outside the manageable interest of the implementing partner, but deserves mention because unreliable salary payments are especially challenging for R2 Health, which largely depends on teacher volunteerism for its flagship activity, SFI. Inadequate outreach and local involvement. R2 Health is adversely affected by the slow launch of SHRP’s community outreach activities, as it stands in great need of the ancillary resources and services that can be mobilized in this way. Such resources could include, for example, parent and community time and materials for talking compounds, or access to District Health Office and clinic staff as knowledgeable resources to provide accurate health and HIV/AIDS-related information at school level SFI sessions. Teacher reticence to discuss sensitive topics. Some teachers felt uncomfortable about talking to children about the risks of early sex, reproductive health, peer pressure, and HIV infections. To alleviate these challenges, the program began using SMS at the end of Year 1 to communicate with teachers and support them in conducting school-based HIV/AIDS activities. SMS messages provide teachers with reminders about conducting activities and contact information for key program stakeholders (e.g., CCTs, DEOs, program staff) so they can seek advice regarding implementation challenges. KAP survey deviations. Prior to the first KAP baseline survey, World Education submitted an IRB protocol to the National HIV/AIDS Research Committee (NARC). NARC approved the research design and associated KAP data collection. However, challenges encountered during the first data collection MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 29 effort in June 2013 led to modifications in subsequent data collection. Specifically, during the first data collection effort (KAP1), for some boarding schools, consent could not be obtained from the parents; as a result in those schools the field teams only surveyed students over 18 who could give consent themselves. Subsequently, the second data collection effort (KAP2) included both schools from new districts as well as boarding schools from the districts already surveyed in KAP1. Unfortunately, an IRB protocol was not submitted to NARC prior to KAP2. Ultimately, NARC invalidated the KAP2 data collection due to the addition of new districts in KAP2 and changes in the NARC-approved Principal Investigator between KAP1 and KAP2. C2. Evaluation Q2, Implementation: R2 Health Analysis and Conclusions • R2 Health has faced many challenges, most particularly: o Significant budget and program modifications due to changes in PEPFAR funding whose impacts were exacerbated by the bifurcated and stove-piped management, planning, and budgeting approach to R1 Reading and R2 Health. While PEPFAR requires separate tracking, and the target groups largely differ (with reading instruction targeting early grades and SFI aimed at older children) with not all schools overlapping, programmatic crossover opportunities became harder to identify due to oversight arrangements. o An MoES counterpart with insufficient interface to represent the breadth of R2 Health within the ministry • Staff turnover and staffing gaps have created delays and persistent institutional memory gaps • All of R2 activities have been negatively affected, although variously, by the different challenges o IR 2.1 is most seriously in jeopardy due to what appears to be an almost complete shift away from systems strengthening, in particular with integration of the HIV indicator data into the EMIS and lack of proactive follow up on issues identified in the institutional gap analysis and coordination framework study o IR 2.2 is mostly on track to meet its targets, in particular reaching the larger numbers of learners mandated by PEPFAR, 250,000; and in teacher training, learners receiving counseling and guidance, and materials provision. Lagging behind however is implementation of the minimum package of HIV interventions, which is seriously off-track (zero schools have achieved this against a target of 658); and reaching schools with service directories (only 9.1 percent have them versus a target of 50 percent) • Importantly, with respect to IR 2.2, while the PEPFAR targets are beginning to be met, the quality of the interface with learners can be significantly improved by upgrading SFI health and HIV/AIDS information and ensuring that regular meetings are held, its extra-curricular status notwithstanding. Achieving this is related to implementing the minimum package of HIV Adolescent Learner Suggestions for SFI • Provide variety in materials for family meetings, e.g., video, music, print, etc. • Distribute more informational reading material* • Update the old PIACSY messages on sign posts (talking compound) and use music and drama to convey messages • Give more time to the meetings (only 40 min.) • Create weekend events • Include “people with knowledge” in meetings; now “have groups of 30 and find no one with knowledge” • Feature speakers, e.g., experts on different topics, PLWHA, etc. • Bring messages into sports events, e.g., banner • Provide notebooks so students can journal learning • “Liven up” meetings, e.g., debates between families • Take field trips, e.g., to visit PLWHA • Heighten motivation, e.g., t-shirts with updated health and HIV/AIDS messages *Straight Talk’s newsletter is main reading currently distributed, about three/school. One head teacher commented that each issue is so highly read that by itself it is contributing to increased literacy! MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 30 interventions. Clearly, SFI will also benefit from the accountability inherent in collecting and entering HIV indicator data into the EMIS, per IR 2.1, which is now on hold • IR 2.3 is on track to meet its targets via the KAP studies. The issues related to KAP2 most probably occurred due to staffing gaps and institutional memory lapses • There is some outreach to the MoH (the BCC TWG) but more outreach and coordination with health entities could bolster R2 Health programming, e.g., the Uganda AIDS Commission at the national level, or the district and community level health structures at the local level • Similarly, R2 Health activities at the local level could substantially benefit from parental and community involvement (in particular through time and material resources as well as home reinforcement of health messages and practices, both of which promote sustainability) but SHRP has as yet only scratched the surface in this regard • There are some very interesting models, best practices, and suggestions arising from experience to date with SFIs at the school level, e.g., student peer leaders at Kitende Secondary School to compensate for lack of teacher interest and motivation, which has collateral benefits of youth leadership development. In our three FGDs, adolescent learners had many exciting suggestions for strengthening the SFI program C3. Evaluation Q2, Implementation: R2 Health Recommendations • Integration of HIV indicator data into the EMIS. Achieve better balance between systems strengthening and direct outreach to build accountability into the system, specifically: SHRP should not drop the activity regarding integration of HIV indicator data into the EMIS -- keeping in mind the adage that “what gets counted gets done” used as a front page headline in a recent SHRP newsletter • Minimum package of HIV interventions.: Prioritize implementation of the minimum package of HIV interventions, which is variously advanced and of varying quality • SFI. Strengthen SFI to ensure quality “family” meetings with well-planned sessions and content, drawing on available local informed resources and integrating varied and interesting programming via youth-oriented health and HIV/AIDS films, music, and print materials. Enlist USAID in accessing other USAID implementing partner resources that might provide speakers and materials. As a flagship R2 Health activity, upgrading SFI is urgent • WEI home office support. Ensure WEI provides the level of home office support needed to fill vacancies quickly and orient new staff fully D. Evaluation Q2, Implementation: Status of Five-Year Program Results While the mid-term evaluation has focused on the SHRP Results Framework, we have also tried to assess progress against overall SHRP five-year results. To this end, Table 10 summarizes PMP targets and progress to date for each of the 10 five-year program results. We group the five-year results by process and outcome, while retaining in parentheses the number each result is assigned in the CA. The comparison of PMP targets against progress to date conforms to the analysis by results areas above. Generally, the project is on track for meeting five-year targets in early grade reading and somewhat behind in meeting five-year results in health and HIV/AIDS. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 31 Table 10: SHRP Five-Year Program Results Progress Against Targets PMP Targets Progress/Status Process Results National policy framework and Thematic Curriculum enhanced to strengthen the pedagogical framework early grade reading and transition to English (#1) 2 2 Language-based, instructional materials developed for teachers and students to support the P1-P4 thematic curriculum and promote a reading culture (#7) 4 (2013) 8 (2014) 9 (2013) 9 (2014) HIV/AIDS education assessment and reporting integrated into MoES systems (#8) 4 (2013) 0 Cross-sector health and education coordination on HIV/AIDS and health strengthened at the national, district, and school levels (#9) N/A, dropped per modification #2 to the CA Outcome Results At least 3.5 million children demonstrating improved reading skills over the baseline levels for those grade levels through direct program support and National Scale Up (bolded language added in CA Mod Two, Oct. 28, 2113) (#2) N/A as national implementation begins in 2015, and five-year mark not yet achieved 65% or more of students meeting Uganda’s national literacy standards by P3 (NAPE) (#4) At least 60% of children reached through direct program support will demonstrate improved reading skills for those grade levels (new result #5, per CA Mod Two, Oct. 28, 2013) N/A as program not yet reached P3 level and program has not yet assessed full year or five-years of implementation Equity improved across genders, geographic regions, and languages in early grade reading fluency, and in literacy at the P3 level (NAPE) (#6) N/A as program not yet reached P3 level At least 10% of P2 students in target schools and districts demonstrating sufficient reading fluency and comprehension to “read to learn” (#3) Comparison of EGRA data at baseline and subsequent years Pending EGRA data following baseline Improved HIV/AIDS and health knowledge demonstrated by teachers and students in target districts over the baseline levels for target group (#10) Comparison of HIV/AIDS and health knowledge baseline, mid-term, and final scores Pending midline KAP survey + Note: This result replaced “55% or more students meeting Uganda's national literacy standards by P6 as defined by NAPE” which could not be achieved under SHRP, as it will not reach P6 by program end. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 32 Evaluation Q3, Results: What are the key results and contributing factors for differences in performance in schools receiving the same interventions? A. Evaluation Q3, Results: R1 Reading A1. Evaluation Q3, Results: R1 Reading Findings Teacher Performance Improvement As reflected in Table 11, 100 percent of key informants interviewed stated they had observed noticeable improvements in P1 and P2 teacher competencies in teaching literacy in the local language and English. District officials and SHRP staff who are directly involved in support supervision visits noted three key observations of improved teacher performance: improved lesson planning, increased use of active and participatory methodology, and increased use of the local language as the medium of instruction. According to district representatives, many trained teachers are now developing lessons aligned with the thematic curriculum compared to non-trained teachers; and the SHRP teacher guide has significantly supported teachers with planning their schemes of work and lesson plans. Secondly, through using SHRP methodologies, such as “I do, we do, you do” methodology, teachers have increased their use of active and participatory methodology in the classroom and consequently learner participation. According to DIS and CCTs’ interviewed in Lira and Kabale, the traditional method (lecturing, scolding pupils, giving instructions before modeling) did not encourage participation; now that the teacher models the behavior first, children know what to expect and are able to fully participate in the lesson. Thirdly, teachers are using the local language as a medium of instruction, which has facilitated communication in the classroom. The Kabale CCT states that as a result of SHRP training, teachers know the orthography, have an orthography language book, and are better able to teach the local language. Classroom observations confirmed that teachers are instructing lessons in the local language, and the majority of teachers are following the lesson plans, as articulated in the teacher guide. Results of Classroom Observations Consistent with SHRP performance measures, the evaluators assessed “5Ts” of effective language instruction: 1) tongue – teaching in the local language, 2) text - ensuring the pupils are reading from printed text, 3) managing time on task (30 Table 11: Observed improvements among P1 and P2 teachers in teaching literacy in local languages and English? District Yes No Kabale 7 0 Kampala 8 0 Lira 7 0 Wakiso 3 0 Total 25 (100%) 0 (0%) Owinyo Primary School, Lira Owinyo Primary School is a low-performing school in Lira District, a very poor area where the uniforms are in tatters, kids are barefoot, and parents ration bits of pencils for fear the kids might lose the whole pencil. This photo of the P2 literacy class captures the engagement in a class with a dynamic SHRP trained teacher. When asked for evidence of the improved literacy, most people cite what is shown in this photo: “Getting the books in the hands of every child.” MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 33 minutes for Literacy 1 and 30 minutes for Literacy 2), 4) testing students using the CAM form, and 5) use of appropriate teaching methods. P1 and P2 literacy and English classes were observed in three primary schools: Nakiwogo primary school in Wakiso, Ihungu primary school in Kabale, and Owinyo primary school in Lira, for the duration of the entire lesson. We offer the following key comments from our observations of Literacy lessons taught by P1 teachers, recognizing the limits set by our small sample size: • Language. The teachers observed taught the entire lesson in the local language in all three districts. In Wakiso, the teacher was assisted by a second teacher and was well prepared with scripted flashcards and a pattern grid on the chalkboard for the writing lesson. This facilitated the smooth transition between activities • Following the sequence of the methodology. Generally, the teachers followed the sequence of the methodology, but did not spend much time on teaching letter sounds or providing opportunities for independent or shared reading. The teacher in Wakiso focused her attention mostly on teaching the theme of the day (accidents) and associated vocabulary words rather than introduction of the letter of the day and letter sounds. The teacher was familiar with the reading methodology, segmenting the word from the previous lesson during the review, but when she began the lesson for the day, she used the whole-word reading approach. Additionally, the vocabulary introduced was not aligned with the scope and sequence in the teacher’s guide. The teacher later explained that because the vocabulary words did not match the theme, she had to revise the SHRP lesson. She further stated that her priority was to teach science, which indicates she is still tied to the traditional curriculum and is not fully aware of the thematic curriculum, which emphasizes literacy and themes over single subject teaching. In Kabale, the teacher largely followed the methodology for Day 4 in the teacher guide. He discussed the previous story, read the book to the students, read with the students, and asked in-text and inferential questions (If a child is hurt, what should they do?). He skipped some steps, such as asking students prediction questions and checking to see whether their prediction was correct after reading the story and allowing time for independent reading. For the most part, he followed the teaching guide • Teaching method. With significant years of teaching experience, the teachers observed demonstrated extensive knowledge of appropriate active teaching and learning methods to engage learners (e.g., singing, breaks, scaffolding, I do/we do/you do). In fact, so much time was often spent on whole group reading and repetition that pupils stopped looking at their books or the board and instead began reciting from memory. Use of the textbooks for reading along, independent or shared reading was limited • Continuous assessment. The P1 teacher in Wakiso was the only instructor who duplicated the CAM form provided by SHRP in Year 1; other teachers did not make the same effort and are using informal assessment methods. The teacher in Wakiso, however, also shared during the teacher interview that she found it difficult to assess learners while marking simultaneously and now records the marks after class. The teacher in Kabale asked several students to read individually, but did not mark their competence level in the continuous assessment form or other record. He and other teachers stated that they had the CAM form for Year 1, but not for Year 2; and because the previous forms had been completed, they could not photocopy them. During the last training in May 2014, teachers requested additional forms from SHRP, but they had not been received yet • Use of the book. Due to the excitement generated by the books among students, some teachers are not fully using them during the lesson. The teacher in Wakiso used the books for a short period for the “Point to the Picture” exercise, but then when she saw the children becoming distracted, she asked them to turn the books face down for the remainder of the class. In Lira and MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 34 Kabale, students followed the text while the teacher was reading the story, but when asked to read independently, they repeated either what the teacher said or read from the board. In another district, the teacher confided that she did not wish to distribute the books because it interferes with the lesson time and prefers to use the blackboard Classroom observations revealed that teachers are not completely comfortable using the books in the classroom because they feel distributing books is a time-consuming process, or because students pay more attention to the pictures in books than the teachers’ instruction. Head Teachers and District representatives thought this was because the books were distributed mid-year. Had they been distributed at the beginning of P1, the students would not take so long to familiarize themselves with the material and, if the teacher used the materials as indicated in the teacher guide, it could be a resource for instruction and practice. One Head Teacher commented that students in P2 who had been exposed to the books in P1 were already using the books correctly and able to read. More time and monitoring are necessary to determine the impact of the books on teaching and learning. While the project has begun to get traction in the schools toward improved literacy instruction, DIS/MIS and CCTs all stated that teachers need additional training, support, and supervision for the new reading methods to become routinized and standardized. Evidence from classroom observations supports this view. Learner Reading Performance Improvement All district stakeholders interviewed observed improvements in students’ general reading interest and efforts (see Table 12). Many informants stressed that pupils are more excited to learn how to read. In Ihunga primary school, a few pupils in P2 who have followed the program in P1 can already read fluently. Having their own book has generated a lot of excitement in the class. As stated by the CCT in Kabale, “For some, it is the first time in their lives that they have had the opportunity to hold a book.” This has motivated pupils to come to school just so they can see the illustrations. It also means that pupils had to be taught how to handle books. The DIS in Kabale has noticed an improvement in the way pupils treat books after a teacher taught them to “handle with care.” Now, as they distribute the books, the students say, “Take care.” The DIS in Wakiso cited the same example. There are several examples of improved student efforts at reading. Due to the availability of printed texts, shy pupils who were afraid to read from the blackboard are now attempting to read. One SMC member stated, “Our children as early as P1 are able to write their names on their own and yet it used not to be like that”. In Lira, the DIS noted there has been a “great transformation” in the level of accuracy that pupils have demonstrated in their exercise books. In some P2 classes observed, learners were able to track the instructional material in the books, following it with their eyes, read aloud, and point to the right words with their fingers; and able to follow teacher instructions to turn to the correct page number. Regional variation. Reading performance among schools and districts vary. A Uganda National Examination Board representative shared his view that the greatest differences are among the urban and rural schools, with the urban and peri-urban performing better. Consistent with the impact evaluation findings, the R1 Reading team has observed that students in Luganda perform relatively better. Based on interviews, possible contributing factors are the orthography is more advanced, education is valued, and the local language is a subject through university. In the northern and eastern regions, which do not perform as well, the language is less developed, there is less stability and economic barriers, which affect pupil reading performance. Therefore, the impact evaluation is critical for identifying results directly Table 12: Observed improvements in literacy among P1-P2 students? District Yes No Kabale 7 0 Kampala 8 0 Lira 7 0 Wakiso 3 0 Total 25 (100%) 0 (0%) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 35 related to the project intervention and not other external factors. The LLB representative interviewed in Kabale also commented that the number of published texts in the local language and whether the language is taught at the university level all contribute to the perceived currency of the language. Factors Contributing to High and Low Reading Achievement High achievement. The top five factors cited for high reading achievement: head teacher leadership, frequency and quality of CCT and inspector visits, teacher willingness, internalization of the methodology by trained teachers, and availability of teacher and pupil materials. Head teacher leadership was ranked first by the R1 Reading team and the most frequent response by stakeholders for both R1 Reading and R2 Health; there was high consensus that an enthusiastic and committed head teacher distinguishes high￾performing SHRP schools from low-performing ones. Other factors mentioned: continuous professional development of trained teachers, supportive parents encouraging children to read and attend school, training head teachers in the methodology, prior exposure to the local language in printed material, and good rapport and teamwork between the CCT, head teacher and teacher. Low achievement. Key factors cited for low reading achievement included: limited parental support, uncommitted head teachers who do not enforce the program, limited or absent supervision, teacher resistance to change, and high pupil and teacher absenteeism. Teachers stressed that orphan children in particular suffer from lack of parental involvement, as guardians are not as concerned with child welfare or education. In schools with high orphan populations, the majority of pupils do not have pencils or exercise books. Other factors cited for low reading performance are: delayed receipt of SHRP materials, , low awareness of the methodology, low trained teacher teamwork, low morale and motivation of DIS and teachers, resistance to local language use, and poor SMC/PTA involvement. The preceding discussion is largely based on respondent perceptions and opinion. The impact evaluation, also being conducted by the NORC team, will provide quantitative evidence on impacts of the R1 Reading interventions on reading performance as measured by the EGRA. Positive impacts of the R1 Reading interventions were not detected in the Year 1 impact evaluation conducted in early 2014; this may partly be explained by the fact that the data collection for the Year 1 impact evaluation occurred just 1-2 months following the training and distribution of instructional materials, leaving little time for that component of the R1 Reading intervention to take effect. A2. Evaluation Q3, Results: R1 Reading Analysis and Conclusions • There is evidence of improved teacher performance in terms of better lesson planning, use of local language as the medium of instruction, and increased use of active learning methodologies. Classroom observations and supervisor interviews indicate that teachers are following teaching guides with varying fidelity to the prescribed steps and are familiar with the reading methodology. Teachers do not spend sufficient time on teaching letter sounds or providing opportunities for students to interact with the text through drawing, independent reading, or shared reading. Some teachers do not distribute the books regularly. Teachers do not consistently assess students’ literacy competencies because the CAM form was not distributed in Year 2, which affects sustainability. Our analysis is that the factors affecting teacher performance are training, years of teaching (some have been teaching for many years and are quite knowledgeable), exposure of teachers and students to the books, teacher initiative to develop their own monitoring forms, and Head Teacher and CCT support • Our analysis of the factors that differentiate high-performing schools from low-performing ones are first and foremost, head teacher leadership; frequency and quality of monitoring visits; degree to which training is systematized, internalized, and reinforced; attitude and willingness of teachers to apply methods; and availability of materials with scripted lessons that are easy to follow and can be MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 36 adapted as necessary • Overall, the evaluation team was impressed with the teachers’ use of the reading methodology, especially P2 teachers. With continued support, supervision, follow-up training, and continuous assessment materials, the training can potentially be internalized and systematized A3. Evaluation Q3, Results: R1 Reading Recommendations • SHRP should continue providing additional and refresher head teacher and teacher training, support, and supervision for the new reading methods so they become institutionalized within MoES structures, internalized by teachers, and reinforced by CCTs, head teachers, and peers • SHRP should monitor book usage and availability to ensure that teachers are distributing them and learners are engaging with them for independent or shared reading, or picture reading • SHRP should train teachers to develop lessons beyond those in the teacher guides so they can accurately improvise material and differentiate instruction (providing remedial exercises, advanced, etc.) while maintaining fidelity to the literacy methodology • SHRP should ensure teachers have a reliable method for assessing students and recording pupil performance, so they can identify struggling learners and provide appropriate interventions B. Evaluation Q3, Results: R2 Health We asked respondents whether they had observed improvements in health-seeking behavior among learners, improvements in providing HIV/AIDS-related information by teachers, and improvements in parent and community engagement in HIV/AIDS activities; and factors contributing to high and low levels of improvements in these outcomes. B1. Evaluation Q3, Results: R2 Health Findings Learner Health-Seeking Behavior Improvement Per Table 13, all respondents perceive there has been an improvement in health-seeking behavior among learners. There is some guidance and counseling data documented in the guidance and counseling registers to support this perception, although the registers we reviewed on our school visits were often not completely filled in. Field Assistants, who usually interact with the learners during support supervision and monitoring, reported that learner personal stories during SFI indicate improved health-seeking behaviors. Teacher Improvements in Providing HIV/AIDS-related Information Respondents perceive that teachers are better able to impart HIV/AIDS and health information to learners because of teacher training in guidance and counseling and enhanced PIASCY. Trained teachers reported that they had benefited from the training, learned more about HIV/AIDS and health, and are now better equipped to provide such information to learners. Many trained teachers also noted the training in guidance and counseling helps them to better engage with learners having particular problems. Most adolescent learners in the FGDs commented that, while they would like more expert resources in their SFI sessions, their parent-teachers are now better able to provide HIV/AIDS and health information and are more open and able to discuss such topics. Table 13: Observed improvement in health-seeking behaviors among students? District Yes No Kabale 6 0 Kampala 6 0 Lira 18 0 Wakiso 5 0 Total 35 (100%) 0 (0%) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 37 Parent and Community Engagement Improvement Given the lack of parent and community engagement in SHRP to date, most respondents did not feel they could adequately respond to this question. Health Interventions with Greatest Impact The SFI was mentioned by all respondents from the national level up to the school level as the main intervention perceived to be having the greatest impact on HIV/AIDS knowledge and practices. It is the “centerpiece” of the school level intervention for R2 Health. Factors Contributing to High and Low Achievement High achievement. Most respondents note the prime importance of head teacher leadership among factors leading to high achievement in R2 Health activities at the school level. Given the extra-curricular nature of the R2 Health activities and the issues in teacher motivation, R2 Health activities, in particular the SFI sessions, can easily be displaced by other competing events on the school schedule unless the head teacher prioritizes them. Other key factors contributing to high achievement are having other highly dedicated trained teachers who, as role models and champions, help generate enthusiasm among the other Adolescent Learners, Kitende Secondary School, Wakiso We observed the SFI at its best at Kitende Secondary School in Wakiso District. The secret ingredient – “show me a good program and you’ll see a good head teacher” – is definitely apparent in both their head teacher and two trained teachers who together provide the leadership this R2 Health program needs to succeed. When the parent-teachers who are to lead the “families” did not follow through sufficiently, they came up with the novel idea of creating peer leaders, shown in the above photo. These peer leaders were very impressive in the FGD, brimming with commitment and good things to say about how the SFI has changed their lives for the better. It has given them an avenue to talk openly about taboo subjects, come to better understandings, and, through practice, gain comfort in speaking up. They were also full of ideas for strengthening the program, including the need for more structure and knowledgeable speakers. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 38 teachers. Support supervision visits from SHRP staff and district officials were also cited as a very motivating factor. It was clear from the Kitende secondary school visit that having strong leadership generates innovative thinking, in their case the creation of peer leaders to make up for absent or unmotivated parent-teachers. Low achievement. There are a range of factors reported by respondents that are perceived as contributing to low achievement in R2 Health school activities. A major factor is that SFI is not part of the core curriculum; it is on the school calendar but extra-curricular and therefore immediately relegated to a lower status. This in turn means that demoralized teachers, who are not paid regularly, feel overworked and may be uncomfortable with sensitive subject matter, will readily agree to having the SFI sessions postponed for any number of reasons. We heard this comment repeatedly, i.e., SFI sessions are easily postponed and only a few of the planned eight sessions per term actually take place. As noted above, as currently organized, SFI sessions require some real effort on the part of teachers to prepare. The enhanced PIASCY materials provide background material and facilitation guidelines but not scripted session material, as provided by the early grade reading materials. Given the situation with teachers, this becomes another factor influencing low achievement. Another contributing factor is infrequent supervisory visits; this reduces accountability and the motivating effect of outside interest and support. B2. Evaluation Q3, Results: R2 Health Analysis and Conclusions • Overall, respondents perceive an increase in HIV/AIDS knowledge and skills though not in all the schools • The SFI program requires too much effort from overworked and underpaid teachers who do not have the time or motivation to read background materials and create discussion plans for weekly sessions • The SFI program is often treated as optional because it is a co-curricular activity • There are three key factors perceived as affecting the robustness of R2 Health activity at the school level: strong leadership by the head teacher, enthusiasm from a couple of other trained teachers, and frequency and quality of monitoring and support supervision visits B3. Evaluation Q3, Results: R2 Health Recommendations • SHRP should raise a discussion within the MoES aimed at exploring ways to elevate the status of the SFI program within the curriculum and school schedule (see footnote 5 above) • Parent-teachers need better materials to guide each session than simple topic headings and background material, including youth-oriented supplementary material (film, print material, etc.) and support in engaging resource speakers (often available locally through the district health offices and Non-governmental Organizations) • Identify a way to surface and share good ideas that are making a difference in schools like Kitende with its peer leader program – whether exchanges, a rapid review of high-performing programs, or some other way of discovering and disseminating low-cost effective ways to improve this program • Public-private partnerships with businesses should be explored for obtaining motivational items – adolescent learners mentioned things like T-shirts and pens that can be co-branded with businesses • SFI and talking compounds should be more proactively linked so that health and HIV/AIDS messaging on school grounds is more updated and dynamic (changed, for example, on a term-by￾term basis) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 39 Evaluation Q4, Sustainability: To what extent will the program’s components and subcomponents continue without USAID assistance? A. Evaluation Q4, Sustainability: Findings Per Table 14, over 70 percent of stakeholders interviewed believe SHRP activities would be sustained beyond the life of the project and that capacity exists within the MoES to sustain the program. Reasons provided were that SHRP has worked through existing structures (e.g., CCTs, NCDC, PTCs) and is supporting the implementation of the thematic curriculum and MoES education strategy. Capacity building was also evident in participants’ description of their involvement in SHRP. Many were well informed, committed, and had been engaged since the beginning of the project. SHRP has built capacity at all levels by working with the NCDC to develop writing boards and panels, training teachers and master trainers with participation from Teacher Instructor Education and Training, and engaging and facilitating the CCTs and DIS at the district level to train teachers and provide support supervision. Field Assistants reported that SHRP has achieved more than 90 percent attendance at training workshops. The project has also instituted measures to retain trained teachers and capacity in schools by training at least two teachers per school and including both the head teacher and assistant head teacher. When asked about which components would be sustained, CCTs believe the teaching content and methodology, CCT supervision of teachers, and books will be sustained. The massive refresher training at the college level was considered less likely to be sustained without further USAID assistance. Additionally, many respondents stated that more capacity building would be needed in order for the MoES to fully replicate the project in other districts. While most respondents believe there is capacity and will to sustain the program, only about 20 percent were confident that funding was available. Activities considered at risk if funds are not sustained, according to district-level MoES officials, are continuous professional development and support supervision. Several MoES officials at the national level appear optimistic about future funding, saying that the GoU has been funding some activities and integration of the program into the budget has been discussed at education sector review meetings. One passionate MoES/SHRP counterpart stated, “It might be sustained on a smaller-scale, but it will be sustained.” USAID, SHRP and district-level officials agree the MoES has the funding and capacity to sustain the program, but not yet sufficient ownership. Some believe that once SHRP’s impact becomes evident, resources will be identified. The external factors described in Question 2 – lack of remuneration, delayed salaries and resistance to local language instruction-- pose serious challenges to sustainability. Negative perceptions or attitudes of those not directly involved in SHRP were frequently cited as a barrier the project would have to overcome to increase the likelihood of sustainability. Poor attitudes were attributed to several factors: lack of understanding of the value-added of the program and local language instruction, teachers’ sense of demoralization due to work without pay, or lack of rewards for additional effort. Unpaid, demoralized, and unmotivated teachers who may view the program as extra work need continuous encouragement, support, and supervision. One DIS stated, “The number of times you go out is equivalent to the number of times they are motivated. When you don’t go out, they relax.” Frequent head teacher transfers is another Table 14: Sustainability Will program be sustained without USAID funding? Funding exists to sustain program? Capacity exists to sustain program? R1 R2 R1 R2 R1 R2 Yes 75% 70% 35% 6% 81% 76% No 25% 30% 65% 94% 19% 24% Total respondents 24 20 23 17 21 17 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 40 factor threatening sustainability given the relationship between reading achievement and the leadership, commitment, and enthusiasm of the head teacher. If a vibrant head teacher is replaced with someone who is complacent, teacher performance and sustainability are jeopardized. . District-level government officials conclude that sustainability of the program boils down to the value attached to the program and increasing a sense of government ownership. Another factor affecting sustainability is lack of a specific exit strategy for SHRP that clearly defines roles and responsibilities. Respondents questioned whether the GoU is in a position to continue training, support, and supervision to currently trained and new teachers; and suggested the PTCs, who have the capacity and resources for in-service and pre-service training, assume responsibility for sustaining the teacher-training component. The GoU will need a system for maintaining the materials (revision, printing, distribution, and storage). At the community level, the language boards asserted that they should own the process and disseminate information about the importance of mother tongue instruction, and that SMCs/parents should ensure reading resources and habits are cultivated to create a culture of reading in schools and communities. Comments on R2 Health sustainability varied. Some did not feel adequately informed to comment. Others felt it could continue through the availability of the enhanced PIACSY materials. Others felt that it would not be sustained or only sustained if integrated with reading and better linked to the thematic curriculum. B. Evaluation Q4, Sustainability: Analysis and Conclusions The majority of respondents believe there is sufficient capacity (trained teachers and materials) to sustain the program, but insufficient funding may hamper the sustainability of continuous professional development and support supervision, which relies heavily on project funding. District-level government officials conclude that there is still a lack of buy-in within the MoES at all levels and that sustainability of the program boils down to the value attached to the program and increasing a sense of government ownership. Government officials need a better understanding of how the program links to their current positions. Sustainability for R2 Health depends on full implementation of the PIACSY minimum package including a high quality SFI program that is treated as a routine part of the school schedule. C. Evaluation Q4, Sustainability: Recommendations • Exit strategy. Donors need a well-articulated SHRP sustainability exit strategy that clearly defines roles and responsibilities for assuming leadership of project components and funding, particularly for materials revision, teacher refresher training, support supervision, and community advocacy • Best practices. Best practices should also be well documented and shared with all MoES stakeholders to increase awareness of the contributions of the project. Ways to do this include: assigning a champion within the SHRP team responsible for helping the team identify and capture best practices (probably the Communications Officer); adding a “best practices” agenda item as a point for discussion and reflection at all SHRP MoES/donor meetings and periodically (e.g., monthly) in SHRP internal staff meetings; adding a section in quarterly and annual reports for presentation of best practices (versus success stories); and using a variety of venues for sharing best practices nationally and internationally, including P&IE mid- and final dissemination workshops for sharing best practices nationally • Community support. Engage the local community, such as the LLB and SMC, in increasing community support for local language instruction and early grade reading to address unsupportive attitudes: o Facilitate the LLB to increase awareness of the importance of the local language. SHRP can support them to develop or print existing materials in the local language for teachers MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 41 and learners in primary school. They could also print leaflets to distribute to community members during awareness-raising events o Sensitize the SMC: “In the future, we as SMC need to be sensitized about this project so that we can contribute in its oversight just like the other projects where we have been very much involved.” (Wakiso, Entebbe SMC) • Teacher morale. The MoES and SHRP should identify ways to boost teacher morale and provide recognition for their participation in continuous professional development activities. For instance, rather than distributing certificates for participation in training at the end of the five-year project, SHRP could provide certificates at the commencement of each training (similar to the practice for R2 trained teachers). The MoES should consider providing professional development credit at the end of the project or after completion of a number of trainings or follow-up activities, so teachers feel motivated to participate and continue applying what they have learned. Teacher recognition programs for effective early grade reading instructional practices would also be effective Evaluation Q5, Use of Funds/Cost Management: What are the implications and recommendations for potential scale-up of program interventions? In what ways can the programs be more cost effective? At this early point in implementation, and based on the financial data available in program documents (CA, work plans, and progress reports) as well as respondent comments, the scope for responding this question is very limited. Still, we include it as a contract required element of the performance evaluation, and will discuss with USAID approaches for a more in-depth approach for the final evaluation. A. Evaluation Q5a, Use of Funds/Cost Management A1: Evaluation Q5a, Use of Funds/Cost Management: Respondent Comments Most respondents perceive that RTI is exercising good cost management. Examples of supporting evidence cited: extensive use of government personnel, systems, structures, and facilities; and strict adherence to compliance U. S. Government requirements related to allowances. Respondents cited a few areas for potential cost savings, such as materials development which is generally viewed as having required substantial staff and monetary resources for development, printing, and distribution. One respondent noted that after the structures and approaches are in place for materials development, replication might be less costly. This will be verified in October 2014 when the initial pilot textbooks will be revised based on feedback and input from users. Respondents also perceive opportunities for better leveraging of resources through corporate, community, and parent mobilization. In interviewing the R2 Health management team, we learned that WEI duplicates finance and administrative functions in the SHRP country office, including vehicles. Interviews with WEI’s home office and SHRP staff indicate substantial turnover in personnel in the home office, extensive use of home office personnel to provide interim coverage of WEI-provided SHRP staff vacancies, and discontinuities in home office support to the field team, all of which contribute to cost inefficiencies. The R2 Health team expressed considerable frustration with budget, staff continuity, and home office support in bridging staffing and budgetary changes. The R2 Health team tries to piggyback onto R1 Reading resources to reduce costs. Following a review of available financial data, we summarize in Annex G SHRP program budget projections and actuals showing use of funds by major lines items over the 22.5 month period from May 2012 – March 2014, i.e., the period of performance under the cooperative agreement up to the latest quarterly report within the timeframe of the mid-term evaluation. Information used to create this annex is MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 42 drawn from program documents, specifically the cooperative agreement, annual work plans (showing projected use of funds), and quarterly and annual reports (showing actual use of funds). In terms of fiscal years, this review and analysis covers: • Year 1, May 21, 2012 - September 30, 2013 (16.5 months), using SHRP’s extended Year 1 Work Plan and associated progress reports that conclude with the end of the fiscal year; and • Year 2 through the mid-year, October 1, 2013 - March 31, 2014 (6 months), using the SHRP’s Year 2 Work Plan and associated progress reports (up to the last one available during the evaluation period) A2: Evaluation Q5a, Use of Funds/Cost Management: Analysis and Conclusions. It is clear that SHRP makes every effort to involve government personnel and make use of governmental systems, structures, and facilities, e.g., trainers, facilitators, and organizers; training and meeting venues, which often include lodging. SHRP also ensures costs are aligned with U. S. Government requirements. However, in many USAID projects, finance and administrative functions are shared among project partners, reducing the burden of overall finance and administrative costs to a contract or cooperative agreement and ensuring uniform practices and approaches. With SHRP, the overlapping finance and administrative functions among the prime and sub-recipient is potentially an area where cost savings could be identified which, if achieved, might also help facilitate synergies and additional programmatic impact. The R2 Health staffing issues appear to have increased personnel costs (home office/international staff versus long-term national staff) and possibly also contributed to R2 Health program delays. An analysis of the financial data available from the cooperative agreement, work plans, and progress reports6 indicates the following regarding use of funds through March 2014: • Total use of funds for the first 22.5 months was $14,885,376, which represents $26% of total project funds (per the cooperative agreement) as compared to a projected use of funds of $19,684,734, or 35% of total funds • Use of funds was less than projected for the first 16.5 months by $4,385,252, and $414,106 in the next 6 months; for a total of almost $4.8 million lower use of funds than projected over the 22.5 month period • Cost Share funds used during the first 22.5 months was $264,683, which represents 4% of total funds used during this period, as compared to a projection of $2,026,135 or 30% of total funds • The key line items where use of funds was significantly less than projected include supplies, sub￾recipient/grants, and cost share, as follows: o Supplies: total of $4,469,377 unused funds. RTI explained that some of this is due to shifting costs for printing teaching and pupil guides to Other Direct Costs (ODCs) using purchase orders to contract vendors o Sub-recipients/grants: total of $798,853 unused funds. RTI attributes this to initial program delays o Indirect Costs: total of $92,375 unused funds 6 RTI notes two discrepancies with their figures, probably due to the difference between actuals and accruals. Our analysis is based on documents available to the evaluation team (cooperative agreement, work plans, and progress reports). It appears, however, that the possible variation in numbers would not substantially change the analysis. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 43 o Cost Share: total of $1,761,452. RTI notes that cost share was delayed because book donations contributing to cost share did not arrive until March 2014 • Labor and fringe benefits use closely follow budget projections, totaling $2,033,561 over the 22.5 months, just $6,249 over projections • The key line items where spending was over projections include: o Travel: total of $88,188 more than projected o ODCs: total of $388,185 more than projected. As explained above, ODCs absorbed costs for printing of materials. RTI also explained that purchase orders to PTCs for teacher training workshops Budget information provided in annual work plans and progress reports is not currently accompanied by explanatory text, nor is a comparison of actual expenditures to projected expenditures provided. This makes it difficult to draw conclusions on use of funds and cost management. This would be easy to rectify by providing brief explanatory notes (overall summary and a comment for each major line item) and a summary of actual versus projected spending, accompanied by explanations for under- or over-spending. . RTI explains the under-utilization of funds as a result of the initial delays in project start-up due to establishing relationships and agreements with the MoES. Based on our observations, respondent data, and review of program documents, RTI seems to be implementing SHRP in a very cost-conscious manner. While there may be some expenses where cost￾savings might be achieved through rationalization (e.g., combining functions such as vehicle support or R1/R2 supervisory monitoring), streamlined management and financial systems, or more effective decision-making and program support (e.g., the second KAP survey), overall it is notable that program results are on track given the under-spending. A3: Evaluation Q5a, Use of Funds/Cost Management: Recommendations • Assess how R2 Health resources are used with a view to rationalizing, weeding out duplicative functions, whether financial, administrative, or programmatic (e.g., separate field assistants for each result, separate monitoring and supervisory trips, etc.) • Provide budget information in annual work plans and progress reports with a comparison of actual expenditures against projected expenditures, and with brief explanatory notes (overall summary and a comment for each major line item) B. Evaluation Q5b, Program Scalability B1. Evaluation Q5b, Program Scalability: Findings Virtually all respondents believe that both the reading and health and HIV/AIDS elements of SHRP are scalable (see Table 16) and commonly cited the perceived success of the initiative to date. The SHRP model has been approved by the GoU for rollout throughout Uganda. With SHRP working ultimately in 33 districts (including in 2017 the four control districts within the P&IE impact evaluation), Global Partnership in Education supporting 27 districts, and new USAID funding planned, more than 75 percent of Uganda’s districts will be using the SHRP model. Given the view shared among most respondent that SHRP expansion into additional districts represents a kind of scale-up, most offered suggestions for an orderly scale-up into additional districts. Many of these suggestions can also be applied to the MoES scale-up under Global Partnership for Education, namely: MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 44 R1 Reading: • Maintain and/or expand presence and use of field assistants • Revise materials to address inconsistencies between content, themes, and sequencing • Accelerate the pace of training new teachers via the PTCs; and increase use of PTCs, e.g., in support supervision activities, delivery of materials to schools, and to conduct training • Ensure sufficient refresher training, and, as if not more important, follow up support, coaching, and mentoring by CCTs and peers • Build NCDC capacity to host and participate in writing workshops, e.g., assign an NCDC staff person to work alongside SIL Language and Education Development Regarding R2 Health, respondents commented that the health and HIV/AIDS activities are scalable, depending on Government commitment to carry out the program. USAID’s PEPFAR POC said she is eager to see integration and mainstreaming of health and HIV/AIDS into curricular and extracurricular (HIV/AIDS clubs) school activities; after that is achieved, it will become part of the school day and be sustained and scaled up automatically along with other sanctioned school features. Other respondents noted that R2 Health scale-up requires integration of health and HIV/AIDS into pre-service training and integrated monitoring of R1 Reading and R2 Health. B2. Evaluation Q5b, Program Scalability: Analysis and Conclusions There are many lessons and best practices that SHRP is surfacing through its experience in launching activities in initial districts that will be of value in expanding into additional districts as well as to the MoES scale-up through Global Partnership for Education. The evaluation team agreed with respondents’ suggestions (listed above) for expansion of R1 Reading interventions into additional districts. We especially note the strategic and implementation importance of creating district-based field assistant positions; the value of “pilot” reading and instructional materials which allows for continual improvement in the materials; accelerated pre-service training and use of PTCs; follow-up to training via refresher training but most importantly institutionalizing a mentoring and coaching approach to supervision visits; as well as building local capacity within NCDC to host writing workshops. We also very much agree that mainstreaming of health topics into the core curriculum and extra￾curricular activities will ensure the sustainability and scalability of R2 Health. Further, R2 Health’s scalability will be advanced if reading and health are tackled as part of a joint programmatic effort, e.g., include health examples in early reading texts (such as bednet use for malaria prevention). As noted earlier, R2 Health scalability will also be advanced if more broadly positioned in the MoES (versus being pigeonholed in a narrow aspect – Guidance & Counseling - of health and HIV/AIDS education) and thereby also more consistent with PEPFAR and GoU multi-sectoral HIV/AIDS policy; as is, R2 Health lacks the platform to garner attention and develop the support and ownership that ultimately lead to resource commitments and long-term sustainability. B3. Evaluation Q5b, Program Scalability: Recommendations • Accelerate implementation of programming with the teacher training colleges to reach the future teaching workforce at point of formation and where large numbers can be reached at a relatively low additional cost using existing structures and trained staff • Deepen capability of local partners in key areas, e.g., materials development MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 45 Evaluation Q6, Management/Coordination/Lessons Learned: How can program design/management and execution become more efficient toward achieving program goals? A. Evaluation Q6, Management/Coordination/Lessons Learned: Findings Overall, respondents commented favorably on the quality of SHRP management and coordination, from the Chief of Party and Deputy Chief of Party to the field assistants in the districts. There is a spirit of respect and collaboration. There is a closer working relationship with government than with partners and, among partners, there are better relations with those most closely associated with core functions which are also those areas and activities that RTI has prioritized. A number of respondents made remarks such as: “SHRP people are organized. Communication is good. They fulfill promises.” And yet a number of respondents suggested that more advance planning and communication would strengthen SHRP management and coordination. These are probably not inconsistent but rather reflect the fact that SHRP was able to scramble and organize to implement a relatively massive early grade reading program, marshalling the materials and training in the nick of time, with last minute solutions (e.g., photocopied materials) as needed to meet timelines. Respondents showed a sense of excitement about being part of an effort with so much energy behind it and such great potential for positive impact. At the same time, they will be content when the demands can be better anticipated. Respondents varied regarding SHRP inclusivity in planning, but most commented that SHRP involves its stakeholders in planning, is open to suggestions, and flexible in events and training to accommodate participant needs. Many respondents would like more information on SHRP activities. There may be too much reliance on word of mouth and a sense that emails and reports are widely shared within organizations. Many key informants, (e.g., within the MoES), have not seen the quarterly reports which may be a problem of distribution from within the MoES. There is great interest among SHRP staff and stakeholders to have additional ways to network, share information, lessons learned, and best practices, which are currently oriented more to success story formats in quarterly reports and distributed to a limited audience. Respondents offered many good ideas for strengthening management and coordination, which are integrated below into our recommendations based on those we prioritize. Many interviews shed light on the bifurcation in planning, budgeting, and management between R1 Reading and R2 Health., in particular those with individual SHRP staff, the result teams, and the many R1 Reading key informants who had little to no knowledge of R2 Health activities and/or their status. B. Evaluation Q6, Management/Coordination/Lessons Learned: Analysis and Conclusions Based on respondent comments and our own observations and analysis, we also hold a positive view of RTI’s management and coordination of SHRP. We would in particular cite the fact that they were able to reverse the situation with government and develop a collaborative and supportive relationship, carry off a very challenging implementation and rollout exercise, and achieve what is perceived as real progress in literacy, with government committing to a national scale-up of the program. Now that it is well-established, RTI can take the time implement a number of relatively low-effort actions aimed at tightening communications, broadening participation, and promoting knowledge exchange. It can also use the mid-point as a time for reflection on the bifurcated management of SHRPs two results in which each result is almost fully relegated to one implementer, R1 Reading to RTI, and R2 Health to WEI, with separate management, planning, budgeting, and coordination. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 46 C. Evaluation Q6, Management/Coordination/Lessons Learned: Recommendations Tighten internal and external communication • Send regular activity reports/updates directly to a broad audience (don’t rely on forwarding) • Use a multi-faceted approach to all communications (email, phone, mailings) not relying on any one approach or a single communication to get the message across • Alert district officials and head teachers to meetings so they are prepared • Use communications to continually deepen government ownership in the program; as one person commented, “RTI needs to improve communications and let the public know why RTI is here, that it is not independent but supporting the thematic curriculum, an existing ministry program, in order to get the political support that will lead to the financial support” Broaden participation • Expand MoES participation in planning and support supervision visits, particularly district government officials, and develop joint monitoring plans at the local level with local officials, PTCs, CCTs, and head teachers • Ensure every school inspection supported by SHRP concludes with a joint debrief so all parties understand findings and agree on a plan to address them with clear roles and responsibilities Promote knowledge exchange • Develop opportunities for knowledge and information exchange among staff and stakeholders, e.g., virtual communities of practice, periodic in-person forums to share experience and approaches, and devoting a brief time segment at all events and meeting for experience sharing • Document experience in sharable formats (beyond internally oriented quarterly reports) to better inform scale-up efforts by SHRP and others, nationally and internationally Maximize synergies between results • Balance leadership time between the two results (more time on R2 Health) • Review SHRP planning, budgeting, and management structures and practices to ensure they support the overall Program Objective and avoid missed opportunities for achieving both results, e.g., combined FA role which reduces costs and better positions SHRP to identify opportunities for linked programming at the district and school levels • Review arrangements with WEI to reduce bifurcated planning, budgeting, and management between the prime and subrecipient; and together develop approaches that better support R2 Health targets and linkages between reading and health, where appropriate, such as: o Involve R1 Reading and R2 Health stakeholders in each other’s activities, e.g., planning sessions, and monitoring and supervision o Weave health and HIV/AIDS concepts and messages as possible and appropriate to different age groups and themes into SHRP P1-P4 reading materials during planned revisions and updates (e.g., hand washing, bed nets, breastfeeding); o Integrate health and HIV/AIDS information and lesson content into pre-service training at teacher colleges when SHRP initiates in-service activities o Combine monitoring and support supervision tools and trips when there is an R1 Reading and R2 Health program (this is the case in most schools with R2 programs: 87 percent or 1,347 schools have SFI and reading programs) to maximize linkages and reduce time and cost MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX A: EVALUATION STATEMENT OF WORK 47 ANNEXES ANNEX A: EVALUATION STATEMENT OF WORK SHRP Mid-Term Evaluation scope of work: extracts from P&IE contract C.3.3 Performance Measurement, Monitoring, and Evaluation External Performance Evaluation: External performance evaluations shall be conducted at the mid-term and end-point of the Literacy and Health Education Program to assess program implementation and achievement of planned program results. These performance evaluations shall document and assess the strategic approaches undertaken by the program, their implementation, and lessons learned during the course of the five-year program. In-line with the goals of the Mission’s CLA agenda, the performance evaluations shall identify necessary adjustments and opportunities for improvements to programming. The purpose of the performance evaluations will be to provide rich qualitative data on program design, implementation and effectiveness in order to investigate how these relate to the quantitative data on student learning that will be collected through MOES-led measurement systems (C.3.3.1) and an external impact evaluation (C.3.3.4). Performance Evaluation 3.A. Mid-term Performance Evaluation 3.B. Final Performance Evaluation Year 3 Year 5 (g) Performance Evaluation Design and Methodologies The purpose of performance evaluation is to evaluate program implementation and effectiveness at the mid-point and end-point of the program. Performance evaluation shall be designed to document key elements of project design and implementation, analyze strengths and weakness, gather perspectives from a wide range of program stakeholders and beneficiaries, and identify opportunities for continuous improvement in USAID’s programming. This section of the Evaluation Design shall identify the research questions, design and methodologies planned for a mid-term performance evaluation in Year 3 and a final performance evaluation in Year 5. Performance evaluation shall focus on assessing implementation and outcomes of the treatments described above, and the means by which these contribute to the Intermediate Results and Results laid out in Program Results Framework. The design shall include plans to validate and disseminate research findings. C.4.3 Mid-term and Final Performance Evaluations (Deliverables 3.A and 3.B) The Contractor of this P&IE contract shall conduct a mid-term performance evaluation (Deliverable 3.A) and a final performance evaluation (Deliverable 3.B) as approved in the Evaluation Design (Deliverable 1). By the mid-point of Year 2 of the Literacy and Health Education Program for Deliverable 3.A, and by the end of Year 4 for Deliverable 3.B, detailed Final Performance Evaluation Plan shall be submitted to USAID/Uganda for approval. It is expected that these performance evaluations shall be highly participatory, involving USAID, the MOES, stakeholders and beneficiaries on the evaluation team and as active participants in the evaluation process. Final Performance Evaluation Plans shall describe evaluation questions, methodologies and tools to be used for the performance evaluation, the evaluation’s proposed timeframe, implementation and management considerations including the proposed evaluation team and budget. Performance Evaluation Implementation Plans shall outline the anticipated structure of the performance evaluation report, as well as the plan and timeline for the validation and dissemination of research findings to the MOES and other key stakeholders. Once approved, the Implementation Plans shall form the basis for the performance evaluations culminating in Deliverables 3.A and 3.B, which shall each contain an executive summary, a detailed written report presenting evaluation findings and strategic recommendations for USAID and the MOES, and any written and/or visual presentations planned for the dissemination of evaluation findings. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX B: WORKPLAN TIMELINE AND OUTPUTS 48 ANNEX B: WORK PLAN TIMELINE AND OUTPUTS This annex includes the SHRP mid-term evaluation timeline and outputs. SHRP Mid-Term Evaluation Activity and Outputs Timeline Timeline (Period, Location) Activity Outcome Planning and preparation (2-3 days, virtual) Complete by June 9 • Desk review and document analysis • Determine KII and FGD list, and districts/school selection • Develop data collection instruments • Schedule all KIIs, FGDs, and site visits and complete all related logistics • Prepare detailed outline for Mid-Term PE • Team Leader and Literacy/Education Evaluation Specialist arrive in Kampala on Sunday, June 15; and join Ugandan Sr. HIV/AIDS Evaluator Highly developed data collection instrument, PE implementation plan including design, schedule, logistics, etc. Week 1 June 16-21 TPM and Kampala￾based data collection • June 16: Team Planning Meeting (TPM) to finalize data collection instruments, clarify roles and responsibilities, review/confirm PE implementation plan including approach to ongoing analysis and writing Days 2-6: • Meeting and KII with USAID (0.5 day) • KII with US Embassy/PEPFAR Coordinator • KIIs with GoU (1 day) • KIIs with RTI (1.5 days) • KIIs with RTI partners/subcontractors, some by Skype (1 day) • Draft notes and analyze issues and themes (1 day) • Review, triangulate and tabulate data • Analyze qualitative and quantitative information and populate PE outline as possible Final data collection instruments, and PE implementation plan Week 2: June 23-28 District site visits and data collection 2 days each: - Central: Wakiso District - Southwest: Kabale District - North: Lira District • Resident Evaluation Manager (REM) joins PE Team • June 22: All four PE team members travel to Gomba • June 23: KII district headquarters, visit urban school • June 24: Visit rural school • June 25: Team Leader and REM travel to Southwest, Kabale District; Literacy Specialist and HIV/AIDS Evaluator travel to North, Lira District • June 26: KII district headquarters, visit urban school • June 27: visit rural school • June 28: return to Kampala, debrief on district visits, possibly brief USAID on summary of preliminary findings and progress • Throughout: o Review, triangulate and tabulate data o Analyze qualitative and quantitative information and populate PE outline as possible Visit a total of six schools in mix of regions, types of districts, rural/urban, high/medium/low performing MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX B: WORKPLAN TIMELINE AND OUTPUTS 49 SHRP Mid-Term Evaluation Activity and Outputs Timeline Timeline (Period, Location) Activity Outcome Week 3: June 30 – July 4 Kampala • June 30 – July 2: o Continue analysis and development of findings, conclusions, and recommendations; dedicated writing o Develop PE report summary for vetting with RTI and USAID • July 3 o Vet PE report contents with RTI and then USAID • July 4 o Integrate feedback into final report o Agree on plan to finalize remaining report elements Draft Mid-Term PE Report Summary By October 15 From consultant offices in U.S. and Uganda • Finalize Mid-Term PE Report • NORC submits Mid-Term PE to USAID on October 31 Mid-Term PE Report The detailed list of KIIs and FGDs in Kampala by category of interviews and time allocated to each is as follows: • Government – 1 day ► MoES ► NCDC ► MEEPP • USAID – COR for P&IE, COA for SHRP, and Organizational Learning Advisor – ½ day • US Embassy/PEPFAR Coordinator • RTI – 1.5 days ► KII: o COP o M&E Advisor o R1 Manager o R2 Manager o RTI district monitors ► FGD: o R1 team as FGD o R2 team as FGD • RTI/SHARP NGO partners: 1 day ► Center for Social Research/Kampala ► World Education/Boston – Skype from Kampala ► SIL LEAD/Nairobi– Skype from Kampala ► Volunteer Services Organization (VSO Int’l)/Kampala KIIs and FGDs at the district and school level will include: • Districts: DEO, DIS, LLB, PTC, CCT- 2days MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX B: WORKPLAN TIMELINE AND OUTPUTS 50 • Schools: 1 day per school, with an urban school in each district visited the same day as district interviews, and a rural school on the second day ► MEO (municipal education officer) ► Classroom observation ► FGDs with :  School management committees  Secondary School learners ► KIIs with Head Teacher and teachers trained by SHRP (R1 and R2) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 51 ANNEX C: DATA COLLECTION INSTRUMENTS The interview and observation guides used in the SHRP Mid-Term Performance Evaluation are presented in this annex. They include: 8. Key Informant Interview (KII) guide for national SHRP stakeholders (GoU, USAID, RTI, and RTI Partners) 9. KII guide for district SHRP stakeholders (DIS, DEO, MEO, PTC, CTT) 10. KII guide for Head Teachers 11. KII guide for Trained Teachers 12. Classroom Observation Guide (P1/P2 Literacy and English) 13. FGD Guide: Secondary/Adolescent School Learners 14. FGD Guide: School Management Committee SHRP Key Evaluation Questions and Sub-Questions. Each guide draws as appropriate on the following key evaluation and sub-questions: 1. Design: Does the program design and structure adequately support and facilitate achievement of the desired results? a. Y ____ N ___ b. Supporting evidence 2. Implementation: Has the project been implemented according to plan and is it on track to achieve its overall goals and objectives? e. Overall opinion Y____ N ___ f. Supporting evidence (key activities undertaken to date): g. Implementation challenges and how addressed h. Remaining implementation challenges and proposed solutions 3. Results: What are the key results and contributing factors for differences in performance (reading skills acquisition and HIV/AIDS knowledge and skills) in schools receiving the same intervention? Result 1: Improved early grade reading and transition to English a. Observed improvements among P1 and P2 teachers in teaching literacy in local languages and English? i. Y ____ N ____ ii. Supporting evidence b. Observed improvements in literacy among P1-P3 students? i. Y ___ N ____ ii. Supporting evidence c. Reading interventions/strategies with greatest impact on reading skills acquisition? d. Factors contributing to high achievement e. Factors contributing to low achievement Result 2: Improved attitudes, knowledge, and practices f. Observed improvement in health seeking behaviors among students? i. Y __N __ MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 52 ii. Supporting evidence g. Observed improvements in providing HIV/AIDS related information among teachers? i. Y__ N __ ii. Supporting evidence h. Observed improvements in parent and community engagement in HIV/AIDS activities? i. Y ___ N___ ii. Supporting evidence i. Health interventions with greatest impact on improved HIV/AIDS knowledge and practices? j. Factors contributing to high and low achievement? 4. Sustainability: To what extent will the program’s components and subcomponents continue without USAID Assistance? a. Will programs continue without USAID assistance? Y____ N ____ b. Does funding exist to sustain programs? c. Does capacity exist to sustain programs? d. Recommendations for strengthening program sustainability 5. Cost-effectiveness: What are the implications and recommendations for potential scale-up of program interventions? In what ways can the programs be more cost effective? a. Is the program run cost effectively? i. Y ___ N____ ii. Supporting evidence b. Recommended strategies for cost-effectiveness c. Is the program scalable? Y ___ N ___ d. Recommended strategies for scaling up the project 6. Management/Coordination/Lessons Learned: How can program design, management, and execution become more efficient toward achieving program goals? a. Comments on management, coordination, capturing of best practices and lessons learned b. Recommendations for strengthening management, coordination, and capturing of best practices and lessons learned MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 53 C.1. KEY INFORMANT INTERVIEW GUIDE: KEY SHRP NATIONAL STAKEHOLDERS (GOU, USAID, RTI, AND RTI PARTNERS) Mid-Term Performance Evaluation of USAID/Uganda School Health and Reading Program (SHRP) Interviewee: ☐U.S. Government (USAID, PEPFAR Coordinator) ☐Government of Uganda officials ☐ RTI Officials ☐ RTI partners Name: Title: Organization: Address: Telephone: Email: Fax: (Note: complete if no business card provided) Interviewer(s): Date of interview: Location of interview: Introduction: Thank you very much for meeting with us today. The purpose of the evaluation is to assess the current performance of the USAID-funded School Health and Reading Project. We would like to hear about your experience with the project, the challenges and the accomplishments to date, and any recommendations you may have to improve the project. We would like to inform you that participation in the evaluation is completely voluntary and that all information you share with us will be confidential and anonymous. May we kindly have your consent to proceed with the questions? Evaluation questions: Intro Questions 1. How long have you been working with the School Health and Reading Program and what is your role? 2. According to your understanding, what are the goals and objectives of the School Health and Reading Program? a. In relation to reading? ________________________________ b. In relation to health? __________________________________ 3. What types of SHRP project trainings/activities are you involved in and/or have participated in? Program Design 1. Were you involved in the project/proposal design? If yes: a. How were you involved? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 54 b. Does the program/proposal design and structure adequately support and facilitate achievement of the desired results? Implementation 1. What is your role in implementation, e.g., joint planning, developing materials, procurement, trainings, communication dissemination, monitoring etc.)? 2. Is the project on track to achieve its overall goals and objectives? a. If yes, what key activities have been achieved to date? Which ones have not and why? b. What instructional materials and resources have been produced by the program? How have they been developed/distributed and with whom? Are they being used as intended to achieve program results? c. How are program results tracked and monitored? How are you involved? 3. What has gone especially well in your opinion? 4. We’d like to ask a few questions about issues, challenges, or setbacks: a. What issues, challenges, or setbacks has the program faced? b. How have they been addressed? c. What issues are still to be addressed, and what solutions would you propose? Impact 1. What results in reading and health has the project achieved to date? a. What percentage of P1-P3 students is demonstrating improvements in increased literacy? b. Which reading interventions/strategies are having the greatest impact on reading skills acquisition? c. What percentage of students is demonstrating improvement in health seeking behaviors? d. Which health interventions are having the greatest impact on improved HIV/AIDS knowledge and practices? 2. Which districts/regions are demonstrating the greatest achievements in reading/health? b. What are the key factors that contribute to high reading/health achievement in these districts? c. What are the challenges or causes of low reading/health performance in target schools and districts? d. Was the implementation of the intervention (and/or take-up) in the Luganda districts different from the other districts? If yes how and why? (Probe: More teachers being trained in Buganda Region) Sustainability 1. To what extent will the program continue without USAID assistance? a. Does the MoES have funding necessary for sustaining program activities? If not, how does the MoES envision raising the funds? b. Does the MoES have the capacity necessary for sustaining program activities? If not, what could be done to create the capacity? c. Does the MoES and SRHP have an integrated work plan? What activities are being implemented jointly? How is the collaboration? d. Is there a system or plan for continued support to trained teachers from the TIET, district inspectors, Associate Assessors and others? e. Has the cost share component been as expected? Can it be sustained? f. What recommendations do you have for increasing program sustainability? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 55 Cost-effectiveness 1. What strategies have been used to ensure cost-effectiveness? 2. How could the program achieve greater cost effectiveness? Are there costs that can be absorbed by the government, community, school budget, or private sector? 3. What is the most cost-effective approach for producing, printing, and distribution materials? 4. Is the program scalable? What are your recommendations for scale-up? Management/coordination/lessons learned 1. What is your opinion of the overall management and coordination of the program? 2. What recommendations do you have for improving program management and execution to best achieve program goals? 3. Are there systems and mechanisms in place for capturing lessons learned and best practices? Is it effective? What are the lessons learned and best practices that have been identified? 4. Do you have any other comments for strengthening the program? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 56 C.2. DISTRICT-LEVEL STAKEHOLDERS INTERVIEW GUIDE (DIS, DEO, MEO, PTC, CTT) Mid-Term Performance Evaluation of USAID/Uganda School Health and Reading Program (SHRP) District: Name of Key Informant: Title: Organization: Address: Telephone: Email: Fax: (Note: complete if no business card provided) Introduction: Thank you very much for meeting with us today. The purpose of the evaluation is to assess the current performance of the USAID-funded School Health and Reading Project. We would like to hear about your experience with the project, the challenges and the accomplishments to date, and any recommendations you may have to improve the project implementation. We would like to inform you that participation in the evaluation is completely voluntary and that all information you share with us will be confidential and anonymous. May we kindly have your consent to proceed with the questions? Introductory questions: 1. Length of time working with project: ____ 2. According to your understanding, what are the goals of the School Health and Reading program? Program Design 1. Were you involved in the project/proposal design? If yes: a. How were you involved? b. Does the program/proposal design and structure adequately support and facilitate achievement of the desired results? Yes _____ No _______ Implementation 1. Does the MoES/PTC and SRHP have an integrated work plan for implementing project activities? What activities are being implemented jointly? 2. Is the project on track to achieve its overall goals and objectives? Y ____ N ___ 3. What types of SHRP project trainings/activities have you participated in personally and what was your role? 4. What kinds of support do you provide to teachers? a. How many schools do you support? ___ b. How often do you visit each school? ___ c. How many teachers do you typically observe/support during school visits and for which grades/classes? ___ 5. Is there anything you are doing differently during school visits to support teachers to teach reading and writing since the training workshop? If yes, what? 6. How do you monitor or assess teacher performance? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 57 a. Do you have a standard form/checklist that you use for school monitoring visits? b. If yes, was this provided by SHRP? Has it, or will it be integrated into the existing MoES/PTC structures? c. Is there anything you are doing differently to assess teachers’ performance as a result of your participation in the SHRP project? 7. How do you monitor student’s academic performance? a. Is there anything you are doing differently to assess students’ progress in reading as a result of your participation in the SHRP project? Result 1: Reading 1. Have you observed any improvements among P1 and P2 teachers in teaching literacy in the local languages and English? Y___ N____ a. If yes, what? 2. Have you observed improvements in literacy among P1-P2 students? a. Y ___ N ___ b. If yes, what? 3. Which of the SHRP reading interventions/strategies are having the greatest impact on improving teacher and pupil performance? 4. What are the key factors that contribute to high reading achievement in schools that have shown the greatest improvements? 5. What are the challenges or causes of low reading performance in target schools? Result 2: Health 1. Have you observed any improvements in teachers’ knowledge, attitudes and skills to teach/facilitate HIV/AIDs awareness activities? Y ___ N ___ a. If yes, what? 2. Have you observed any improvements in student engagement in HIV/AIDS activities? Y___ N____ a. If yes, what? 3. Have you observed any improvements in parent and community engagement in HIV/AIDS activities? Y _____ N _____ 4. Which of the SHRP health interventions/strategies are having the greatest impact on improving teacher and pupil health practices? 5. What are the key factors that contribute to schools that have shown results? 6. What are the challenges or causes of low HIV/AIDS awareness in target schools? Implementation Challenges 1. What challenges have you faced in carrying out monitoring and supervision under the SHRP project and how were they addressed? 2. What issues are still to be addressed, and what solutions would you propose? Sustainability 1. Will program activities continue without USAID assistance? Y____ N ____ 2. Does funding exist to sustain programs? Y ____ N _____ 3. Are there any associated costs that might affect the sustainability of the project? If yes, what? 4. Can any of these expenses be absorbed by the government, community, school budget, or private sector? If yes, which ones? 5. Does capacity exist to sustain programs? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 58 6. Is there a system or plan in place for continued support to trained teachers after the SHRP project ends? 7. What recommendations do you have for strengthening program sustainability? And for whom – the MoES or USAID? Cost-Effectiveness and Replicability 1. In your opinion, is the program run cost effectively? Y ___ N____ 2. Do you have any recommended strategies for increasing the cost-effectiveness of the project? 3. Is the program scalable? Y ___ N ____ 4. What recommendations would you propose for replicating/scaling up the project? Management/Coordination/Lessons Learned 1. Do you have comments/feedback on the management and coordination of the project? 2. What recommendations do you have to improve the coordination, management or implementation process of the project? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 59 C.3. KEY INFORMANT INTERVIEW GUIDE FOR HEAD TEACHERS Mid-Term Performance Evaluation of USAID/Uganda School Health and Reading Program (SHRP) Date: District: School Name: Head Teacher Name: Address: Telephone: Email: Fax: (Note: complete if no business card provided) Introduction: Thank you very much for meeting with us. We are here today to hear your views about the USAID-funded School Health and Reading Project. We are part of an assessment team to learn about the accomplishments to date and any recommendations you may have to improve children’s reading and health status. We would like to inform you that participation in the evaluation is completely voluntary and that all information you share with us will be confidential and anonymous. May we kindly have your consent to proceed with the questions? Introductory questions: Length of time as HT: ____ Key evaluation questions: 1. According to your understanding, what are the goals of the School Health and Reading program? 2. What types of support (training, materials, etc.) has your school received from the SHRP project? 3. Are there any other donors/programs providing support to your school? 4. What types of SHRP project trainings/activities have you participated in? What did you learn? 5. What kinds of support do you provide to teachers to help them teach reading in the local language and English? a. How often do you visit/observe their classes? b. What do you do when you observe the class? 6. Is there anything you are doing differently during classroom observations since the training workshop? If yes, what? 7. Have you faced any challenges in carrying out your expected role in the project? Result 1: Reading 1. Have you observed any improvements among P1 and P2 teachers’ ability to teach literacy in the local languages and English? Y___ N____ a. If yes, what? b. Which of the SHRP reading methodologies, materials or support has contributed to this improvement? 2. Have you observed any improvements in children’s reading ability among P1-P2 students? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 60 a. Y ___ N ___ b. If yes, what? 3. Which of the SHRP reading interventions/strategies are having the greatest impact on improving teacher and pupil performance? 4. What are the key factors that contribute to high reading achievement in students that have shown the greatest improvements? 5. What are the challenges or causes of low reading performance? 6. Has there been any improvement in the support received from the community or SMC for improving reading performance? Result 2: Health 1. Have you observed any improvements in teachers’ knowledge, attitudes and skills to teach/facilitate HIV/AIDs awareness activities? Y ___ N ___ a. If yes, what? 2. Have you observed any improvements in student engagement in HIV/AIDS activities? Y___ N____ a. If yes, what? 3. Have you observed any improvements in parent and community engagement in HIV/AIDS activities? Y _____ N _____ 4. Which of the SHRP health interventions/strategies are having the greatest impact on improving teacher and pupil health practices? 5. What are the key factors that contribute to schools that have shown results? 6. What are the challenges or causes of low HIV/AIDS awareness in target schools? Sustainability 1. Is the support provided by the SHRP project sufficient to achieve the results of the project? If not, why? 2. Will the changes you have seen in your school as a result of the SHRP project be sustained after the project ends? Y____ N_____ a. Why or why not? 3. Are there any associated costs that might affect the sustainability of the project? If yes, what? Can any of these expenses be absorbed by the government, community, school budget, or private sector? If yes, which ones? 4. What recommendations do you have to improve the project performance or sustainability? And for whom – the MoES or USAID? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 61 C.4. TEACHER INTERVIEW GUIDE – PRIMARY SCHOOL Mid-Term Performance Evaluation of USAID/Uganda School Health and Reading Program (SHRP) Date: District: School Name: Head Teacher Name: Address: Telephone: Email: Fax: (Note: complete if no business card provided) Introduction: Thank you very much for meeting with us. We are here today to hear your views about the USAID-funded School Health and Reading Project. We are part of an assessment team to learn about the accomplishments to date and any recommendations you may have to improve children’s reading and health status. We would like to inform you that participation in the evaluation is completely voluntary and that all information you share with us will be confidential and anonymous. May we kindly have your consent to proceed with the questions? Introductory questions: Total number of teachers interviewed: _____ Grade level of teacher interviewed: P1 _____ P2 ______ Length of time as teacher: ____ Length of time as teacher in this school: ____ Participated in SHRP training: Y ____ N ____ Key evaluation questions: 1. According to your understanding, what are the goals of the School Health and Reading program? 2. Which project trainings or activities have you participated in? Did you attend the MOES/USAID/RTI School Health and Reading Program training in: a. May 2013 b. September 2013 c. January/February 2014 d. May 2014 e. Other ___________________ 3. How many teachers at your school have been trained in reading methodologies by the SHRP project? _______ 4. What did you learn during the early grade reading training workshops? a. Is there anything you are doing differently to plan for lessons since the training? b. Which of the SHRP teaching strategies did you find most effective to improve your students’ reading skills? 5. Was the training sufficient? Y _______ N _______ MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 62 a. Was there anything you learned that you were not able to use in your classroom to improve reading? If yes, what? b. What are some possible solutions to overcome this? 6. How have the teachers at your school been supported to teach reading since the training? a. How does the HT, CCT, or District Inspectors support you? 7. How often does the HT, CCT, or District Inspectors observe your reading classes? Do they give you constructive feedback after the observation? a. HT Visits/term ______ Feedback Y ___ N ___ b. CCT Visits/term ______ Feedback Y ___ N ___ c. District Inspector Visits/term ______ Feedback Y ___ N ___ d. Area Inspector Visits/term _____ Feedback Y ___ N ____ 8. What materials do you use to teach reading? Which of these were provided by the SHRP project? (Request to see the materials) a. When did you receive the materials? b. How many copies of the TG and pupil textbook did you receive (1textbook per learner)? c. Where are the materials stored? 9. Has there been any improvement in your students’ reading skills as a result of the SHRP materials or trainings provided? a. If yes, what has improved? 10. How do you monitor students reading performance? a. Uses Continuous Monitoring Assessment form Y ____ N ____ 11. What percentage of your P1/P2 students have shown improvement in reading abilities? 12. What factors have contributed to improved reading performance? 13. What are the challenges or factors cause low reading performance? 14. Has there been any improvement in the support received from the community, parents or SMC to help children learn how to read? 15. Are the materials, teacher guides, training and other support provided by the SHRP project sufficient? If not, what other support is needed to help you successfully teach reading in the classroom? 16. Are the materials user-friendly? If not, how could they be improved? 17. Will the changes you have seen in your school as a result of the SHRP project be sustained after the project ends? Why or why not? 18. How often are teachers transferred to other schools? What are the reasons? Do you have any suggestions for how to ensure that teachers retain the knowledge and skills that they’ve learned through the project? 19. Do you have any recommendations or comments for how SHRP or the MOEST could further support you to teach reading? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 63 C.5. CLASSROOM OBSERVATION GUIDE (P1/P2 LITERACY AND ENGLISH) USAID/Uganda MOES School Health and Reading Program -Lesson Observation Teacher Identification Information (complete this before the lesson begins with information from the head teacher and/or teacher) 1 Grade level of Lesson to be observed: a.  P1 : b.  P2 : c.  P3 : d.  P4 2 How many streams of the grade level being observed does the school have? ________________ If more than one stream, which does the teacher teach?: a_____ b_____ c_____other:_______________________ 3 Lesson/learning area to be observed: a.  Reading (literacy 1) b.  writing (literacy 2) c.  English d. oral literature c.  news 4 Number of learners registered in Class (from class register): Total _____ Boys_________ Girls_______ Teacher has class register  Yes  No Register is up to date  Yes  No 5 Language school uses for instruction in P1-P3: a.  Ateso b.  Luganda c.  Runyankore/Rukiga d.  Leblango e.  English f.  Other__________(specify) g.  Leb Acoli h.  Lugbarati i.  Runyoro/Rutooro j.  Lumasaaba Pre-observation Teacher attended USAID/SHRP/RTI early grade reading training in: January 2013: Yes  No May 2013: Yes  No September 2013: Yes  No January 2014: :  Yes  No May 2014: Yes  No If answer is no, If they are the appropriate class teacher, why didn’t they attend training? 6 Does the Teacher have? Thematic curriculum teachers’ guide  Yes  No : SHRP teachers’ guide  Yes  No : MOES Primary school curriculum  Yes  No : Teachers’ resource book  Yes  No: 7 Teacher has class timetable?  Yes  No Teacher is following the timetable  Yes  No Ask to see the teacher’s lesson plan and scheme of work 8 Lesson follows the thematic curriculum (includes theme/sub-theme corresponding to school term).  Yes  No District__________________________________________CC_____________________________ ________ School Name:___________________________________________________________________________ Teacher Name:___________________________________________________Male/Female___________ Name and title of Monitor:________________________________________________________________ Date of Monitoring Visit (and day of the week):________________________________________________ MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 64 Notes: 9 Lesson plan includes aspects of appropriate reading methodology and follows the teaching reading guide  Yes  No Notes: 10 (Page through the teacher’s planning book). Does it appear that teacher consistently develops lesson plans? (Plans have dates and are in a logical sequence). Notes: Classroom Environment 11 Fewer than half of the learners have seats  OR More than half have seats  OR ALL Have Seats  OR NO Seats  Notes: 12 Learners’ work is displayed in the room/space or on the walls. a.  Yes b.  No c.  Not applicable (or no walls) Notes: 13 Are there reading displays in class? a.  Yes b.  No NOTE: Remember to record the time when the lesson started and when it ended in the section on top of this page Lesson Observation: It may be possible to observe more than one lesson. Class Start-up (tick “observed” only once as the action first occurs. At end of observation, tick actions that were not observed) Lesson 1 _______________starts: ____:___ lesson ends: ____:___ length of lesson _____________minutes Lesson 2 ______________ starts: ____:___ lesson ends: ____:__ length of lesson _____________minutes Observe d Not observed Instructional Content: 13 Followed the steps in the SHRP teachers’ guide Notes: Teacher guides learners to…. (Tick what is relevant for Lit 1 and 2, Oral literature, News and English. 14 Read words from printed material or book (individual reading of printed material) 15 Read words/texts from the chalkboard 16 Make correct letter sounds 17 Differentiate between letter name and the correct letter sound 18 See words as made up of syllables (writes words by syllable on board e.g. “ki-tten”). 19 “beat the word” – clap/beat/tap/stamp the syllables of words 20 Blend letter sounds to make words 21 Write a letter pattern in the air 22 Hold the pen/pencil correctly 23 Recite a traditional text from memory 24 Answer questions related to the oral traditional text/story 25 Write/tell their own news stories 26 Use sentence structure Learner assessment 27 Teacher assesses some learners during class (checks exercise books, gets them to read/write individually) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 65 Notes (Specify the type of assessment(s) observed); 28 Is there evidence that the teacher keeps regular records to track learners’ performance in reading/writing?  Yes  No If “yes” please note what method is used to track learner performance in reading /writing Teaching Learning Material Use: Teacher uses… (note how used if relevant) 29 Blackboard/Chalkboard 30 Textbook 31 Supplementary reading materials 32 Work sheets 33 Poster/wall charts (with letters, words, pictures) 34 Flash cards 35 Slates 36 Learners books 37 Manipulatives (e.g. real objects, bottle caps, clay, sand, cut out words, etc.) 38 Girls and boys were given equal chances to answer questions  YES  NO 39 Teacher taught lesson in local language (not applicable for English lesson): a.All the time b.  some of time c. Not at all Please provide any other observations about the lesson or classroom. If useful, you can even draw what the teacher has written on the board. The following questions are asked/answered at the end of the lesson. 1 Number of learners attending class: total______ boys ____ girls______ [Ask boys to stand, count, then girls] 2 (Comparing learners registered and attending) Number of learners absent today: total______ boys ____ girls____ For Questions 3-6 ask children to hold up book, exercise book/paper/slate, and pencil for the lesson. Count. 3 How many learners have reading or printed material?  All  Half or more than half Less than half  None 4 The language of the reading or printed material is local language being used by teacher English other language 5 How many learners have a pen and exercise book?  All  Half or more than half Less than half  None 6 Are there other books accessible for children to read in the class? a.  none b.  less than 20 books c. more than 20 For questions 8-9 randomly select 2-3 exercise books (from learners or a stack if they are all in one place). 7 What is the date of the last exercise? Are there regular exercise? Notes: MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 66 8 Has the teacher marked them in the last week? Notes: 9 Teacher taught the lesson as indicated in the lesson plan a. Yes b. Partly c. Not at all d. No lesson plan Classroom Observation Protocol 1. Ask teacher to show which lesson in TG and pupil book they are teaching prior to start of class (should be Term 2 Week 7 Day 1 – Mon or Day 4 - Thurs) 2. Ask if they have the thematic curriculum and to show which section they are teaching 3. Observe class 4. Complete checklist 5. Assess pupils for books and writing utensils, exercise books and marked exercise books towards end of class 6. Ask teacher follow-up questions during writing lesson or after class (check continuous assessment form, support supervision book, Reflection book) 7. Ask where they store the books 8. Ask whether pupils take books home What to look for? • Continuous Assessment Form • Teacher Guide • Pupil Book • Support Supervision Book • Reflection Book • Stationary Kits (rulers, scissors, pencils, etc.) 6 Ts of Effective Language Instruction: 1. Teacher – Is the teacher following the lesson in the teacher guide? 2. Tongue – Is the teacher teaching in the local language? 3. Text – Are pupils reading from printed material? 4. Time – Are they using the entire time for Literacy 1 and Literacy 2 (30 min each lesson) 5. Test – Are they assessing students using the Continuous Assessment Form? 6. Teaching methodology - Are they using appropriate modeling and teaching methods? Class P1 Methodology Sample Literacy Lesson P1 Literacy 1: Days 1 & 3 (Mondays and Wednesdays) (TG Luganda, pgs. 34 & 35) 1. Sing a song and distribute materials for the lesson (Getting Ready Song) 2. Introduce the sound of the letter of the day by using a rhythm (Sound and Rhythm) 3. Write the new letter on the board and pupils practice sounding out the letter on the board and in their books (My Name, My Sound, New Letter) 4. Ask learners to point to a picture, word or letter and describe name, name or read it to the class (Find Me) 5. Ask learners to tell their neighbor what’s happening in the pictures 6. Guide children to beat the word while clapping on each syllable; teacher assesses students’ ability to segment words MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 67 7. Quick Read – Write word on blackboard and read the word to the class (I do), Ask learners to put their finger on the word and read it with you (We do), and Ask learners to read the word by themselves and with their neighbor (You do) 8. Do the same as quick read, but with a sentence. 9. Teacher should assess at least 5 students continuously throughout the lesson 10. Go to writing lesson (30 minutes) (when they go into handwriting function – they’ve passed into Literacy 2) (See pgs. 48-49 for Literacy II: Writing lesson) P1 Literacy 1: Days 2 & 4 (Tuesdays and Thursdays) (TG Luganda, pgs. 38 & 39) 1. Getting Ready Song 2. Before Reading: a. Word Exploration - tell learners the word they will be exploring; ask them what they know about the word, elaborate on their knowledge, etc. b. Read the Title of the Story (recall first part of story read on Day 2 and predict what’s next in the story) c. What do I want to Know? 3. Read the Story 4. After Reading: a. Was my prediction correct? b. Ask one question in the text c. Ask question based on story reflecting on personal experience 5. Go to writing lesson (30 minutes) (when they go into handwriting function – they’ve passed into Literacy 2) (See pg. 51 for Day 2 and 4 Writing lesson) Class P2 Methodology P2 Literacy 1: Days 1 and 3 (Mondays and Wednesdays) (Methods: Pgs. 28 – 33) (Term 2 Week 7 Lesson pgs. 168 – 169) 1. Getting Ready Song 2. Discuss the theme and sub-theme 3. Before Reading: Predicting a. Ask thematic guiding questions prior to reading 4. Read story a.Students read silently for first few minutes b.Teacher reads aloud while students listen and track words with finger 5. After reading: Ask about predictions 6. Read story phrase by phrase while class repeats in chorus 7. Students read again in pairs (Teacher moves around classroom and assesses individual learners’ reading fluency using the CAM form). 8. Selected learners read to the entire class 9. After reading: ask one question from text found in the TG and assess learner comprehension. Mark in CAM form 10. Ask learners to come up with their own questions in their minds. 11. Go to writing lesson (30 minutes) writing letters or story P2 Literacy 1: Days 2 and 4 (Mondays and Wednesdays) (Methods: Pgs. 28 – 33) (Term 2 Week 7 Lesson pgs. 168 – 169) 1. Getting Ready Song MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 68 2. Our vocabulary words (Teacher reads; learners read in pairs) 3. Word Exploration (3 steps) a. Read the Word b. Use the word in a meaningful sentence c. Find another word that means the same or the opposite 4. Word Structure a. Takes word and breaks down into meaningful segments MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 69 C.6. FOCUS GROUP DISCUSSION GUIDE: SECONDARY/ADOLESCENT SCHOOL LEARNERS Mid-Term Performance Evaluation of USAID/Uganda School Health and Reading Program Date of focus group: Venue of focus group: District: Municipality: School Facilitator: Documenter: Translator: Total number of participants:____ Female: ____ Male: ____ Purpose: Thank you very much for meeting with us. We are here today to hear your views about the USAID-funded School Health and Reading Project. We are part of an assessment team to learn about the accomplishments to date and any recommendations you may have to improve children’s reading and health status. We would like to inform you that participation in the evaluation is completely voluntary and that all information you share with us will be confidential and anonymous. Also, please note that we are recording the session as much of it may be in a local language that our whole team does not know. May we kindly have your consent to proceed with the questions? If so, please sign the consent form. Introductions: Ask each participant to briefly introduce his/herself, including age and grade: Questions: 1. Tell us what you know about the School Health and Reading Program. Implementation 1. In which SHRP program activities have you participated? Take a count of hands. Probe: a) HIV and AIDS Clubs b) School Family initiative c) HIV counseling and guidance d) Any other activities? 2. What did you do during the activity? (Go around the room so each person speaks.) 3. What activities did you like most? Why? (Continue to go around the room if everyone is not speaking up on this or following question.) 4. What activities did you not like? Why? Results 1. Is there anything that you are doing differently as a result of participating in one of these the activities? Recommendations/Comments 1. Do you have any thoughts on how to improve any of the SHRP activities you have been involved in? 2. What else could SHRP could do to promote healthy behaviors among learners and teachers? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 70 3. Is there anything you would like to mention about the SHRP program? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 71 C.7. FOCUS GROUP DISCUSSION GUIDE: SCHOOL MANAGEMENT COMMITTEE Mid-Term Performance Evaluation of USAID/Uganda School Health and Reading Program Date of focus group: Venue of focus group: District: Municipality: Facilitator: Documenter: Translator: Total number of participants:____ Female: ____ Male: ____ Time start: ______Time end: ______Duration: ______ Introduction to Evaluation: Thank you very much for meeting with us. We are here today to hear your views about the USAID-funded School Health and Reading Project. We are part of an assessment team to learn about the accomplishments to date and any recommendations you may have to improve children’s reading and health status. We would like to inform you that participation in the evaluation is completely voluntary and that all information you share with us will be confidential and anonymous. Also, please note that we are recording the session as much of it may be in a local language that our whole team does not know. May we kindly have your consent to proceed with the questions? Personal Introductions Ask each participant to briefly introduce his/herself: • Role on School Management Committee: ___________________________ • Length of Time on SMC: ______________ Ask Chair: • How many members on the SMC: _____ Total _____ M ____ F • Total number of SMC members present today: _____ Total _____ M ____ F Questions 1. Are you familiar with the SHRP program? Y _____ N _____ (count of hands) 2. According to your understanding, what are the goals of the School Health and Reading program? Implementation 1. What types of SHRP project trainings/activities have you participated in and what was your role? (Go around the room.) 2. What is the role of the SMC? a. How often does the SMC meet? _______ b. What types of issues do you discuss at the meetings? Probe: - Does the SMC discuss pupils’ academic performance at your meetings? If yes, how often? - Does the SMC discuss health-related issues affecting learners and teachers? - Are there any academic or health-related issues your SMC has identified at the school within the past year? 3. How has SHRP supported you to perform your role? Was the support sufficient or not and why? Results 1. Have you observed any recent improvements in students’ ability to read in grades P1-P3? If yes, what has changed and why? (Probing questions below) a. Improvement in reading ability? Y _____ N ___ MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX C: DATA COLLECTION INSTRUMENTS 72 b. Has there been any change in availability of learner textbooks or reading materials? c. Has there been any change in HW assignments given by the teacher? Y____ N ____ d. Has there been any change in students completing HW assignments? Y ____ N _____ e. What other new reading-related activities have you observed that is different from before the SHRP program? f. Other __________________________ 2. Have you observed any recent improvements in students’ health/HIV status? If yes, what has changed and why? (Probing questions below) a. Increased knowledge of HIV prevention: Y ___ N___ b. Reduction of stigma against HIV-infected/affected students Y ___ N____ c. Other activities they have observed: __________________________________ 3. Do you think that the improvements in early grade reading/literacy are a direct result of the SHRP interventions? If yes, which interventions have had the greatest impact? 4. Do you think that the improvements in children and youth’s health status have improved due to the project’s interventions? If yes, which ones have had the greatest impact? Implementation Challenges 1. Have there been any challenges with the support received from the SHRP project? a. Any outstanding issues related to early grade reading that are not being addressed? b. Any health-related issues not being addressed? Recommendations 1. Do you have any recommendations for how the School Reading and Health project could be improved? MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX D: PROGRAM BACKGROUND 73 ANNEX D: PROGRAM BACKGROUND Targets. Under R1 Reading, the program aims to train approximately 12,000 teachers in early grade reading and provide effective reading instruction to almost 1,000,000 learners in primary school levels 1-4 (P1-P4). In collaboration with the NCDC, SHRP will develop instructional materials in 12 local languages and English and produce 2 million reading primers to 3,300 schools in 30 districts. With funding from the Global Partnership for Education (GPE) to the MoES, the program will provide technical assistance to the MoES to scale up the Early Grade Reading Model to an additional 1.4 million children in 2,644 primary schools and 56 districts. Thus, by the end of the project, SHRP is expected to reach a total of 1 million learners in almost 3,300 schools and 30 districts. Under R2 Health , 8,000 teachers will be trained, and 500,000 learners across nearly 1,700 primary schools, secondary schools, and business, technical, vocational, education, and training (BTVET) institutions will be reached through learning activities designed to help them understand and practice healthy behaviors aimed at preventing HIV/AIDS. In total, the program will work in 4,148 primary schools – 810 schools will have both R1 Reading and R2 Health activities. The program will cover a total of 12 languages in 30 districts for R1 Reading. R2 Health activities will take place in 17 districts. Strategy and program initiatives. Implementation of the Early Grade Reading model supports and is aligned with the GoU Thematic Curriculum, which has three elements, literacy, numeracy, and life skills. RTI’s early grade reading method offers an approach for the literacy element while R2 Health activities support goals for life skills. R1 Reading supports the MoES mother tongue language of instruction policy, based on the premise that children learn to read fastest in the language they speak best, and that literacy skills developed in one language help a learner gain literacy skills in a second language. It employs a number of reading tactics, such as phonological and phonemic awareness (e.g. clapping/beating the words), phonics/decoding, vocabulary, and comprehension, as well as collaborative learning and continuous assessment. Result 2 supports implementation of the ESS HIV Prevention Strategic Plan 2011 – 2015 through technical support and systems strengthening at the national and district levels, and support at the school level for an enhanced PIACSY program (building on the predecessor USAID program). RTI works through local and national structures and systems, building capacity, and deepening ownership for long-term sustainability. Specific R1 Health interventions by IR • National Policies. (1) Advance Uganda’s National Literacy Strategy with a focus on harmonizing reading assessment (including the role of EGRA) and developing national reading benchmarks. Key stakeholders in this effort include Uganda National Examination Board (UNEB) and Directorate of Education Standards (DES). (2) Support LLB formation and development in the three language clusters (see text box), working closely with key stakeholder NCDC and focusing on orienting LLBs in their roles and responsibilities and building their capacity for materials development. (3) Support the Special Needs Education (SNE) Unit to develop SNE materials, provide SNE teaching training, and develop an SNE assessment model • Early Grade Reading Materials. Develop P1-P3 reading and writing instructional materials for both students and teachers in target local languages and English, and P4 materials in English only, with all materials following the thematic curriculum and working closely with NCDC and LLBs. For students, materials include pupil primers and supplemental reading material; and for teachers, teacher guides and rubrics for continuously assessing student reading competencies. SHRP’s Language Cluster Strategy RTI uses a cluster strategy to sequence implementation based on the 12 target languages’ instructional readiness, existence of instructional materials and NCDC and MoES guidelines, and size of the population speaking the language. • Cluster 1: Ateso, Leblando, Luganda, and Runyankore/Rukiga in 10 districts where the languages are well established and rapid start-up is possible • Cluster 2: Lugbarati, Acholi, Lumasaba, and Runyoro/Rutoro in 11 districts where the languages have some orthography and LLB development, and some existing materials • Cluster 3: Lugwere, Nkarimojong, Lukhonzo, Lusoga in 9 districts where significant work remains to develop orthographies and reading materials MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX D: PROGRAM BACKGROUND 74 Materials are directly distributed to program primary schools and, importantly, marked as pilots to allow an iterative process of input and upgrading • Teacher Training. (1) To improve classroom teaching of reading, RTI uses a three-level cascade training model in which Master Trainers are coached to run a five-day training-of-trainers (TOT) programs for district officials (e.g., the CCT, DIS, and DEO), and then CCTs deliver initial and refresher training to teachers. Training uses the Teacher Development Management System and includes reading pedagogy and use of the materials. (2) Training in monitoring and support supervision (providing constructive feedback). (3) Training in leadership and management to improve school and classroom management • Reading Advocacy. Developing support for the program includes (1) developing materials for schools with reading and health messages, e.g., calendars and posters; (2) mobilizing parents and communities in program districts and schools to generate support for reading and health (includes developing a community mobilization manual, organizing reading competitions, and direct outreach); (3) participating in national events, e.g., National Book Week, to showcase the program, raise awareness on reading and health, and network; (4) producing a quarterly newsletter for stakeholders with program information; (5) using technology, e.g., SMS messaging to teachers; and (6) engaging LLBs as local champions for mother tongue language instruction • Data-based decision-making. EGRA is the centerpiece for encouraging the use of data and research for programming and policy, specifically (1) collecting EGRA data in program areas; (2) harmonizing reading assessment efforts with UNEB, NAPE, and DES; (3) incorporating continuous assessment into teacher training; (4) assessing MoES capacity to scale up program reading interventions via a scale-up capacity assessment report and action plan Specific R2 Health activities by IR • Improved MoES HIV Prevention Response. To develop and HIV/AIDS education assessment, reporting, and decision-making system, activities include: (1) technical support to the Education Planning and Policy Analysis Unit to integrate approved HIV indicators into the MoES’ existing EMIS; (2) holding a national stakeholders meeting to review HIV/AIDS education progress, outputs of the assessment and reporting system, and KAP data; and (3) support the MoES HIV Technical Working Group (TWG), e.g., meeting support, help to finalize and/or follow up on actions associated with studies, disseminate KAP data, integrate HIV indicators into the EMIS, and develop a sustainability strategy for ESS HIV response MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX D: PROGRAM BACKGROUND 75 • HIV/AIDS and Health Education at School-Level. To reach 500,000 learners with AB and health messages, activities are undertaken in five areas: (1) teacher training at primary and post-primary schools on HIV/AIDS and health topics (for which existing PIASCY materials were enhanced) and in guidance and counseling; (2) develop guidelines for the rollout of the PIASCY package of minimum school interventions (see box); (3) support teachers to deliver the enhanced PIASCY program via monitoring and support supervision, referral information, and message support; (4) strengthening SFI (see box) to reach learners on a co-curricular basis; and (5) increasing and institutionalizing parental and community involvement in school-based HIV/AIDS education for sustainability, specifically orienting SMCs on school R2 Health interventions • Data-based decision-making. The main activity for improving data-based HIV-related education programming and policies is conducting and supporting a baseline, midline, and endline KAP survey SHRP sub-awardees. In its original proposal, RTI defined the following roles for its implementing partners to realize its vision and strategy for bringing “successful health education and reading instruction to all Ugandan children.” • RTI, as prime, is responsible for program management and Monitoring and Evaluation (M&E), and serves as the technical lead on reading assessment, teacher training, policy dialogue, and transitioning to English as language of instruction • SIL Language and Education Development (SIL LEAD) is responsible for local language analysis, orthography, and developing teaching and learning materials for local languages and English • WEI leads all aspects of the HIV/AIDS and health education component • Centre for Social Research (CSR) is responsible for technical and logistical support for EGRA data collection • Voluntary Service Overseas (VSO) supports reading via experienced voluntary labor and teacher trainers • African Development Corps provides supports supplementary book processing, training schools on library management and distribution of supplementary books • International Book Bank mobilizes supplementary books for school libraries to raise program cost share • Perkins International is implementing the Special Needs activities • Peace Corps: while originally planned for CCT support at core PTCs, Parent Teacher Association (PTA) advocacy, youth volunteer coordination, and English language support; now RTI coordinates relevant activities where appropriate with the Peace Corps • Mango Tree: ultimately not involved but originally was advise on the reading program, local communication strategies, strengthening LLBs, and improving publishing standards Modifications to the CA. In the only substantive modification to the CA (Modification #2 dated October 28, 2013), the following significant changes were made. R1 Reading. Two of the five-year program results were modified as follows: • Program Result #2 was modified to include “National Scale Up” as part of the 3.5 million children demonstrating improved reading over baseline, so that both what SHRP achieves directly Minimum package of school HIV interventions:  Integrate HIV/AIDS education into classroom subject lessons  Integrate HIV/AIDS issues into co-curricular activities  Provide referrals for HIV and AIDS services  Have participatory and active HIV/AIDS talking compound (HIV/AIDS information on school grounds)  Provide counseling and guidance of pupils by trained HIV/AIDS counselors  Carry out the School Family Initiative (SFI): “SFI uses small learner groups or “School Families” under the care of a teacher who plays a parental role for the group. The purpose…is to provide a mechanism for in-depth information sharing, guidance, care and support to learners in the area of HIV and AIDS, general health and life skills to small groups of 20-25 pupils…happens on a weekly basis during the school term.” (SHRP Year 2 Work Plan, p. 30) MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX D: PROGRAM BACKGROUND 76 and what the GoU achieves through scale up of the SHRP methodology contribute to the 3.5 million goal. In addition to 1.4 million learners directly targeted through the SHRP program, the current plan is for the MoES to scale up the program in the same 12 local languages to an additional 1.4 million children, starting in 2015. With this scale-up, it is expected that by the end of SHRP and MoES GPE-funded efforts, a total of 2.8 million learners will benefit directly and indirectly from the program. This is summarized in Table 1 below. SHRP numbers are based on their PMP dated January 28, 2014, and confirmed on August 29, 2014, when SHRP provided updated GoU numbers • Program Result #5, which originally measured P6 exam results, has been revised from the original target to capture the grade levels and pupils that will be reached through direct support during the life of the program. It now reads, “at least 60% of children reached through direct program support will demonstrate improved reading skills for those grade levels”. This allows the SHRP program to measure its direct impact on program beneficiaries over the five years, which will include pupils in P1 – P4, who will not yet have reached P6. R2 Health. Due to a major shift in PEPFAR’s strategic priorities that came about after SHRP was awarded, now emphasizing direct interventions such as treatment and circumcision over abstinence and prevention, PEPFAR funds for SHRP were reduced from $15 million over five years to $9.8 million during SHRP’s first year of implementation. Additional reductions are anticipated. Because PEPFAR judged SHRP’s targets to be too low, higher targets were renegotiated in January 2014. As a result, SHRP targets changed from training 6,800 teachers in the minimum PIASCY package and reaching 85,000 learners in 800 schools over five years, to reaching 250,000 learners in 2012 and 500,000 annually from 2013 to 2017. SHRP is on track to meet the new higher targets in 2014 through its SFI in 1,651 schools (94 post-primary institutions and 1,557 primary schools). In addition, the five IRs under R2 Health were streamlined to three, with efforts focused more on school-level activities and impact and less on national level activities and systems strengthening. Two IRs (2.2 and 2.4) were dropped. All but three activities were shifted to the other three IRs. The three activities that were dropped are: developing a national HIV/AIDS coordination framework, integrating HIV/AIDS education into education sector work plans at district level, and integrating SNE into program interventions. Some central activities of IR 2.2, such as support to the HIV TWG, were retained. The focus of MoES staff participation shifted to delivery of school-level interventions in teacher training, data collection, monitoring and support supervision, and development of the HIV Education Assessment and Reporting System. The IR changes are presented in Exhibit 3 below. Table 1: Learners Reached with Reading Interventions Number of Learners Reached Out-Years Learners Reached Total Learners Reached SHRP (direct program) 952,765 493,875 1,446,640 MoES through GPE (indirect) 847,410 567,450 1,414,860 Total 1,800,175 1,061,325 2,861,500 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX D: PROGRAM BACKGROUND 77 Exhibit 3: R2 Health Original and Modified IRs IR Original Modified Modified Activities 2.1 Improved planning of education and sports sector HIV prevention response Improved planning of MoES HIV prevention response No change: • Develop HIV/AIDS Indicator technical guide • Develop training modules for data management assessment and reporting • MoES training of DEOs and data officers on HIV/AIDS education data gathering 2.2 Improved coordination between MoES and other actors in HIV/AIDS education. School level impact of HIV/AIDS and health education improved Dropped: • Development of national HIV/AIDS coordination framework Shifted from IR 2.2.: • Development of HIV/AIDS communication strategy for the education sector in coordination and consultation with USAID’s health communication partner • PIASCY minimum package developed • PIASCY training at primary and post primary levels • Orientation of CBT and SMCs to support HIV interventions at school level • Strengthen HIV/AIDS counselling, care, and support at school level Shifted from IR 2.4 • Orientation of SMCs ,PTAs, and head teachers on resource mobilization to support HIVAIDS intervention at school level No Change: • Mapping of district-based services and revision of HIV services directory • Orientation of school-based stakeholders on referrals 2.3 Improved school level impact of HIIV/AIDS education. HIV/AIDS education programs and policies informed by data and research Shifted: • All activities under this IR were moved to IR 2.2 Activities shifted from IR 2.5 • HIV/AIDs KAP survey conducted • Data analysis and report workshops, student-level data reporting and dissemination 2.4 Improved integration of HIV/AIDS education into MoES Investment Plan. IR eliminated - 2 activities dropped and 1 shifted to IR 2.2 Dropped: • Support the integration of HIV/AIDS education into education sector work plans at district level • Support for integration of SNE into program interventions 2.5 HIV/AIDS education programs and policies informed by data and research IR eliminated; all activities moved to IR 2.3 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX D: PROGRAM BACKGROUND 78 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 79 ANNEX E: MATRIX SUMMARY OF MONTHLY PERFORMANCE FEEDBACK MEMOS Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested KAP Assessors Training and Fieldwork: May – June 2013 More time given to the debrief for supervisors held after the pre-test and/or there could be more supervisory training added to the main training. The training focused more on roles and responsibilities of enumerators and less on supervisory responsibilities Program has taken note and more supervisory training will be added to the main training. KAPS follow up data will be collected in October 2015; preparation (assessor training, logistics planning, instrument revision) will take place in the preceding month (September, 2015). These issues will be addressed at this time. Some questions were ambiguous and unclear to learners and teachers. For example: • Q.4: Are you a member of any club where HIV and AIDS are discussed? – Does this question refer to membership in clubs at school, outside, or both? • Q.7: Are you a boarding or day learner? – Response choices were Yes or No. • Q.8: If a mother has HIV can she pass it on to her baby? – Some respondents asked if the mother is enrolled in PMTCT or not, as the risk varies. These errors are noted and will be fixed, we are liaising with Evelyn and Stella during tool revision. There is also a need to pre-test the revised tool with different groups of learners. While data collectors did a good job overall on explaining questions during learner interviews, some did not pause to give learners time to think through their answers. Program has taken note of that and will work on it for future trainings. Regarding logistics, some vehicles were old and did not fare well on the roads, and some drivers (e.g., in the pre￾test) drove too fast. RTI may wish to consider other companies. Issue noted, there is need to search for different vehicle suppliers. Also design a way to directly pay drivers since the data collectors were also disorganised by unpaid drivers. Parental concerns and misperceptions (e.g., that their children would be circumcised or tested for HIV against their will) can be better addressed, perhaps with more advance communication or possibly a communication that indicates government involvement, e.g., the MOES logo on the consent forms. In the future the program plans to involve SMCs and PTAs to create awareness and mobilise parents to allow their children to participate in KAP survey. In the previous survey, time did not allow this to happen. Quarterly report, April 1 – June 30, 2013 What is your guidance on how to best compare the quarterly reports to the work plan and PMP? The discussion helped SHRP realized disconnect between quarterly reports, PMP, and work plan. They said in the next quarterly report that they will seek to provide more clarity on the relationship of the reporting to the work plan and PMP. The program has developed a “dashboard,” which includes all PMP indicators. This is included in the quarterly reports and includes a column explaining the status of indicators and reasons for not reaching targets (and planned programmatic modifications). Review of progress on PMP indicators (May 2013 version) Indicators 2a – 2c: when are baseline data expected to be available? E.g.: Program agrees they need to do a better job of telling the story behind the numbers. We have re-doubled efforts to collect success stories from the field and include these, along with more narrative, in the reports. All of this information is included in the dashboard. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 80 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested 1.1.1 Number of laws, etc.; value is 0 – it would be useful to provide an explanation 1.2.2 Number of textbooks; again, value is 0 – it would be useful to provide and explanation 1.3.1 Number of teachers; actual breakouts are very different from expected numbers (more teachers than expected and fewer CCTs) – is there a reason for that, or does this not matter? No target is given for 2013, so why is data now being collected? Should there be a target? 1.4.1 Number of PTAs or structures supports; target was 410, actual is 0 – is there a reason? No actual values were provides for number of indicators, e.g.,1.4.2, 1.5.2, 2.1.1, 2.1.2, 2.2.5, and others – if these are not relevant because, for example, components have been postponed and redesigned, should probably indicate N/A Rapid Monitoring for HIV and AIDS activities implemented at school, July 2013 Some schools did not seem to be clear about the data/recordkeeping requirements during the consultative meetings with school administrators and teachers, it appeared that many R2-related activities have been undertaken at schools but not documented. There appears to be a need to clarify as well as provide reminders to schools on recordkeeping requirements. As this process is just being rolled out, the program anticipates the on-going need for oversight and training. The program has been exploring options on how to address the issue of data/record-keeping at the school level. One possible solution is to have a journal where teachers and administrators can record interpretation of feedback. School family initiative, club and guidance and counselling registers are distributed to all program 1651 schools at the beginning of the school year or the end of the last school year. The registers cover a full school year and support our data collection processes. DQ checks have been done on the data from the registers and the results how the quality of the data to be acceptable. Support supervision for EGR activities, July 2013 SHRP Clinical Support Supervision approach emphasizes the importance of the post-observation feedback session where teachers engage in self￾assessments and also receive feedback from supervisors and thereby recognise areas of competency and areas for improvement. However, some teams were not organised to present systematic feedback and/or to build from appreciative to constructive feedback. In addition, teachers were busy responding to questions from a number of supervisors and couldn’t take notes. Perhaps there is need to have a process before the post￾observation sessions where the supervisors can better organize their feedback so it is systematic and ordered from appreciative to constructive; and also a way to document the feedback for teachers to use in future self￾reflection and to create a baseline for future support supervision activities. Program will work out the best way to provide teacher feedback. However, having multiple supervisors observing one teacher was unique in this case because they were simultaneously modelling the support supervision methodology. The support supervision process has come a long way since it was initiated in 2013. Feedback processes have been streamlined and teachers are equipped with reading journals to take note of the feedback. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 81 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested Joint Planning meeting for year 2 work plan, July 2013 We would suggest that future work planning meetings include participation of the MoES Planning Unit and Statistics Department which are anticipated consumers of SHRP data. It appears that these officials rarely attend SHRP meetings and activities. Program agreed with this comment to some extent and said they will try to address it. The program has become involved in the work carried-out by MoES (i.e. sector review, M &E working group meetings) rather than only expecting participation in program activities. The planning unit has been in the Result 2 activities which include incorporating HIV indicators into the national EMIS. For this work planning cycle, planning with the MoES will take place next week. Various MoES offices were involved in the initial mapping of activities also (NCDC materials production for example). The program is exploring options for carrying forward the work of integrating the HIV indicators into EMIS. Early Grade Reading Support Supervisions in Kole and Wakiso, August 2013 Teachers challenged by the extent to which they are currently involved in lesson plan development. We do not know if this will be resolved when the instructional materials become available. We have seen many errors in lessons, both by native and non-native speakers of the local language. For example, we observed many teachers presenting lessons with spelling errors, in some cases with as many as five out of six vocabulary words in a list spelled incorrectly. While this problem may be substantially resolved when schools receive printed instructional materials, it presents as something that needs to be addressed in the interim. What is the updated estimate of when teaching materials will be distributed to schools? Will the instructional materials include sufficient content for lessons so as to eliminate or at least reduce errors in lesson plans? Program agreed that teachers are challenged in preparing lessons in their respective local languages. They believe this problem will be reduced when the instructional materials reach schools which include most of the content teachers need to prepare lessons, e.g., vocabulary words, sentence structure, etc. Teachers received a lesson plan template that guides lesson preparation in line with SHRP methodology and national curriculum requirements. Teachers will continuously receive further support from CCTs/School Inspectors trained in providing technical support supervision. Inconsistencies in the newly developed orthographies will be corrected as teachers provide feedback on the instructional materials. The program (through MoES channels – CCTs) continues to provide in-class support supervision to ensure teachers are better able to deliver the reading curriculum. The materials were available for the majority of training this year and this made a big difference. The hope is that the next round of materials will be ready for the January 2015 training. The content is sufficient for lesson planning and templates of lesson plans are distributed as part of the training. Result 2/HIV Data Management, Assessment, and M&E, August 2013 The scope of the training was too wide to be covered in 2.5 days. Some major topics were short shifted as a result. The training included a relatively long presentation on FPO job descriptions, orientation to SHRP result 2 activities, and corresponding data collection needs. These consumed a great deal of time and even so did not appear to conclude to the satisfaction of participants. Perhaps only obvious in hindsight, but such large issues/topics should be the focus of separate sessions, and each session organized to focus on fewer and related topics that can be covered in the allocated training timeframe. Program recognized the comment and will apply this learning to future workshops. The program strives to ensure that workshop agendas are not overcrowded leaving participants time for genuine learning and reflection. During the teacher training in January the program developed a facilitator’s manual that ensured that the participants participated actively but at the same time ensured that the timetable was not too crowded. Feedback from that training was positive. The days were busy but not too busy so that learning did not happen. Cluster 2 P1 Material Writing Sessions, August 2013. No officials from NCDC/MOES were observed supporting material writing activities. Are they not needed at this stage? NCDC is responsible for recruiting the panel of writers for SHRP materials, and it is understood that NCDC officials will be checking into the sessions but not attending full-time. Program will discuss this further with the Literacy Advisor and also inquire as We have tried to get the MoES slots filled and the curriculum specialists to join but unsuccessfully. It is difficult to get ministry officials to recognise their role in the materials development function since they cannot be paid, and the NCDC officials are In addition, one language group had fewer members. It would be good to have equal teams to ease the work of MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 82 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested trainers to allocate tasks to the different pairs within a language group. It was observed that members in the group work in pairs. But groups with fewer writers seem to have individuals working on a task alone in order to achieve the same output at end of the day. to whether there were individuals in the different local language groups who worked alone (writers are expected to perform the tasks in groups). always busy in other activities. Some panels have fewer people because not everyone that was elected can come, if they are a member of the language board, for instance, or if they are a CCT that their principal will not allow to come away from the college. When positions have fallen vacant due to such constraints we have requested NCDC to mobilise replacements but this too seems one of the most difficult tasks for them to accomplish. For subsequent groups – Cluster 2 P2 and Cluster 3 P1 we agreed to select some individuals who had demonstrated knowledge of the orthographies of their languages and understanding of the methodology, even if they were not writers, and have them join the panels. It has been productive. Refresher training for TOTs on Cluster 1 P1 materials (teachers guides and primers), September 2013 It was not clear whether trainers had presentations they followed during the sessions. Some sessions were not systematically delivered, and many seemed to lack logical order or content. For example, at one station, the session on learner continuous assessment lasted for less than 30 minutes and was taught together with support supervision; however the agenda showed these topics as two separate presentations. At another station, support supervision was held separately but for only 8-10 minutes. Trainers did not appear to be clear on what to present on these two topics. Program has noted this and recognizes the need to do more in the packaging of materials and improve on the sequencing in a systematic way. This has sparked a lot of discussion around the realities of having 5 hours of training not 8. More detailed training materials (facilitator’s guides) have been developed which include objectives and timing for each session. We have made sure that we have more hours to train by limiting logistics and other non-training activities (such as registration). Also, now the teacher guides and learner primers are available in adequate amounts at the training venues. This was not the case in September. This tremendously facilitates training efforts. Trainers were not prepared to answer questions about the support supervision book. They could not answer the following kinds of questions: • How should the book be used? • Was one book to be used by all trained teachers? • Would SHRP’s support supervision book replace the MoES template currently used by head teachers to monitor classroom lessons? The teachers’ guides had the key sessions to be presented. Continuous assessment was in the teachers guides as well. Sessions that were not in the teachers’ guides like support supervision books had separate write-ups to guide the presentations. If the trainers did not get it right: we will address it during support supervision. We do realize in the future, we need to outline the training topics more specifically and clearly – writing up more detailed training plans. However, this information is good for us to be more keen on trainers’ capability in future trainings. The use of the SS book has been a topic of discussion in standalone and the on-going teacher trainings – and also demonstrated during SS. It is now in common use. The use of the SS book has been a topic of discussion in standalone and the on-going teacher trainings – and also demonstrated during SS. It is now in common use. This has been done MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 83 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested Head teacher requests for guidance on key issues to assess when observing teacher during lessons; we understand RTI has a classroom lesson observation too), were the head teachers supplied with a copy? Each head teacher was and will be given a copy of the lesson observation tool that SHRP has developed; which has also been reviewed by MoES. We are trying to get away from relying on paper copies since this has not proven to be sustainable in the past. This indicates a need for more focus on this during the TOT. Program is planning to have leadership training in January 2014 for Head Teachers and will provide more guidance on the use of the book/tool during the training. This instrument has been widely shared and utilized during support supervision. The blue books (referenced above) are also now in wide use (rather the relying on the duplication of the form). The books will be discussed again at the leadership training which is being held this month (August, 2014). Refresher training for teachers on Cluster 1 P1 materials (teacher guides and primers), September 2013 We observed that during the practice lesson planning session, participants were divided into large groups ranging from 12-15 people which were too large for full participation. Some participants dominated the discussion. Issue will be addressed in future; trainers will be advised to break teachers in smaller groups. Teachers are now in groups of no more than 8 teachers. KAP Assessors Training, September 2013 The trainees were a combination of new and veteran assessors. The returning assessors were more active and handled more of the questions than the new assessors. In some instances, the veteran assessors seemed to dominate. New assessors may need to have additional training so that they are equally prepared as the returning/veteran assessors and able to contribute fully. This is a good point that they have also observed in training EGRA assessors, and will be addressed more fully in future training to new assessors. This will be taken into consideration for the next round of KAPS planning in September, 2015 (see above). EGRA survey data collection, October 2013 The assessors consistently made short introductions to prepare learners for interviews. The introduction did not seem to have the desired effect of building confidence of learners to speak up. Some learners remained timid, avoided eye contact with the assessor, and did not seem to listen to the instructions or really read the protocol they were provided. If there are many learners that can in fact read some letters/words but are too shy in front of the assessors to talk, it will affect the data. Overall, the assessors seemed insufficiently skilled in making learners comfortable and attracting and maintaining learner attention. Future trainings should build skills in these areas so that assessors are more adept in conducting interviews with timid learners as this kind of learner will be encountered at all EGRA stages. Building learner rapport is included in the training and emphasized throughout field work. We will look at our training plan to see if there is sufficient emphasis and practice. We believe most of our assessors are very skilled in this area. Of course, when there are outside observers’ things always seem tenser on all sides. This was re-emphasized in the February, 2014 training and the program will continue to do so. One of the sub-tasks the assessors involved asking learners to identify objects placed on a table, e.g., a pencil, paper, rubber (eraser), etc. While this would seem an easy exercise, it often did not go well. Learners might have been confused by the instructions, many did Learners are never asked to identify objects verbally but to follow English commands (“on the paper”, “behind you”). We of course are, always re￾evaluating the use of various tasks and will do so with this one as well based on input from the field Consistency in administration was emphasized in February, 2014 training. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 84 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested not speak up, and others said the name but did not point to the object. Some assessors did not place objects on the table. This exercise needs to be reviewed, e.g., to make sure the placing of objects is handled consistently and to address how learners get sufficient information to understand they are supposed to say the word and point to the object. but also based on EGRA results. This will also be an emphasis in future training. Program will evaluate the results of this task. P&IE emphasized the main point here is to ensure that the task is consistently carried out the same way by assessors. Orientation of C2 Language Board members in Masindi (Runyoro-Rutooro) The two documents distributed during the workshop (scope of work-SOW and workshop program) contained differing objectives, i.e., the SOW stated two objectives (take the LLB members through terms of reference and constitutional framework, and fill vacant positions for LLBs and writing panels), and the workshop program stated three objectives (orient the LLB in its roles and responsibilities; fill vacant positions for LLB and writing panels; and recommend 5 people to work with SIL LEAD to review the orthography). This continues to be confusing and should be easy to remedy. We will work to ensure closer alignment between the participants’ documents in the future. To allow for adequate engagement in the workshop the numbers of days for the LB meetings was increased to 2 for C3 Subsequent LB meetings for had clear objectives consistent with the Program. During workshop we observed communication issues between the facilitators and the participants. Some participants preferred communicating in the local language but the facilitators selected did not know the local languages. It was quite challenging for facilitators to respond to questions forwarded in the local language or get involved in discussions when participants changed to local language. While it is difficult for RTI to recruit a LLB Consultant who speaks all SHRP languages or to have a Consultant for each region, perhaps the LLB Chairs or some other appropriate person could be enlisted to provide translation during the meetings. Program agrees there should be clear communication between the facilitators and participants when it comes to discussions leading to agreements on critical issues. We understand there is no way participants can carry on productive discussions if the facilitators are not involved. We try to have the discussions in English which all the LB members know and understand as a minimum requirement. In cases where there is a switch to LL translations into English will be encouraged to enable the non-local language speakers engage and follow as a matter of procedure. The workshop had lengthy reading sessions by one facilitator of the different articles in the Constitution, during which participants lost focus and some nodded off. A better way to present the articles of the LLB Constitution is needed to maintain interest and engagement. We will note that and ensure that facilitators are equipped with skills to vary presentations. Will see how to borrow interactive techniques used in other SHRP trainings to enrich the delivery of LLB activities. Participants were given two major offsite assignments, localizing the LLB Constitution and developing a work plan for the LLB. Facilitators/organizers assumed these were easy tasks for the group, which eventually turned out to be different. Participants expressed challenges leading themselves through the assignments especially This point is noted. Written guidance will be developed where necessary for these assignments. A template for developing a work plan (the major written task mentioned) has been created and distributed to the LB teams. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 85 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested the development of a work plan. In order for RTI to improve on outputs of this activity, there is need to provide participants with written guidance on these tasks and orientation to work plan development as one of the activities on the agenda. Early Grade Literacy Master training on Cluster 1 (P1&P2) and Cluster 2 (P1) materials, December 2013. While a very effective training overall, the registration process was disruptive. Participants were registered and trainers were registered at different times in the training rooms, interrupting sessions. This is something that will be communicated to the officers supervising the trainings. Program has noted and attempted to change in more recent trainings. It is often more efficient and manageable to collect attendance information in the classrooms (especially since it has to be collected twice a day) though collecting during the sessions is discouraged. Early Grade Literacy Training for Trainers on C1 P1 & P2 and C2 P1 SHRP books, December 2013. Lunch meals were always served late, which delayed the afternoon program by an hour or more, and meant that the day’s tasks were not completed by the closing time of 5:30 pm. To try to catch up, trainers rushed through Orthography sessions, one of the last items on the daily agenda, and did not have time for participant questions or had to forego Reflection, the last activity on the daily agenda. We will try to work more closely with the colleges on this; they are the ones who provide the meals. We might even suggest they hire more staff in order to accommodate the large numbers. In other trainings, we often check in closely with the catering staff and end for lunch only when it is ready. The colleges have been doing a better job in this regard – feeding upwards of 500 participants at once is not an easy task. Orthography sessions have been taking place in plenary format and facilitators are encouraged to be flexible and work until lunch is ready so as not to lose time. Trainers, especially P2 trainers, need more support in the fourth day of training when using “How to Teach English” materials. There was only one Writer/Expert covering 12 groups. We observed trainers struggling to get clarifications on various instructions and concepts and one trainer misrepresented the SHRP reading model which discourages use of local language in English lessons. We observed a lengthy discussion where participants worried that the English vocabulary word “mama” would confuse learners since it also means mother in local languages; fortunately, the Expert/Writer ultimately visited the class and settled the matter. We need to rethink the way we distribute and pair up P2 trainers and assess their skills. In some ways, the trainers have only just been trained themselves. During the refresher training in May, we made effort to address this issue by pairing the more experienced trainers with the less experienced ones. We shall continue pairing the trainers carefully to ensure quality training. It would be good to cluster administrative announcements and find a way to handle participant registration outside of sessions so as to disrupt sessions less. This will be communicated to the program training team. The registration process has improved. In most cases participants register before the first sessions, during tea break and lunch time but participant registration of several hundred participants (except for the initial intake) is too difficult to do outside of the classrooms; however facilitators are instructed not to use session time for this. Early Grade Literacy training for teachers The training stations were short of training materials (one station had no materials) and participants had to share the few copies available. We are considering developing a training material pack -- or a pack of general instructive materials that will help minimize the risk of collating and printing various sections of our pupil book and teacher It is true, delays in the production of materials hampered more than the ability to get the books into the hands of learners – it negatively impacted the MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 86 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested on C1 (P1 & P2) and C2 (P1) SHRP books guides for the training. This we hope may also assist teachers to get started in the classroom in the event that materials are not available at the beginning of the term. The identification of this pack will eliminate any confusion on the materials that need to go to the training vs. the materials that will be used at the schools. Though this has not been possible to date, it is the hope with Cluster 3 P1 materials that they will be available for the training next January and then teachers can go right to the classrooms. This will alleviate the need for the development of stop gap materials. training as materials had not been adequately available. The team in charge of material production has assured us that the teachers’ guides and the primers for all clusters will be ready by the time we begin training in January 2015. During the refresher training in May, there were sufficient teachers’ guides, pupil books and orthography guides except the Ateso orthography that was being reviewed. There were inadequate trainers to create manageable groups of participants. At one station, there were more than one hundred participants with only two trainers. In one group, one of the trainers did not speak the local language so the second trainer had to handle most of the load. We have increased the number of trainers; trainers who do not speak the area local languages are advised to present English lessons. The training content was overly ambitious which meant that some subjects were left out, such as Orthography. Trainers hoped to fit them in elsewhere in the agenda but that wasn’t possible. The Situation in May 2014 was much better than 2013.We shall plan manageable content in future trainings. In addition, we shall advise trainers to manage time well because those who do not manage time hinder completion of the planned content. EGRA C2 Baseline Assessors and Supervisors Training, February 2014 Preparation The training of trainers was very short, primarily consisting of a review of the agenda for first two days of training and watching video of letter sounds. Preparation did not include slides or role￾playing. Consequently, there were issues that came up in the training (e.g., transitions between tasks, focus of training modules, etc.) which could have been avoided with adequate training of trainers. E.g., Assessors were invited to select their own language instrument instead of having a single standard instrument for the first run-through of the tablet version. This caused a One issue is that the English instruments all have instructions that are translated into the various local languages. In fact there is not just one “English version” but 4 and the numbers should all be in agreement between these 4 versions. We will look into this. The next round of EGRA data will be collected in October, 2014 with training also starting in October. This information will be taken into consideration during the planning for those efforts. As mentioned, there is no one single version – but perhaps we can disable the random function on the tablets so all start with either English or LL. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 87 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested lot of confusion because local language versions have a different numbering system than the English version. Materials There were errors or missing instructions on the paper version of the instruments and some inconsistencies with the tablet versions. Errors of this type have the potential to reduce the confidence of assessors in the instruments, so we would recommend allowing adequate time for a careful review before the next round of data collection, taking into consideration the large number of instruments that require review. Instruments will be reviewed. We aim to do a thorough job of reviewing all instruments and there are in fact differences between paper and Tangerine administration (slashing vs tapping for example). If we are aware of mistakes, we will change them. It should be noted that no assessment were done on paper for the last two data collections. An alternative is to print paper copies directly from Tangerine as is the practice in some other countries. The downside to this is that it uses a considerable amount of paper as the formatting within Tangerine has yet to be optimized. Review of the instruments for the October data collection started in June. Consistency will continue to be the priority. There were no scripts for the demonstrations until the final IRRs. This resulted in some unrealistic and confusing demonstrations, particularly when volunteer assessors carried out the role-play demos. Yes, this is agreed. There was a script (the marked up instrument with previously selected errors) but it is agreed that we need to select volunteers more carefully and then practice the IRRs with more rigor. This will be a priority for our DQA (regional supervisor) training which will take place in September. We recommend that the interviewer manual be updated and include a “QxQ” (Question-by-Question) explanation of how each question is to be treated by the interviewer. The manual should be provided to assessors before or at the beginning of training and be considered required reading. Creating and adhering to this manual would reduce conflicting answers provided during training. This will be developed for the learner context questions. The paper instrument itself provides ample explanation of the EGRA tasks. Training videos were difficult to hear and understand. Both the sound-letter video and the videos of sampling were shown to assessors accompanied by some description. The sampling video was particularly difficult to This seemed to be clear. We believe that seeing and hearing examples is more effective than just speaking. It is not easy to make good videos and we are still working on this. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 88 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested understand and led to confusion on the part of assessors. Activities Trainers carried out role-plays to demonstrate the assessor-pupil interaction of the main instrument. Except for the final IRRs, these role-plays were not scripted, so they were a bit disorganized and the trainers did not introduce the tasks in advance of the demonstrations clearly. When volunteer assessors were demonstrating, they presented a number of errors. Demonstrating poor techniques before demonstrating good techniques tends to confuse interviewers, so we recommend trainers carry out scripted demos for the first three days and provide scripts to volunteer assessors. We will work ahead of time to ensure that examples are clearer and that there is enough practice ahead of time. There was a session devoted to this which included a role play. We will look into this but do believe we had ample time to cover everything in our agenda. This will be a focus of September DQA training. The consent/ introduction to the pupil instrument was reviewed quickly on the first day and skipped entirely when the assessors started using the tablets. Supervisor Training The field manual made its first appearance at the supervisor training. Supervisors read from a few pages, although trainers quickly noticed that a number of the tasks listed for supervisors were only applicable to paper instruments or were only carried out by DQAs. The entire supervisor training lasted about an hour and there were few opportunities for questions from the new supervisors. We very strongly recommend that a minimum of 2/3 of a day of a well￾organized training be dedicated to the supervisor training, as supervisor The supervisor training was not rushed and there was time for questions, it ended at half day and the supervisors remained behind to support packing. We will look into areas that may need more support. An initial day long DQA/supervisor training will take place in September, prior to the larger training. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX E: MATRIX SUMMARY OF PERFORMANCE FEEDBACK MEMOS 89 Activity Constructive Feedback Action Points (response from SHRP) Comments about status of actions suggested must guarantee the day to day data quality and logistics of their teams. EGRA administration of certain subtasks We noted that certain guidelines given to enumerators about EGRA implementation raised concerns for the impact evaluation. These concerns mainly involve: • The types of sounds accepted for the letter sound knowledge and segmenting subtasks • The types of pronunciation accepted for words in the reading passage We have provided detailed notes regarding the implication of these EGRA implementation guidelines on the impact evaluation in a separate memo that we shared with RTI and USAID on February 21st. NORC and RTI are meeting to discuss the memo on March 18th 2014. These issues were discussed on March 18th. SHRP will ensure that the range (narrow) of acceptable letter sounds is well known by the assessors. We are also working on passages to ensure that letters and combinations of letters that are prone to maternal language interferences/ transfer will be minimized. In August, home office and local literacy advisors will hold a half-day session with facilitators and trainers to ensure that all facilitators/trainers are fluent with this range of letter sounds. Reading passages were developed to minimize the issues around maternal language interference. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX F: SOURCES OF INFORMATION 90 ANNEX F: SOURCES OF INFORMATION F.1. NATIONAL AND DISTRICT INTERVIEW AND FGD CONTACT LIST USAID Organization Title Contact Name Contact Number Contact Email USAID Education Specialist Sarah B. Mayanja Office: 414-306-001 smayanja@usaid.gov USAID Strategic Information Unit, Director Support Office of Health and Education Joseph Mwangi Mob: 0772-138506 jmmwangi@usaid.gov USAID/Health PEPFAR Point of Contact HIV/AIDS Prevention Specialist Rhobbinah Ssempebwa Mob: 0772-138526 rsempebw@usaid.gov MOES and Affiliated Institutions Organization/Unit/Department Title Contact Name Contact Number Contact Email Special Needs Education Commissioner of Special Needs SHRP Focal Point Officer Martin Omagor Mob: 0750-58739, 0772-428483 mlomagor2006@yahoo.com Guidance and Counselling Commissioner, Guidance and Counselling, SHRP R2 Component Manager George Opiro Mob: 0772-977100 Opiro66@yahoo.co.uk MOES HIV Unit Chief of MoES HIV/AIDS Unit Roland Biryahwaho Mob: 0782-452452 rolandbiryahwa@yahoo.com Teacher Instruction and Educational Training (TIET) Pre-primary and Primary Teacher Education Elizabeth Kisakye Mob: 0772-411548 Directorate of Education Standards Senior Inspector of Schools, Desk Officer for Special Needs Education Sarah Ayesiga Mob: 0772-453354 Sarah_ayesiga@yahoo.com National Curriculum Development Centre Deputy Director, NCDC Angela Kyagaba Mob: 0772-196666 akyagaba@yahoo.com National Curriculum Development Centre Curriculum Specialist, Pre-primary and Primary Sarah Natunga Mob: 0789-756889 0772-683585 sarahnatunga@gmail.com Uganda National Examinations Board (UNEB) Senior Evaluation Officer, NAPE Opaman Amos Mob: 0772-601726 0702-601726 opamos@yahoo.com MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX F: SOURCES OF INFORMATION 91 RTI Staff Title Contact Name Location Contact Number Contact Email Chief of Party Saeeda Prew Kampala Mob: 0791-252525 Office: 0312-202884 sprew@rti.org Deputy Chief of Party Nkata Derek Kampala Mob: 0772-799970 dnkata@shrp.rti.org Professional Development Specialist Scholastica Tiguryera (Schola) Kampala Mob:0772-766011 stiguryera@shrp.rti.org R1 Component Manager Robinah Kyeyune Kampala Mob:0772-766006 rkyeyune@shrp.rti.org M&E Director Tracy Brunette Kampala Mob:0791-252526 tbrunette@shrp.rti.org Finance Manager Adem Abdella Kampala Mob:0776-766636 adem_abdella@world.org Literacy Field Coordinator Rwanyonga Consilous (R1 FGD) Kampala Mob:0783-896417 crwanyonga@shrp.rti.org M&E Officer Peter Muyingo (R1 FGD) Kampala Mob:0772-423691 pmuyingo@shrp.rti.org M&E Specialist Rehema Nabachwa (R1 FGD) Kampala Mob:0782-394819 rnabacwa@shrp.rti.org Field Coordinator Eunice Alum (R1 FGD) Kampala Office: 0312-202884 Consultant R1 Material Development Anna (R1 FGD) US Program Education Officer Allen Atutambira (R1 FGD) Kampala Office: 0312-202884 Field Assistant Ayeke Frank Tadeo Lira Mob:0772-699030 0704-904294 Frank_ayeke@yahoo.com Field Assistant-R2 Loy Akello Lira Mob: 0784-094424 0772-699030 loyakello@yahoo.com Field Assistant - R1 Peter Mutimbo Kabale Mob:0702-827198 mutimbop@gmail.com Field Assistant – R1 Ntundubaire Mark Kabale Mob:0784-943511 ntundubairemark@yahoo.com SHRP Implementing Partners Organization Title Contact Name Location Contact Number Contact Email World Education Result 2 Component Manager Health Advisor Sarah Kyobe Kampala Mob: 0772-492200, 0772-205796 skyobe@shrp.rti.org World Education MIS Advisor Moses Bagyendera (R2 FGD) Kampala Office: 0312-202884 mbagyendera@shrp.rti.org World Education M&E Specialist Apolot Florence (R2 FGD) Kampala Mob: 0776-766634 fapolot@shrp.rti.org World Education Vice President/Africa Shirley Birchfield Boston, MA (617) 482-9485 ext. 3825 sburchfield@worlded.org MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX F: SOURCES OF INFORMATION 92 SHRP Implementing Partners Organization Title Contact Name Location Contact Number Contact Email SIL LEAD Uganda Project Manager Susan Mubala Kampala Mob: 0772-411882 susan_mubbala@sil-lead.org SIL LEAD Project Support Specialist Stacey Maresco Washington, DC (202) 466-0552 Stacey_maresco@sil-lead.org Center for Social Research Director Wilson Asiimwe Kampala Mob: 0772-685728 wilasatmisr@yahoo.com VSO Special Needs Education Volunteer Veronica Stapleton Kampala Mob: 0774-774804 ronniestapleton_2000@yahoo.co m African Development Corps (ADC) Country Director Tom Evans Kampala Mob: 0777-944838 tevans@africadevcorps.org Peace Corps Literacy Coordinator Audrey Spenser Kampala Mob: 0772-200533 Aspencer@peacecorps.gov DRASPAC (KAPS technical assistance) Principal Investigator (KAPS) Stephen Kirya Kampala Office: 0312-516619 draspac@draspac.org Kabale District Institution Title Contact Name Contact number Local Government Kabale Municipal Council Education Office Municipal Inspector of Schools Mutahunga Elia Mob: 0772-613898 Kabale District Education Office District Inspector of Schools Beyendera Vastine Mob: 0772-572300 Local Language Board Runyankore-Rukiga Language Board Local Language Board Member Reverend Dr Muranga Mob: 0782-319133 Primary Teachers College Kabale Bukinda Core Primary Teachers College Principal Javan Rwamafa Mob: 0781-080198 Coordinating Center Tutor (CCT) Ndeego CC Christopher Musinguzi Mob: 0782-609884 Result 1 Primary school Ihunga Primary School Ndeego Coordinating Center Head Teacher Turyahikayo Daniel Mob: 0773-228138 Teachers P1 (2), P2 (2) SMC Chairperson Reverend John Kakiyangye SMC member Mr Tushabomwe Edson Result 2 Secondary School Kihorezo Secondary school Head Teacher Turyatemba Arthur Mob: 0756-398736 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX F: SOURCES OF INFORMATION 93 Kabale District Institution Title Contact Name Contact number Deputy Head Teacher Tugume Norbert Kaima Mob: 0754-755810 Learners (FGD) Total 9 (4 males and 5 females) Ages: 18-20 SMC Chairperson Edison Kiconco Lira District Institution Title Contact Name Contact number Local Government Lira District Education Office District Inspector of Schools Bosco Bwonyo Mob: 0772-567560 acupbwonyo@gmail.com Lira Municipal Council Education Office Municipal Education Officer Jayne Francis Offungi Mob: 0754-681465 jayneffunge@gmail.com Local Language Board Leblango Local Language Board Local Language Board Member Jayne Francis Offungi Mob: 0754-681465 jayneffunge@gmail.com Primary Teachers College Loro Core Primary Teacher College (PTC) Principal Simon Odwiro Mob: 0772-594869 Deputy Principal Outreach James Atalo Mob: 0782-569950 Coordinating Center Tutor (CCT) Adwila CC Richard Omara Mob: 0772-841739 Result 1 and 2 Primary school Owinyo Primary School Adwila Coordinating Center Head Teacher Geoffrey Ogwang Mob: 0777-676267 Teachers Seven trained teachers for R1 and R2 SMC Members Total 8 (3 women and 5 men) Result 2 Secondary School Agweng Secondary School Head Teacher Francis Oleke Mob: 0782-696110 folekeolero@gmail.com Trained Teachers Jimmy Eling and Dillis Aol 2 trained teachers Mob: 0776329559 Aoldillia@gmail.com Learners (FGD) Total 14 (11 boys and 3 girls) Parent Teachers (FGD) 10 parent teachers Wakiso District Institution Title Contact Name Contact number Local Government WakisoDistrict Education Office Municipal Inspector of Schools Daniel Ndaaga Mob: 0772-315797 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX F: SOURCES OF INFORMATION 94 Wakiso District Institution Title Contact Name Contact number Wakiso district Education Office CCT Lake Victoria CC Jacqueline Nshemereirwe Mob: 0772-473903 Local Language Board Luganda Local Language Board Local Language Board Member Jascent Ndagire Mob: 0772-937426 Primary Teachers College Simon Teachers Core PTC NONE Result 1 and 2 Primary school Nakiwogo P/S Head Teacher Namulumba Rosemary Lipa Teachers Tsetuyi Mary Goretti Drania Teresa Asana Rose Nyirabazungu Irene SMC members (FGD) Total 4 (2 females and 2 males) Result 2 Secondary School Kitende Secondary School Head Teacher Ruth Muyinda Mande Trained Teachers Kisembo Edward Peter Nanziri Rose Nakabugo Learners (FGD) Peer leaders (FGD) Total 13 (8 female, 5 male) Total 10 (6 female, 4 male) SMC members (FGD) Total 2 (males) F.2. LIST OF DOCUMENTS REVIEWED School Health and Reading Program • SHRP Quarterly Report: 1 July – 30 September, 2012 • SHRP Quarterly Report: 1 October – 31 December, 2012 • SHRP Quarterly Report: 1 January – 31 March, 2013 • SHRP Quarterly Report: 1 April – 30 June, 2013 • SHRP Quarterly Report: 1 July – 30 September, 2013 • SHRP Quarterly Report: 1 October – 31 December, 2013 • SHRP Quarterly Report: 1 January – 31 March, 2014 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX F: SOURCES OF INFORMATION 95 • SHRP Quarterly Report: 1 April – 30 June, 2014 • SHRP Quarterly Report: 1 July – 30 September, 2014 • SHRP Quarterly Report: 1 October – 31 December, 2014 • SHRP Annual Report May 2012 – September 2013, November 8, 2013 • SHRP Cooperate Agreement: AID-617-12-00002, RTI, May 16, 2012 • SHRP Cooperate Agreement Modification 01: AID-617-12-00002, RTI, May 21, 2012 • SHRP Cooperate Agreement Modification 02: AID-617-12-00002, RTI, May 21, 2012 • SHRP Cooperate Agreement Modification 03: AID-617-12-00002, RTI, May 21, 2012 • SHRP Year 1 Annual Work Plan: May 2012 – April 2013, November 20, 2012 • SHRP Year 2 Annual Work Plan: October 2013 – September 2014, October 16, 2013 • SHRP Performance Monitoring Plan (PMP), January 28, 2014 • SHRP Program Brief • Facilitator’s Guide for Enhanced PIASCY: Teachers’ Workshop, January 6 – 10, 2014 and January 13 – 17, 2014 • Enhanced PIASCY Training of Teachers Program: Training Report, Kitante Primary School, 2 – 6 December 2013 • Pupil Book and Teacher’s Guide, English • Process Evaluation: EGRA and KAP Data Collection Observation Tool, October , 2013 Performance and Impact Evaluation Documents • SHRP Performance and Impact Evaluation: Mid-Term Performance Evaluation Implementation Plan, April 30 2014 • SHRP Performance and Impact Evaluation: Year 1 Impact Evaluation Report, April 30 2014 USAID • USAID Evaluation Report Template, http://usaidlearninglab.org/library/evaluation-report-template • USAID ADS 200 Series, http://www.usaid.gov/who-we-are/agency-policy/series-200 • USAID Evaluation Policy, January 2011, http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Government of Uganda • Republic of Uganda National HIV Prevention Strategy 2011 – 2015, June 2011, http://uganda.um.dk/en/~/media/Uganda/Documents/English%20site/Danida/National%20HIV%20prevention%20strategy.pdf • Republic of Uganda Ministry of Education and Sports, National Curriculum Development Centre, Primary School (Thematic) Curriculum, Primary 2, http://www.ncdc.go.ug/primary/P%202%20PDF/P2%20Thematic%20Curriculum%20June%202011.pdf MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX F: SOURCES OF INFORMATION 96 • Republic of Uganda Ministry of Education and Sports, Education Planning Department, Education Sector Strategic Plan 2004 – 201, June, 2004, http://planipolis.iiep.unesco.org/upload/Uganda/Uganda_ESSP_2004_2015.pdf MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX G: SHRP USE OF FUNDS MAY 2012 -- MARCH 2014 97 ANNEX G: SHRP USE OF FUNDS MAY 2012 – MARCH 2014 Summary of SHRP Budget Projections and Actual Expenditures May 2012 – March 2014 Cost Elements May 2012 - September 2013 (16.5 months)7 October 2013 - March 2014 (6 months)8 May 2012 - March 2014 (22.5 months) Budget Overview Budgeted Actual Variance Budgeted Actual Variance Total Budgeted Total Actual Total Variance Percent of Total Budget Labor $947,876 $1,099,350 $151,474 $671,868 $477,350 $ (194,518) $1,619,744 $1,576,700 $(43,044) 10.41% Fringe Benefits $278,262 $272,946 $(5,316) $129,307 $183,915 $54,609 $407,569 $456,861 $49,293 3.02% Travel $117,958 $189,867 $71,909 $55,070 $71,348 $16,279 $173,028 $261,215 $88,188 1.72% Equipment $319,242 $217,985 $(101,257) $16,402 $196,284 $179,883 $335,644 $414,269 $78,626 2.73% Supplies $3,730,829 $304,658 $(3,426,171) $1,380,257 $337,051 $ (1,043,206) $5,111,086 $641,709 $(4,469,377) 4.24% Other Direct Costs $1,413,001 $1,734,135 $321,134 $1,611,572 $1,678,622 $67,051 $3,024,573 $3,412,757 $388,185 22.53% Sub recipients $4,308,554 $3,323,325 $(985,229) $1,982,838 $2,508,464 $525,627 $6,291,392 $5,831,789 $(459,603) 38.49% Grants $339,250 - $ (339,250) - - - $339,250 - $(339,250) 0% Total Indirect Costs $1,446,682 $1,374,136 $(72,546) $935,769 $915,940 $(19,829) $2,382,451 $2,290,076 $(92,375) 15.12% Subtotal $12,901,654 $8,516,402 $(4,385,252) $6,783,080 $6,368,974 $(414,106) $19,684,734 $14,885,376 $(4,799,358) - Cost Share $1,350,757 $172,630 $(1,178,127) $675,378 $92,053 $(583,325) $2,026,135 $264,683 $(1,761,452) 1.74% 7 Materials used to develop these figures were gathered directly from the following reports; USAID/Uganda School Health and Reading Progarm, Annual Work Plan, May 2012-April 2013, page 29. School Health and Reading Program, Quarterly Report, 1 July - 30 September 2012, page 18. School Health and Reading Program, Quarterly Report, October 1-December 31, 2012, page 28. USAID/Uganda School Health and Reading Program, Quarterly Report, January 1-March 31, 2013, page 35. USAID/Uganda School Health and Reading Program, Annual Report, May 2012-September 2013, page 36. School Health and Reading Program, Quarterly Report, October 1-December 31, 2013, page 26. 8 Materials used to develop these figures were gathered directly from the following reports; USAID/Uganda Health and Reading Program, Quarterly Report October 1- December 31, 2013, page 26. USAID/Uganda School Health and Reading Program Quarterly Report, January 1-March 31, 2014, page 24. USAID/Uganda School Health and Program, Quarterly Report, April -June 30, 2014, page 35. MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX G: SHRP USE OF FUNDS MAY 2012 -- MARCH 2014 98 Total Program Costs $14,252,411 $8,689,032 $(5,563,379) $7,458,458 $6,461,027 $(997,431) $21,710,869 $15,150,059 $(6,560,810) - MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX H: DISCLOSURE OF ANY CONFLICTS OF INTEREST 99 ANNEX H: DISCLOSURE OF ANY CONFLICTS OF INTEREST MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX H: DISCLOSURE OF ANY CONFLICTS OF INTEREST 100 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM ANNEX H: DISCLOSURE OF ANY CONFLICTS OF INTEREST 101 MID-TERM PERFORMANCE EVALUATION: SCHOOL HEALTH AND READING PROGRAM 102 U.S. Agency for International Development/Uganda U.S. Mission Compound-South Wing, Plot 1577 Ggaba Road PO Box 7856 Kampala, Uganda