New Horizons for Primary Schools in Jamaica: Inputs, Outcomes and Impact by Marlaine Lockheed, Ph.D. Abigail Harris, Ph.D. Paul Gammill, M.S. Karima Barrow, M.A. Tamara Jayasundera, M.A. Presented to Claire Spence, Deputy Office Director, Office of General Development United States Agency for International Development USAID/Jamaica Kingston, Jamaica by Academy for Educational Development Education Policy and Data Center Washington, DC 20009-5721 January 31, 2006 (Revised) Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact ACKNOWLEDGEMENTS This evaluation report was prepared for USAID by Marlaine Lockheed, Abigail Harris, Paul Gammill, Karima Barrow and Tamara Jayasundera.1 A preliminary version was discussed at a meeting in Kingston, Jamaica on January 9, 2005 with Ms. Karen D. Turner, USAID Mission Director; Ms. Margaret Sancho, Director, Office of General Development (Heath and Education); Ms. Claire Spence, Deputy Office Director, Office of General Development; Ms. Joan Davis, Project Management Specialist (Education), Office of General Development; Ms. Joan Taffe Program Management Specialist (Health and Education backstop) Office of Program Development and Management; Dr. Jean Beaumont, NHP Chief-of-Party and Dr. Doreen Faulkner. The evaluation was funded under contract USAID Contract Number NE-I-000-00-0076-00 to the Academy of Educational Development. We wish to thank the Jamaican Ministry of Education, Youth and Culture, and particularly Ms. Adelle Brown, Chief Education Officer, for support and insights. Other thanks go to the Planning Unit, under the direction of Ms. Barbara Allen, for openness to ideas, providing technical support and enabling rapid access to school census data; to the Student Assessment Unit, under direction of Ms. Sephlin Myers-Thomas, for meeting with the team and providing key information; to Ms. Ruth Morris, Executive Director, National Council on Education, for her suggestions and insights in planning this study; to the project staff of the New Horizons Program for providing copies of technical reports and project documentation; and to Ms. Carol Watson of DPK for assistance with the GSAT data. Field data collection was carried out by the Caribbean Consulting Group of Kingston, Jamaica under the direction of Ms. Carol Rose Brown. Suggested citation: Lockheed, M., A. Harris, P. Gammill, K. Barrow and T. Jayasundera (2006). New Horizons for Primary Schools in Jamaica: Inputs, Outcomes and Impact. Washington, DC: Academy for Educational Development Education Policy and Data Center Report to USAID. 1 Marlaine Lockheed is a former World Bank manager and is currently a Visiting Fellow at the Center for Global Development, Abigail Harris is Associate Professor in the Graduate School of Education at Fordham University, Paul Gammill is a former Academy of Educational Development data systems manager and is currently Senior Quality Assurance Manager for Electronic Data Systems and VP of Technology of EdCount.com,, Karima Barrow is a Research Analyst at the Academy of Educational Development, and Tamara Jayasundera is Research Analyst at the Academy of Educational Development and a doctoral candidate in economics at the American University. 2 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact EXECUTIVE SUMMARY The New Horizons for Primary Schools (NHP) was implemented in 72 government schools in Jamaica, from 1998-2005. The program provided support to schools on the basis of needs identified through the preparation of a School Development Plan (also called a School Improvement Plan). This independent evaluation report first compares the schools in the NHP with a statistically matched comparison group of government schools, with respect to (a) support received for literacy and numeracy, (b) learning outcomes as measured by national tests in grades 3, 4 and 6, and (c) impact of NHP on learning comes. Second, the report assesses the effects of various school factors on student learning outcomes. Third, the report assesses the utility of the existing national tests for monitoring change over time. Finally, the report makes recommendations for better measuring NHP outcomes and for better assessing the impact of the NHP on student learning. Support for literacy and numeracy was greater in NHP schools as compared with a group of statistically matched non-NHP schools. In October and November, 2005, researchers visited 71 NHP and 67 statistically matched comparisons schools and carried out group interviews with the principal and experienced teachers2 . The survey asked about innovative mathematics and literacy programs at the school, in-service teacher training, governance and leadership training, parent education and training, supplementary reading and math materials, computer use in schools and training teachers about computers, training resource teachers, nutrition and health programs, integrating statistical data bases, and linking project schools with national EMIS. In all areas, other than parent education and health and nutrition, NHP schools were more advantaged than matched non-NHP schools. Learning outcomes in NHP schools and matched non-NHP schools fluctuated over time along with national fluctuations in test results and were not consistently higher in NHP schools as compared to matched non-NHP schools. National school-level performance data on the Grade 3 Diagnostic test and the Grade 4 Literacy test and individual student performance data on the Grade 6 Achievement test (GSAT) were obtained and analyzed. No differences were observed in seven of the ten tests analyzed. The performance of students in NHP schools was higher than the performance of students in the statistically matched non-NHP schools on two tests: • 2004 Grade 3 Communications Task I • 2004 Grade 6 Communications Task I The performance of students in NHP schools was lower than the performance of students in the statistically matched non-NHP schools on one test: 2 Although 72 schools originally participated in NHP, one school closed. 3 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact • 2005 Grade 6 Communications Task I The performance of students in NHP schools was no different from the performance of students in the statistically matched non-NHP schools on the following seven tests: • 2004 Grade 3 Math • 2004 Grade 3 Language • 2004 Grade 4 Literacy • 2004 Grade 6 Math • 2004 Grade 6 Language • 2005 Grade 6 Math • 2005 Grade 6 Language Multiple regression analysis of the effects of various NHP inputs on school average literacy and numeracy found few systematic impacts of any inputs. However, at the margin: • Schools that were provided with more different types of supplementary reading materials had a smaller share of students performing at non-mastery and less-than￾mastery levels on the Grade 4 Literacy test • Schools where reading resource teachers received more days of training also had a smaller share of students performing at non-mastery on the Grade 4 Literacy test and higher school averages on the GSAT2005 language test • Schools with a functioning computer for administrative purposes and schools with internet access had higher GSAT math and language scores At the same time, GSAT math and language scores were lower in schools receiving more in-service teacher training on the revised curriculum and lower in schools where principals participated in peer learning, with these inputs provided possibly in response to school needs. The existing national tests have drawbacks that limit their utility as USAID PMP indicators. Each of the national tests was analyzed to assess their utility as monitoring indicators. Drawbacks differ from test to test, and the principal ones are: • Possible changes in the timing and content of the Grade 3 Diagnostic test • Lack of precision in reporting sub-scores and total scores from the reporting sheets of the Grade 4 Literacy test, which reports only non-mastery, near-mastery or mastery levels of performance • Lack of annual, post-test equating, through empirical IRT methods, to place test results on a common scale to ensure comparability of measures over time; this was a drawback for all tests, including the GSAT. Monitoring and evaluation could be strengthened, to better measure the impact of the NHP and provide guidance for system improvement. We recommend that two steps be taken for better monitoring of changes over time: • Facilitate statistical equating of national tests through providing the MOEYC/SAU with appropriate technical support 4 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact • Encourage the MOEYC to collect actual Grade 4 Literacy test item responses and scores, not just mastery levels We also recommend that three steps be take for better measuring the impact of the NHP: • Prior to implementing the new NHP, create a prospective comparison group through propensity score matching (PSM) of all NHP schools with a sample of non￾NHP schools; use the most recent data from the Annual School Census and 2005 national test results to create the propensity scores • Beginning in 2006, regularly collect data from all NHP and all statistically matched non-NHP schools; install the JSAS in the statistically matched non-NHP schools to routinely collect administrative data; replace obsolete computers in NHP schools and provide computers as needed to matched non-NHP schools; monitor use of JSAS. • Use independent evaluators to (a) extract routine administrative data, (b) collect any additional data as necessary, and (c) use statistical methods to assess the impact of the NHP program as a whole and the contribution of various interventions to improvements in learning outcomes. 5 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact ACRONYMS GSAT Grade Six Achievement Test CEE Common Entrance Examination 11+ HLM Hierarchical Linear Modeling MOEYC Ministry of Education Youth and Culture NCE National Council on Education NHP New Horizons for Primary Schools PDU Project Documentation Unit PIU Project Implementation Unit SCOPE School Community Outreach Program for Education SDP School Development Plan SIP School Improvement Plan USAID United States Agency for International Development 6 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact TABLE OF CONTENTS Acknowledgements................................................................................................................ 2 Executive Summary............................................................................................................... 3 Acronyms............................................................................................................................... 6 Table of Contents................................................................................................................... 7 Chapter 1: Background and Key Research Questions........................................................... 8 Chapter 2. Matched Comparison School Groups ................................................................ 12 Criteria for Matching NHP Schools with Non-program Schools.................................... 12 Comparison of NHP schools with Matched Schools in 1999.......................................... 13 Chapter 3. Did NHP Schools Receive More Support than Non-NHP Schools? ................. 15 NHP Schools Generally More Advantaged ..................................................................... 16 NHP Schools No Different in Two Areas........................................................................ 21 Chapter 4. Did the NHP Boost Literacy and Numeracy in Grades 3 and 4?....................... 23 Background on the NHP Relative to Grades 3 and 4....................................................... 23 Background of Follow-up Analyses................................................................................ 23 Differences in Mean Achievement .................................................................................. 24 OLS Regression Results .................................................................................................. 26 Chapter 5: Hierarchical Linear Model Analyses of NHP Effects on Grade 6 Achievement28 Analytic Samples............................................................................................................. 28 HLM Models.................................................................................................................... 29 Discussion........................................................................................................................ 34 Conclusion ....................................................................................................................... 36 Chapter 6. What Interventions were More Effective in Boosting Learning? ...................... 37 Analytic Samples and Method......................................................................................... 37 Curious Results on the Effects of the Interventions......................................................... 40 What Accounts for These Results? ................................................................................. 42 Conclusions...................................................................................................................... 43 Chapter 7: Use of National Tests to Monitor Change Over Time ....................................... 44 Grade 1 Readiness Inventory........................................................................................... 44 Grade 3 Diagnostic Test................................................................................................... 45 Grade 4 Literacy Test....................................................................................................... 45 Grade 6 Achievement Test (GSAT) ................................................................................ 46 Conclusion and Recommendations.................................................................................. 50 Chapter 8. Conclusions and Recommendations................................................................... 52 Conclusions...................................................................................................................... 52 Recommendations............................................................................................................ 54 References............................................................................................................................ 56 Annex A. Propensity Score Matching ................................................................................. 58 Annex B. Three Non-NHP Comparison Groups................................................................. 60 Annex C: Jamaica AED School Survey 2005...................................................................... 63 7 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 1: BACKGROUND AND KEY RESEARCH QUESTIONS The New Horizons for Primary Schools Project (NHP) was initiated in school year 1997-98 and fully rolled out in school year 1998-99. The objective of the program was to improve the language arts and mathematics performance of 72 of Jamaica’s poorer performing schools, through a school-based model of intervention. School improvement plans (SIPs) were to be developed for each school, with interventions selected from a menu of ten project interventions (see Box 1) in accordance with each school’s need. Lead institutions were identified for each of the NHP interventions, with an institutional contractor taking responsibility for all but three of the interventions. The MOEYC’s Professional Development Unit (PDU) was responsible for training of resource teachers and the MOEYC in partnership with the National Council on Education (NCE) was responsible for providing governance and leadership training for schools, communities and parents and for offering parent education and training. In the spring of 1999, a diagnostic survey of all NHP schools was undertaken, to assess the schools’ training and other needs (Juarez and Associates, June 1999). Systematic evidence regarding the extent to which various interventions were implemented in the NHP schools is modest, but shows improvement over time. Some evidence comes from evaluations of School Development Plans (SDPs), which are the “focal point of NHP’s approach to school governance” and also are essential in the needs assessment process (Dye et al, 2002, p. 11). One evaluation codified the quality of literacy and numeracy initiatives in SDPs for NHP schools as of November 1999, based on SDPs received from approximately three-quarters (56 of 72) of the NHP schools; of these, the quality of fewer than 20 percent was judged “satisfactory” and the quality of about 40 percent was judged “weak” (Juarez and Associates, December 1999). Only about half the SDPs included a statement of actions that the school would take to reach their specific literacy or numeracy attainment target. The report notes that Box 1: New Horizons Interventions 1. Developing innovative mathematics and literacy programs 2. Providing in-service teacher training in reading and mathematics 3. Providing governance and leadership training for schools, communities, parents 4. Offering parent education and training 5. Facilitating selective nutrition and health programs 6. Providing reading and mathematics materials 7. Establishing computer use in school and training teachers in educational technology 8. Training resource teachers 9. Integrating databases using MIS 10. Improved school management through EMIS 8 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact “very few schools appear to be in a stated position of readiness to deal with literacy and numeracy in their schools” (Juarez and Associates, December 1999). This number was apparently higher a few years later. An analysis of 56 SDPs in 2003 judged all but four of them to be “good” or “very good.” (Summary Evaluation of School Improvement Plans (SIP) of NHP, Spring 2003). This later evaluation, however, noted that half (28 of 56) of the NHP schools for which SIPs were available lacked the desired three-year action plan for implementation of the program. As anticipated, the NHP interventions were not implemented uniformly across all 72 schools; implementation varied across schools in accordance with local needs. For example, only 14 NHP schools received breakfast programs. The intensity of the interventions also varied, with training program duration lasting from a few hours to several days. Table 1.1 summarizes the main features of the implemented program. Table 1.1 Features of NHP as Implemented by 2003 Intervention Implementation 1. Developing innovative mathematics and literacy programs 100s of site visits, deployment of 16 “NHP Associates” to work at classroom level 2. Providing in-service teacher training in reading and mathematics Consolidated with #8 3. Providing governance and leadership training for schools, communities, parents Procurement of governance and Leadership Coordinator and Officers Examination of SDPs Site visits in 60 NHP schools Finalized Manual on Governance and Leadership Training for School Boards and Principals NHP Principals’ Workshops Other training 4. Offering parent education and training National Parenting Conference (1999, 2002) 5. Facilitating selective nutrition and health programs Subsidy of breakfast program in 14 schools Community mobilization to sustain program Teacher training on integrating health and nutrition in teaching of core subjects Nutrition Specialist and community development specialist. 6. Providing supplementary reading and mathematics materials Supplementary materials and equipment distributed to schools 7. Establishing computer use in school and training teachers in educational technology 5 “technology-intensive” NHP schools established Two three-day and one overlapping six-day Educational Technology Workshops held for teachers in 5 NHP schools (2002) One-week consultancy on use of technology for student literacy Consultations with 72 school principals on incorporating technology into SIPs 8. Training resource teachers Trained 180 Math and Literacy Resource Teachers in workshops and in-school training activities 9. Integrating databases using MIS JSAS software 5.0 developed and utilized within NHP schools Support guides and training manuals for 200 non-NHP schools prepared 10. Linking Project Schools to EMIS Network 25 large and medium schools received additional computers (2002) and 140 computers were networked Source: O’Neil, October 2003 9 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Over the period of implementation of the NHP a large number of formative and other evaluations were carried out; nearly 100 were catalogued by the Curriculum and Support Services Unit of the MOEYC (O’Neil 2003). However, none of these studies addressed, in a comprehensive manner, a series of questions posed by USAID: 1. Have NHP schools made achievement gains over the years under review? 2. What factors in the project schools may have affected gains or lack of gains? 3. Is the use of mastery/non-mastery/non-mastery categories on the GSAT masking real gains in student achievement in schools? 4. Is the GSAT the best measure of student performance for the project schools, considering that it is based on the content delivery system of the Old Curriculum? 5. How can valid measures of students’ computational skills in numeracy be assessed for students who are unable to comprehend the language in which most numeracy items are couched in the GSAT examination? 6. How has “social promotion” to grade 6 affected average performance results among students 7. How effective were the indicators used for tracking the results of the NHP and what suggestions could be made for the future? 8. How effective are the methodologies used to collect data? In 2005, the Academy for Educational Development carried out an evaluation of the NHP that addressed many of these questions. However, at the time of the investigation there were several shortcomings with regard to available data. Questions were raised about the adequacy of the GSAT to measure changes in literacy and numeracy of low performing students. Little was known about the inputs received by comparable non￾NHP schools, leaving open the possibility that non-NHP schools had benefited from resources that would have been comparable to those received by schools in the NHP program. This, in turn, could have offset anticipated achievement differences between NHP and matched non-NHP schools. The present report addresses these questions. Chapter 2 describes the process of identifying matched non-NHP schools. Chapter 3 reports the findings of the school survey that collected data from the NHP schools and a set of matched non-NHP schools; it describes the support provided to NHP schools and compares this with support provided to other, matched, non-NHP schools. Chapter 4 extends the analysis presented in Lockheed, Harris, Gammill and Barrow (2004) to examine the effects of NHP on student literacy and numeracy in Grades 3 and 4. Chapter 5 deepens the analysis presented in Lockheed et al (2004) by employing hierarchical linear modeling to assess the impact of NHP on student performance on Grade 6 tests and Chapter 6 explores the question of what factors contribute to increased learning. Chapter 7 analyzes four national tests (Grade 1 Assessment, Grade 3 Diagnostic Grade 4 Literacy and Grade 6 Achievement) and assesses their utility in monitoring change over time and as PMP indicators. Chapter 8 presents our conclusions and recommendations. For this evaluation, the MOEYC provided us access to Grade 3 and Grade 4 data from the school year 2003-2004, to GSAT data from 2005 (in addition to GSAT data 1999- 2004 provided previously), and to selected data from the Annual School Census of 10 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact 2004 (in addition to Annual School Census data 1999-2003 provided previously). In addition, we fielded a survey of the 71 extant NHP schools and a statistically matched comparison group of non-NHP schools to obtain independent evidence regarding the degree to which NHP support had reached the schools and whether non-NHP schools had received comparable support, although from other sources. 11 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 2. MATCHED COMPARISON SCHOOL GROUPS We used propensity score matching (PSM) to identify a set of government schools in Jamaica that were matched with NHP schools in 1999 across a wide range of characteristics, but that did not participate in the program. We identified three different comparison groups, using three different PSM techniques: (a) PSM with replacement, (b) PSM with non-replacement, and (c) PSM with replacement and substitution. Propensity score matching and the three different comparison groups are described in Annexes A and B, respectively. The results do not differ substantially across the three groups. CRITERIA FOR MATCHING NHP SCHOOLS WITH NON-PROGRAM SCHOOLS Of the 72 NHP schools, we were able to match 71 with equivalent schools not participating in the program (Annex B). The number of unique matched schools was 58 for PSM with replacement (i.e., some NHP schools were matched with the same non-NHP schools), 71 for PSM without replacement and 66 with PSM with replacement and substitution (i.e., reasonable substitutions were identified for selected non-NHP schools). To the extent possible, using 1999 school census data available for all government schools in Jamaica, we matched schools on the basis of eight criteria that were initially used to place schools in the program; these were: (a) performing at or below the national mean in language arts and mathematics in the National Assessment Programme, (b) performing at or below the national mean in language arts, mathematics, science and social studies in the National Assessment Programme, (c) geographic location, (d) evidence of Board, or principal and teachers taking action to address the under-achievement of students in the school, (e) active functioning School Board or SCOPE Committee, (f) recipient of grant for Jamaica School Investment fund or civil works in the IDB PIEP project, (g) potential for providing inspiration and leadership in the project, (h) participation in other initiatives complementary to the project. In 1998, 194 schools met these criteria (PIU, December 1998) and were eligible for selection; data related to some of these criteria were included in the 1998 or 1999 School Census for all schools. In addition, we included regional “dummy” variables and indicators for four other characteristics of schools and teachers that we hypothesized were important determinants of student achievement and for which we could obtain data from the School Census: (a) teacher quality, (b) teacher experience, (c) poverty level of school community, and (d) size of school. These twelve characteristics and the data sources for each are summarized in table 2.1. 12 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Table 2.1. Criteria for school selection into New Horizons for Primary Schools program NHP Selection Criteria Indicator from School Census or GSAT Performing at or below the national mean in language arts and mathematics in the National Assessment Programme, School mean GSAT Scores on language arts and mathematics, school year 1998-99 Performing at or below the national mean in language arts, mathematics, science and social studies in the National Assessment Programme, School mean GSAT Scores on language arts, mathematics, science and social studies, school year 1998-99 Geographic location School Census classification of school as rural, remote rural Evidence of Board, or principal and teachers taking action to address the under-achievement of students in the school, None Active functioning School Board or SCOPE Committee, School Census: presence of School Board or SCOPE Committee Recipient of grant for Jamaica School Investment fund or civil works in the IDB PIEP project, None Potential for providing inspiration and leadership in the project, None Participating in other initiatives complementary to the project. None Other school factors Teacher quality in Grades 1-6 School Census: Percent teachers with CXC as highest level of school attainment School Census: Percent master teachers in school Teacher experience in Grades 1-6 School Census: Average number of years experience as a teacher School Census: Average number of years experience in the school School Census: Percent teachers with less than two years experience Poverty level of school community School Census: School breakfast program Size of school School Census: Number of teachers, Grades 1-6 School Census: Number of students, Grades 1-6 School Census: School on Shift PTA School Census: presence of PTA COMPARISON OF NHP SCHOOLS WITH MATCHED SCHOOLS IN 1999 All three groups of non-NHP schools were well-matched with the NHP schools at the outset of the program. Inspection of school, teacher and student characteristics of the NHP and matched non-NHP schools as of 1999 demonstrates the similarity of the four sets of schools (table 2.2). For none of the initial 1999 characteristics, including average student performance on the GSAT1999, are there statistically significant differences between the NHP schools and any of the three comparison groups, other than for inclusion on the list of initially qualified schools. NHP schools do appear to be larger, in terms of both numbers of teachers and numbers of students, but these differences are not statistically significant. 13 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Table 2.2: School and teacher characteristics, NHP schools and PSM comparison groups, 1998-1999 NHP Comparison Schools Schools in 1999 PSM with replacement PSM with non￾replacement PSM with substitution Number of Observations 72 58 71 66 School Characteristics 1999 Size: Enrollment in Grade 1-6 382 298 (-1.2373) 299 (-1.3185) 298 (-1.2843) Size: Number of teachers in Grades 1-6 12.89 10.38 (-1.2986) 10.63 (-1.2491) 10.25 (-1.4062) School shift (percent) 13 10 (-.3796) 9 (-0.7576) 9 (-.6693) Rural location (percent) 39 50 (1.2672) 44 (.6488) 46 (0.8757) Remote rural location (percent) 24 26 (0.294) 27 (.4806) 27 (0.4388) Breakfast program (percent) 18 14 (-.6524) 17 (-.1418) 15 (-.4931) Active PTA (percent) 97 97 (-.2184) 97 (-.0284) 96 (-.5344) Board and/or scope (percent) 88 90 (.3796) 91 (0.7576) 87 (-.1625) On list of initially qualified schools (percent) 100 14 (-21.0494) 19 (-17.6422) 69 (-5.6918) Teachers Characteristics 1999 Qualifications: CXC highest (percent) 27 31 (1.0321) 31 (1.0759) 30 (.7593) Qualifications: Certificate highest (percent) 67 65 (-.6455) 65 (-.7593) 65 (-0.6342) Qualifications: Bachelors degree (percent) 5 4 (-.9632) 4 (-.7250) 5 (-.2038) Qualifications: Masters degree (percent) 39 43 (.3019) 39 (0.0071) 41 (.1243) Experience: Mean years in service at grade 1 to 6 15.25 14.93 (-.2529) 14.79 (-.3959) 14.55 (-.7505) Experience: Mean years in service at grade 1 to 6 in school 11.05 9.94 (-1.1484) 9.78 (-1.4062) 9.9 (-1.2611) Experience: Percent of teachers in school with 2 or less years experience 24 21 (-.7613) 20 (-.9924) 22 (-.6189) Student Achievement (GSAT school means) 1999 Mathematics 1999 28.34 29 (.8233) 28.65 (0.3932) 27.63 (-.9149) Science 1999 22.85 23.36 (.7687) 23.05 (.3198) 22.42 (-.688) Social Studies 1999 33.84 34.48 (.678) 33.93 (0.0932) 33.11 (-.8349) Language 1999 32.87 33.59 (.7037) 33.14 (.2807) 32.34 (-.53) Communications task 1 1999 2.36 2.36 (0.0657) 2.33 (-.3043) 2.32 (-.5746) Communications task 2 1999 1.33 1.28 (-.6758) 1.27 (-.9488) 1.26 (-1.1981) Note: In parentheses is the t-value of the difference between the group mean and the NHP school mean. The remaining chapters in this report utilize these three comparison groups for assessing the effects of the NHP. In addition, schools from the PSM with substitution group were surveyed along with the NHP schools to assess the degree of support received by schools. 14 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 3. DID NHP SCHOOLS RECEIVE MORE SUPPORT THAN NON-NHP SCHOOLS? Given that the NHP was implemented over a period of time – 1999 through 2005 – when other programs were being implemented in primary schools in Jamaica, a key question was whether the schools in the NHP program received more support that other, comparable schools, who might have been receiving assistance through other programs; for example, the PESP provided support to 15 “IT Pilot” schools and 12 “Lighthouse” schools. In October and November, 2005, researchers visited 71 NHP and 67 statistically matched comparison (PSM with substitution) schools and carried out group interviews with the principal and experienced teachers. The survey asked about innovative mathematics and literacy programs at the school, in-service teacher training, governance and leadership training, parent education and training, supplementary reading and math materials, computer use in schools and training teachers about computers, training resource teachers, nutrition and health programs, integrating statistical data bases, and linking project schools with national EMIS. The survey instrument and summary of responses are provided in Annex C. In most areas, NHP schools were more advantaged than matched non-NHP schools. Table 3.1 summarizes the results from this survey according to the ten interventions of the NHP as initially conceived. The results show that NHP schools were substantially more advantaged than the matched non-NHP schools, in all but parent education and health and nutrition. In the remainder of this chapter, we discuss differences between NHP and matched non-NHP schools with respect to the areas in which the NHP schools were more advantaged, aggregated into: (a) support for the Revised Primary Curriculum (RPC), (b) governance and leadership training, (c) in-service teacher training, and (d) computers and IT support. We then discuss the two areas in which NHP and matched non-NHP were similar. Table 3.1: Differences between NHP and Matched non-NHP schools in 10 NHP intervention areas NHP intervention NHP vs. Matched non-NHP schools 1. innovative mathematics and literacy programs NHP more advantaged 2. in-service teacher training NHP more advantaged 3. governance and leadership training NHP more advantaged 4. parent education and training No difference (both low) 5. selective nutrition and health programs No difference 6. supplementary reading and math materials NHP more advantaged (both high) 7. establishing computer use in school and training teachers NHP more advantaged 8. training resource teachers NHP more advantaged 9. integrating databases (JSAS) NHP more advantaged 10. linking project schools with EMIS NHP more advantaged (both low) 15 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact NHP SCHOOLS GENERALLY MORE ADVANTAGED NHP schools reported significantly more support for reading and literacy at the primary level, including preparing School Improvement Plans, and receiving supplementary reading and math materials, governance and leadership training, in-service teacher training, and computers and IT support. Support for mathematics and literacy programs A major objective of the NHP was to help school develop innovative mathematics and literacy programs that would be effective in boosting the performance of lower performing students. However, NHP schools were not the only schools in Jamaica receiving this support from the MOEYC and other donor programs. Some of these initiatives were known variously as professional development centers, best practice demonstration schools, learning centers, lighthouse schools, and pilot schools for the Revised Primary Curriculum or for Instructional Technology. In order to determine if the matched non-NHP schools were benefiting from these types of programs, we asked all schools how other schools might have heard about the school. A higher share of NHP schools (86 percent) than matched non-NHP schools (51 percent) reported that they might be recognized by others, and this difference was statistically significant (p < .001). Many NHP schools mentioned that they were NHP schools, but 72 percent of NHP schools mentioned another reason they might be recognized by others. More NHP schools than matched non-NHP schools reported having school improvement plans; however, concurrently with implementation of the NHP, the MOEYC was implementing other activities to support SIPs and SDPs. As a consequence, nearly all schools -- 94 percent of NHP and 85 percent of matched non-NHP schools -- reported having a School Improvement Plan (SIP) for the purpose of improving reading and math skills of students in the school; this difference was, however, statistically significant ( p < .05). A higher share of NHP schools (84 percent) than matched non-NHP schools (64 percent) reported having specific activities to support the Revised Primary Curriculum (p < .01). However, there were no differences between NHP and matched non-NHP schools in the presence of such specific activities as a Drop Everything and Read program, the presence of Competence Shelter (most schools did not have one) or being a Summer Literacy Camp venue (most schools reported being one). A higher share of NHP schools reported receiving supplemental reading and math materials, and NHP schools reported receiving a greater variety of materials, including books and computer software, all in support of the RPC (figure 3.1). Ninety-three percent of NHP and 79 percent of matched non-NHP schools received supplementary reading materials (p < .05), and 85 percent of NHP and 54 percent of matched non-NHP schools received supplementary math materials (p < .001). About half of both types of schools reported receiving books for student reading (44 percent of NHP and 45 percent of matched non-NHP), but NHP schools received a greater variety of types of books including story 16 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact books, shared reading books, and books for student independent reading (p < .10). In support of math, NHP schools received more types of math materials than matched non￾NHP schools (p<.001). For example, 51 percent of NHP and 1 percent of matched non￾NHP schools received calculators for students, 72 percent of NHP and 27 percent of matched non-NHP schools received manipulative such as blocks; 57 percent of NHP and 34 percent of matched non-NHP schools received math activity books; Approximately 80 percent of both types of schools reported having a local sponsor (such as Kiwanis, Rotary, Lion’s Club, a local church, Adopt-a-School) that provided cash or in￾kind resources to the school (82 percent of NHP and 76 percent of matched-non-NHP schools). Ninety-six percent of NHP schools and 79 percent of matched non-NHP reported that these resources were for the purpose of improving students’ reading and math skills (p < .01), and about the same share reported that the resources directly supported a SIP (93 percent of NHP and 73 percent of matched non-NHP (p < .001) Figure 3.1 School Improvement Plans, Support to Revised Primary Connecticut, Supplemental Materials for Reading and Math (% of schools) 0 10 20 30 40 50 60 70 80 90 100 SIP or SDP Activities to support RPC Reading Materials Math Materials NHP non-NHP Providing governance and leadership training for schools, communities and parents A core feature of the NHP was its emphasis on promoting school improvement through governance and leadership training for school principals, school boards and the community. NHP schools were advantaged in training for leadership (figure 3.2). The training was designed to support school improvement and school development action plans SIPs and SDPs). Twice as many NHP schools (54 percent) reported that they had a trained School Improvement Action group, compared with 26 percent of matched non-NHP schools having such groups, a statistically significant difference (p<.01). 17 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Figure 3.2. Leadership Training in NHP and matched non-NHP schools for Principals, Community Coordinators and School Boards (% of schools reporting) 0 10 20 30 40 50 60 70 80 90 Principal training Comm. Coord. Board Workshops NHP non-NHP Interestingly, about the same share of NHP principals and principals of matched non-NHP schools, over half of both, reported having received training (e.g. 58 percent of NHP and 59 percent of matched non-NHP school principals participated in training provided by Mt. St. Vincent University; 55 percent NHP and 58 percent of comparison principals received training in implementing the RPC). However, the variety of training received by NHP principals was greater than that received by principals of matched non-NHP schools (p < .01). In addition many more NHP schools recruited and trained communication coordinators compared with matched non-NHP schools (86 percent compared with 53 percent, p < .001) and NHP schools were also more likely to receive School Board workshops delivered by the National Council on Education. Providing in-service teacher training in reading and math An innovative aspect of the NHP was the substantial training provided to “support teachers” at the school level. The survey describes support teachers, as follows: “In some primary schools, experienced teachers have been identified as persons who can provide support to other teachers. These support teachers are called Resource Teachers or Literacy/Numeracy/Assessment Coordinators. Some larger schools have Support Teachers that help other teachers on a grade by grade basis, called Grade Coordinators. …We call all of these types of teachers Support Teachers.” The objective of asking the question in this way was to elicit information about the type of support provided in-school by teacher peers. 18 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact A higher share of NHP schools reported having support teachers, and the support teachers in NHP schools received more types of training to help them work with their peers (table 3.2). On average, support teachers in NHP schools received twice as much training as support teachers in the matched non-NHP schools: six days of training during the school year 2004-2005, compared with less than three days of training received by support teachers in matched non-NHP schools (p < .05). Table 3.2. Support Teachers in NHP and matched non-NHP Schools NHP Schools (%) Matched Non-NHP Schools (%) Stat. Sig. (p < ) Presence of support teachers Any support teachers 86 67 .01 Math support teachers 88 44 .001 Literacy support teachers 91 47 .001 Training of support teachers How to teach other teachers 50 19 .001 Cooperative learning 82 36 .001 Peer to peer observation 40 10 .001 The survey asked about a wide range of opportunities for teachers to receive in-service training; respondents from NHP schools identified significantly more training and more different types of training than did respondents from the matched non-NHP schools. Ninety percent of NHP schools reported that in the past five years, teachers of grades 1-6 participated in school-based in-service teacher training provided by outside specialists, whereas only 60 percent of matched non-NHP schools reported such training. Figure 3.3 shows the estimated percent of teachers who received more than ten days of training on the RPC since it was introduced; NHP schools reported a significantly higher share of teachers having received training. In addition NHP schools report a higher amount of training by outside experts, such as provided by the NHP. The differences are statistically significant at the p < .001 level. Figure 3.3 In-service Teacher Training Received by NHP and matched non-NHP schools (% of schools) 0 10 20 30 40 50 60 70 80 90 Gr. 1-3 More than 10 days training Gr. 4-6 More than 10 days training Training by outside specialists NHP non-NHP 19 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact In addition, NHP schools reported receiving a wider variety of types of in-service teacher training for both literacy and math, including clinical supervisory practice, demonstrations by specialists, training in cooperative learning and unannounced observations by Ministry officials (figure 3.4 for literacy). The differences were statistically significant( p < .001). Figure 3.4. In-service Training to Improve Literacy (% of schools reporting) 0 10 20 30 40 50 60 70 80 Clinical Sup. Practice Demos by specialist Coop Learning Obs by MOEYC NHP non-NHP NHP teachers also benefited from on-site training provided by in-school resource teachers in math and literacy. About twice as many NHP schools had resource teachers for math and literacy (88 percent compared with 44 percent in matched non-NHP schools). Moreover, the content of in-service training provided by in-school resource teachers covered more topics of math and literacy in NHP schools than in matched non-NHP schools. For example 71 percent of NHP schools reported training in “making mathematics fun” compared with 25 percent of matched non-NHP schools reporting training on that topic. In general, about twice as many NHP as matched non-NHP schools reported training on each topic. Computers for teaching and administration According to NHP program documents, the final two years of the program placed increased emphasis on providing computers to NHP schools. The survey shows the effects of this effort: 72 percent of NHP and 28 percent of matched non-NHP schools had working computers for students (p < .001); 79 percent of NHP and 38 percent of matched non-NHP schools had a computer resource teacher (p < .001); and 21 percent of NHP schools and 7 20 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact percent of matched non-NHP schools received software as supplemental materials for literacy and math. Figure 3.4. Computers for Teaching and Administration (% of schools) 0 10 20 30 40 50 60 70 80 90 100 Working computer forstuden Receivedsoftware Internet access Computerresourceteache Working computer foradministratio JSAS installed NHP Non-NHP In addition, a substantially higher share of NHP schools had computers that could be used for administrative purposes. For example, 87 percent of NHP and 67 percent of matched non-NHP schools had a computer for school records (p < .05) and virtually all NHP schools had the JSAS software installed. In support of the objective to link project schools with a central EMIS, 41percent of NHP compared with 12 percent of matched non-NHP schools had internet access (p < .001); a remaining large share of NHP schools, however, does not benefit from access to the internet. NHP SCHOOLS NO DIFFERENT IN TWO AREAS In two areas, parent education and training, and health and nutrition programs, we found no difference between NHP schools and matched non-NHP schools. In the latter case, this is undoubtedly due to the relatively small number of 14 NHP schools targeted for such programs. In the case of parent education and training, the survey found inconsistent responses. Parent education and training Neither NHP nor the matched non-NHP schools reported substantial programs of parent education and training, but responses to open- and closed-ended questions were inconsistent. On an open-ended question, more than half (53 percent) of NHP and 70 percent of matched non-NHP schools reported that training was provided for community and parents, with about one-third of schools reported offering some type of “parenting” workshops for parents (38 percent of NHP schools and 33 percent of matched non-NHP 21 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact schools). However, in response to direct questions about programs for parents (“Does this school have afternoon or evening programs for parents to help them understand the Revised Primary Curriculum so they can help their children?”), only a small fraction of either NHP or matched non-NHP schools reported having such programs: nine percent of NHP and 17 percent of matched non-NHP schools. Only about ten percent of both types of schools offered a Family Literacy program (such as Reading Starts with Us). Selective nutrition and health programs The survey asked about the identification of and programs for “special needs” students. Few schools reported professional screening of children by a doctor, nurse or psychologist. Instead, 86 percent of NHP and 93 percent of matched non-NHP schools reported that screening was carried out by classroom teachers. Growth monitoring was reported more by NHP schools (28 percent) than by matched non-NHP schools (14 percent). Programs for “special needs” students were typically school meals programs, with 35 percent of NHP schools and 21 percent of matched non-NHP schools reporting that “special needs” students participated in breakfast programs. In all 64 percent of NHP and 60 percent of non-NHP schools reported some form of meal program for “special needs” students. About 15 percent of both types of schools participated in the PATH program. Over 90 percent of both types of schools reported that a cooked meal was available at the school and abut half of both types of schools reported availability of nutribuns and a drink. NHP schools reported serving meals slightly more regularly than matched non-NHP schools but over 85 percent reported serving meals on a daily basis. Ninety-seven percent of all schools provided free meals to some students. 22 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 4. DID THE NHP BOOST LITERACY AND NUMERACY IN GRADES 3 AND 4? This chapter examines the effects of the New Horizons for Primary Education Program (NHP) on the literacy and numeracy of Grade 3 and Grade 4 students in Jamaica. It concludes that the performance of students in NHP schools is no higher than the performance of students in a statistically matched comparison group of non-NHP schools on either the Grade 3 Diagnostic test or the Grade 4 Literacy test. BACKGROUND ON THE NHP RELATIVE TO GRADES 3 AND 4 The NHP was initiated in 1999 with an objective of improving the literacy and numeracy of primary school students attending some of the poorest schools in Jamaica. Its introduction coincided with the introduction of the Revised Primary Curriculum (RPC) in NHP schools in all grades in the school year 1999-2000 and with the phased introduction of the RPC in all primary schools beginning with Grades 1 and 4 in the school year 2000-2001. By the school year 2003-4, therefore, students in Grades 3 and 4 in all government primary schools, including NHP schools, would have experienced the RPC since the beginning of Grade 1. Primary teachers in all government schools received training about teaching the RPC, but teachers in NHP schools received additional support, as did the schools themselves. This support included both training and provision of school inputs, as summarized in table 1. Did this additional support boost the literacy and numeracy of Grade 3 and 4 students in NHP schools, as compared with students in comparable non-NHP schools? This chapter addresses this question. BACKGROUND OF FOLLOW-UP ANALYSES In an earlier report (Lockheed et al. 2004), AED assessed the impact of the NHP on the literacy and numeracy of Grade 6 students, using the 2003-4 GSAT tests as indicators. This approach had three main shortcomings. First, students completing Grade 6 in the school year 2003-4 had not experienced the RPC throughout primary school, and therefore their teachers may not have participated in the training provided by the NHP, even though their schools may have benefited more broadly from the program. Second, the GSAT is a “high stakes” test, which is designed to identify the highest performers; since the NHP is targeted 23 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact at raising levels of basic literacy and numeracy, the GSAT is not likely to be sensitive to changes at the lower ends of the performance scale. Third, the 2003-4 GSAT did not reflect the curriculum changes of the RPC, so that even if students had been exposed to the revised material the test would not have measured that exposure. Nevertheless, the GSAT was the most widely implemented and nationally reported test of student achievement at the primary level, and was therefore the only indicator available during the life of the project. To address these shortcomings of the previous analysis, it was agreed to examine the effects of the NHP on achievement at Grades 3 and 4, for students whose entire primary school experienced could have benefited from both the RPC and the NHP. For this analysis, 2003-4 results from the Grade 3 Diagnostic Test and the Grade 4 Literacy Test were obtained and analyzed. While not all schools forwarded their results on these tests to the central administration, results were available for more than 85 percent of all schools, including a similar share of NHP schools. DIFFERENCES IN MEAN ACHIEVEMENT We compare NHP schools with non-NHP schools in two different ways. First, we examine the mean scores of schools in 2004, using t-tests. Second, we use a simple OLS regression of school means, where the dependent variables are the school mean scores for the Grade 3 Diagnostic Tests and the average percent of students scoring at non- and near-mastery on the Grade 4 Literacy Tests in 2004, and the NHP program is considered an independent variable, with the school mean GSAT1999 score in math or reading as a control. In the earlier analysis, we used a matched-pair comparison t-test, which is a more powerful test of differences than the t-test for independent samples. In the present analysis, we use the t-test for independent samples, since PSM with replacement means that the same school can serve as a match for more than one NHP school, and therefore it would be counted multiple times when paired on a one-to-one basis with an NHP school The t-tests for independent samples, therefore, is based on different number of cases, depending on which comparison group is used. In all cases, all NHP schools with data are compared to the full number of unique non-NHP schools with data. The number of unique schools in each comparison groups is: 58 for PSM with replacement, 70 for PSM without replacement and 66 for PSM with replacement and substitutions. No Differences between NHP and Non-NHP Schools at Grade 3 or Grade 4 We found no differences in the average performance of students in NHP schools, as compared with any of the three comparison groups on either test. That is, neither Grade 3 Diagnostic Test scores in math and reading nor Grade 4 Literacy Test scores were higher in NHP schools than in comparable non-NHP schools, as of the school year 2003-4. 24 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Grade 3 Diagnostic Test The Grade 3 Diagnostic Test provides teachers with information about student skills in reading and mathematics and is intended to identify children in need of instructional remediation. It comprises tests of reading, math and communications. Reading and math tests have five subtests each (phonics, structure, vocabulary, study skills, reading comprehension for reading and number, measurement, geometry, algebra and statistics for math) with multiple-choice questions, while the communications task is graded on a five￾point “mastery” scale (0-4). Grade 3 students are tested by their teachers towards the end of the school year, in the spring, with tests scored on site and summary sheets of results forwarded to the SAU/MOEYC. Approximately 85 percent of primary schools forwarded school results to the SAU/MOEYC in 2003-4, and 95 percent of Grade 3 students in NHP and matched non-NHP schools that forwarded scores were present for the Grade 3 tests. For only one of the comparison groups (PSM with replacement and substitution) was there a statistically significant difference in favour of the NHP schools, and that for one test only (Communications Task). For all other comparisons, there were no differences in performance on the Grade 3 Diagnostic Test between students in NHP schools and students in the matched non-NHP schools (Table 4.1). Table 4.1: Average Grade 3 Diagnostic Test Scores, NHP schools and PSM comparison groups, 2003-4 NHP Matched non-NHP schools Test Result PSM with Replacement PSM with Non￾Replacement PSM with Replacement and Substitution Average Math Score 31.58 32.02 32.09 31.03 Average Language Score 34.11 34.22 34.34 33.42 Average Communications Task 1.63 1.54 1.57 1.40* N 59/60/60 48/51/51 58/62/62 55/55/55 * p< .05 Grade 4 Literacy Test The Grade 4 Literacy Test is intended to identify children who are at risk of not being literate by the end of Grade 6. The test comprises three sections: word recognition (40 multiple-choice items), reading comprehension (30 multiple-choice items), and communication/writing (scores range from 0-8). All children sit the exam in the spring of grade 4 (pretest), their results are scored locally, and their performance classified as demonstrating mastery (mastery on all three subtests), partial mastery (mastery on 1 or 2 subtests) or non-mastery. Children classified as having non- or partial-mastery may be recommended for Summer Literacy Camp or for repetition of Grade 4. While results are scored locally and forwarded electronically to the MOEYC, only the summary mastery classifications are provided (that is, the raw scores or sub-scores are not submitted). Approximately 85 percent of primary schools forwarded school results to the SAU/MOEYC in 2003-4, and nearly 95 percent of Grade 4 students in NHP and matched non-NHP schools that forwarded scores were present for the Grade 4 tests. Slightly fewer 25 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact students in NHP schools were present for the literacy test (93 percent) as compared with students from the matched non-NHP schools (95 percent) and this difference was statistically significant for the PSM with Replacement and the PSM with Non-replacement groups. Table 4.2: Average Grade 4 Literacy Test Scores, NHP schools and PSM comparison groups, 2003-4 NHP Matched non-NHP schools Test Result (Percent students) PSM with Replacement PSM with Non￾Replacement PSM with Replacement and Substitution Percent Partial Mastery 31.91 31.93 32.28 34.29 Percent Non-Mastery 17.84 16.82 16.82 16.96 Percent for Summer Camp 44.32 43.27 43.53 43.32 N 69 57 69 66 There were no differences in average performance on the Grade 3 Diagnostic Test between students in NHP schools and students any of the three comparison group schools (Table 4.2). Nor were there differences in the share of students recommended for Summer Literacy Camp, with about 44 percent of students recommended in all groups. OLS REGRESSION RESULTS We next examined the effects of NHP on the Grade 3 and Grade 4 tests, using OLS regressions, in which the initial performance of the school, as measured by the GSAT1999, was held constant. We carried out this analysis for all three comparison groups and found no differences with respect to the impact of the NHP. Table 4.3 reports the results from the PSM with replacement. For none of the five regressions is there an effect of the NHP on average achievement at the school level. That is, other things equal, NHP and non-NHP schools did not perform differently in terms of the average literacy or numeracy of students in Grades 3 and 4, as measured by the available tests. Table 4.3: NHP Program Effects on School Mean Grade 3 Diagnostic Test and Grade 4 Literacy Mastery Scores in 2004, OLS regressions Grade 3 Diagnostic Test Scores Grade 4 Literacy Test Math Reading Communi￾cations % Near Mastery % Non-mastery School GSAT1999 Math -.1057 School GSAT1999 Reading -.0482 -.0020 -.0049* School GSAT1999 Comm1 .1939 .0395 .0087 NHP school -.5298 -.1576 .0798 -.0003 .0068 Constant 35.114*** 35.839*** 1.086*** .2913*** .3109*** N 107 111 111 126 126 r2 .0077 .0025 .0250 .0127 .0454 F .4034 .1364 1.383 .5251 1.9323 legend: * p<0.05; ** p<0.01; *** p<0.001 Why should this be the case? The average characteristics of the NHP and non-NHP schools are very similar, and the empirical values of these characteristics as of 1999 at the outset of 26 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact the program did not differ statistically. The NHP supported the schools in the program with additional training and physical inputs not available to other similar schools. Given these differences in advantage documented in Chapter 3, it is difficult to understand the absence of differences between the NHP and matched non-NHP schools in student achievement, as measured by Grade 3 and Grade 4 tests. 27 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 5: HIERARCHICAL LINEAR MODEL ANALYSES OF NHP EFFECTS ON GRADE 6 ACHIEVEMENT This chapter examines the effects of the New Horizons for Primary Schools (NHP) program on student performance as measured by the GSAT. The approach compares the performance of GSAT test-takers from NHP schools with the performance of GSAT test￾takers from matched, non-NHP schools, using a statistical technique (hierarchical linear modeling) that takes into account the nested data structure of students within schools and the variations in numbers of students across schools, which our previous analysis did not (Lockheed et al 2005). On the basis of the analysis detailed below, we conclude that the NHP did not boost achievement as measured by the GSAT, although it may have improved the performance of older test-takers as compared with younger test-takers. We speculate that sharp increases in enrollments in NHP schools may account for the absence of strong program effects. ANALYTIC SAMPLES The present analysis is based on the GSAT test-takers in NHP schools and two different comparison groups, the first estimated through propensity score matching (PSM, without replacement) and the second constructed on the basis of propensity scores but including a large number of schools initially identified as qualified for the NHP but not selected (PSM with substitution).3 PSM was carried out on the basis of baseline 1999 data available for all schools. The analysis compares 71 NHP schools4 with each of the two comparison groups, for two consecutive school years: 2004 and 2005. Students were all those reporting Grade Six Achievement Test (GSAT) scores, for whom gender and age information were also available; 10 students5 in 2004 and 69 students in 2005 lacked age data. A summary of the analytic sample is provided in table 5.1. 3 We also examined the PSM with replacement group; results were no different and are not reported. Details on the methods for identifying these comparison groups are provided in Annexes A and B.. 4 Although 72 schools originally participated in NHP, one school closed. 5 Nine for the PSM with substitution comparison group 28 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Table 5.1: Analytic samples of students and schools NHP and PSM without replacement NHP and PSM with substitution 2004 2005 2004 2005 Students (level 1) 7289 7845 7081 7645 Schools (level 2) 142 141 138 137 Schools were not randomly assigned to participate in the NHP. Rather, several criteria were used to select schools into the program, as detailed in Lockheed et al (2005). Our earlier analysis (Lockheed and Jayasundera 2005) found that empirical measures of the criteria used for selecting schools into the program were generally unrelated to the statistical probability of being in the program. Thus, other non-specified and unobserved criteria are likely to have been used to identify schools qualified for NHP. We compared NHP schools with both comparison groups and found no difference in such major school characteristics as baseline student performance, teacher quality, school size or location. HLM MODELS In this analysis, we examine the effects of the NHP on student outcomes at grade 6, controlling for the two student characteristics for which we have data: gender and age. The NHP was designed to improve literacy and numeracy, and we have therefore selected three outcome measures at grade 6 that are related to these objectives: student scores on the GSAT math test, student scores on the GSAT language test, and student scores on the GSAT Communications Task 1 assessment. The GSAT Mathematics and the GSAT Language tests are both 80-item multiple-choice tests having a maximum score of 80 points. The GSAT Communications Task 1 is an open-ended task that is scored holistically on a rating scale of 1-6. The tests are not the same for 2004 and 2005, and therefore the raw scores should not be compared over time. In particular, the 2005 tests were based on the Revised Primary Curriculum, whereas those in previous years were not. We therefore refer to these tests as GSAT2004 and GSAT2005, to mark the difference. The results are organized as follows. For each of the outcome measures, we present (a) the null or empty model for both comparison groups, and (b) a model that tests for the effect of NHP on the outcome, controlling for student age and gender and for school size, location and the baseline 1999 school mean score on the outcome measure, also for both comparison groups. Our Level 1, or within-school, model nested students within schools and is written as: ij r j Age ij Age j β ij Male j β j β ij Y = + + ( − ) + 0 1 1 where we assume ~ (0, ) for 2 rij N σ n j i = 1,..., students in school j. 29 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact which represents, Yij, grade 6 achievement for student i in school j regressed on gender and age, plus the Level-1 residual variance, σ 2 , remains unexplained after student gender and age has been taken into account. In this model, each student’s age is centered around its school mean, thus controlling for school-to-school differences in age of enrollment and possible repetition. Boys are coded as one and the girls as zero in the gender variable. Hence, the intercept, β0j, is the school-mean grade 6 achievement for girls. We treat the within-school gender and age gaps – the difference between GSAT scores for boys and girls, and for older versus younger students – as random at Level 2 to understand the impact of NHP on these gaps. At Level 2 of the model, we estimate the impact of NHP treatment on the mean grade 6 achievement outcome in school j. We included school-level covariates to account for key differences among schools: school size in 1999, geographical location, and school average 1999 GSAT score. The Level 2 model is written as: j NHPj SchoolSize j SchoolSize Remote j ( ) 0 00 01 02 03 β = γ + γ + γ − + γ j 04 GSAT j GSAT u0 + γ ( 99 − 99) + 1 j = 10 + 11NHPj + u1 j β γ γ 2 j = 20 + 21NHPj + u2 j β γ γ where the mean GSAT intercept for school j is regressed on the school-level covariates, the NHP treatment indicator, and a residual. The within-school GSAT differences by gender and age given by the parameters, β1j and β2j, are specified as random, predicted by the NHP treatment. NHP Effects on GSAT Math Achievement The first result (table 5.2) shows that there is very little variation among schools to be explained; less than 10 percent of the variance in mathematics achievement for either year or either comparison group. There was, however, a statistically significant amount of Level 2 variability that was potentially explainable by NHP program participation and other school-level characteristics. Table 5.2: Multilevel model partitioning variance in Mathematics, by comparison group and year PSM w/o replacement PSM with substitution GSAT2004 GSAT2005 GSAT2004 GSAT2005 Within-school 211.95 350.32 205.71 345.33 Between Schools 16.08 (7 %) 28.00 (7.4 %) 20.64 (9.1%) 29.00 (7.7%) Chi-square 737.86 (p<.001) 804.38 (p<.001) 897.29 (p<000) 875.23 (p<.001) 30 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact The second result (table 5.3) shows a significant gender gap in math achievement, with girls outperforming boys by about 4 points in 2004 and by about 9 points in 2005. Older students perform less well than younger ones, a difference that is statistically significant for both analytic groups in 2005. The gender gap in favor of girls in Jamaica is not surprising, as it has been the subject of a number of studies (see, for example, Bailey 2000). It is also not surprising that older students perform less well than younger ones, as it is probably the older grade 6 students whose past performance has contributed to the grade repetition that is most likely responsible for being older at this level. Table 5.3: Multi-level model predicting Mathematics outcomes, by year and comparison group PSM w/o replacement PSM with substitution GSAT2004 GSAT2005 GSAT2004 GSAT2005 Fixed effect School mean achievement Intercept 30.99**** 43.48**** 30.07**** 41.72**** NHP -0.49 -1.76 0.03 -0.47 Enrollment Gr. 1-6 0.003** 0.004*** 0.004*** 0.005*** Remote school 0.21 1.75 0.89 2.53** School GSAT1999 0.36**** 0.45*** 0.27*** 0.43*** Male intercept -4.38**** -9.06**** -3.90**** -8.90**** Age intercept -0.11 -0.35*** -0.19* -0.33*** NHP on male slope -0.08 0.79 -0.59 0.70 NHP on age slope -0.02 0.24* .005 0.21 * p<.10, ** p< .05, ***P<.01, ****p<.001 The NHP has no statistically significant impact on boosting GSAT math achievement or reducing the gender gap. For the GSAT2005, however, the NHP mitigates the effects of age on achievement. That is, even though older students in all schools perform less well than younger students on the GSAT2005, the difference between older and younger students in NHP schools is less than the difference between older and younger students in the matched non-NHP schools. This effect is statistically significant for the PSM without replacement comparison. Other level-2 control variables are correlated with GSAT math achievement. First, school level performance on the GSAT math in 1999 is positively related to student performance on both the GSAT2004 and the GSAT2005. Second, school size is positively related to math achievement, although the size of the effect is small. For each standard deviation in enrollment in grades 1-6, the average GSAT math score increases by 0.75- 1.50 points, depending on the year and comparison group. This could be due to better teacher qualifications in larger schools, better access to instructional material or more opportunity for site based training of teachers in mathematics. Remoteness is positively related to math achievement, but this association is statistically significant only for the GSAT2005, for the constructed control analytic sample, and amounts to about 13 percent of a standard deviation. 31 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact NHP Effects on GSAT Language Achievement The amount of variance to be explained between schools is also very small for GSAT2004 and GSAT2005 Language scores, although for this outcome, too, there is variability to be explained (table 5.4). Table 5.4: Multilevel model partitioning variance in Language, comparison group and year PSM w/o replacement PSM with substitution GSAT2004 GSAT2005 GSAT2004 GSAT2005 Within-school 225.98 293.70 220.96 289.50 Between Schools 15.96 (6.6 %) 23.81 (7.5 %) 18.28 (7.6%) 25.03 (8.0%) Chi-square 661.17 (p<.001) 791.56 (p<.001) 727.76 (p<.001) 860.86 (p<.001) Both GSAT2004 and GSAT2005 Language scores show a statistically significant and large gender gap favoring girls for both analytic groups. The age effect on language outcomes is statistically significant for both analytic groups and both years, with older students scoring lower than younger students. The effects are modest, however, with scores only about 15 percent of a standard deviation lower for students who are one year (12 months) older than their peers. The effect of the NHP is counter-intuitive, as scores in NHP schools are lower than scores in control schools in three of the four analyses, reaching statistical significance for the PSM control analytic sample for GSAT2005 (table 5.5). The size of the effect is modest, about 13 percent of a standard deviation, but the direction of the effect is not as expected. The NHP has no effect on reducing the gender gap. At the same time, the NHP has a positive effect on reducing the age gap for three of the four comparisons, and in one case – for the PSM with substitution control – the effect is statistically significant. Table 5.5: Multi-level model predicting Language outcomes, by year and comparison group PSM w/o replacement PSM with substitution GSAT2004 GSAT2005 GSAT2004 GSAT2005 Fixed effect School mean achievement Intercept 35.04**** 40.41**** 34.31**** 38.77**** NHP -0.29 -2.33** 0.22 -0.77 Enrollment Gr. 1-6 0.002** 0.004*** 0.003*** 0.004*** Remote school -0.06 2.51** 1.03 2.26* School GSAT1999 0.26*** 0.19** 0.20*** 0.17* Male intercept -6.01**** -9.39**** -6.12**** -8.70**** Age intercept -0.15* -0.26*** -0.26** -0.28*** NHP on male slope -0.48 0.46 -0.40 -0.19 NHP on age slope -0.02 0.17 0.10 0.19* * p<.10, ** p< .05, ***P<.01, ****p<.001 The effects of other level 2 covariates are similar to those for mathematics. Students in larger schools score higher, as do students in remote schools on the GSAT2005. The school 32 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact baseline 1999 score on language is a statistically significant predictor of language scores for both GSAT2004 and GSAT2005. NHP Effects on GSAT Communications Task 1 Achievement The Communications Task I is a measure of basic writing competence. It is scored holistically, following careful rubrics, on a range of 1-6. The lower end of the scale represents no writing competence and the upper end of the scale represents high writing competence at the basic level. Again, there is relatively little between-school variance to explain, although there is about three times as much for GSAT2004 as for GSAT2005 (table 5.6). Table 5.6: Multilevel model partitioning variance in GSAT Communications I Task, by year and comparison group PSM w/o replacement PSM with substitution GSAT2004 GSAT2005 GSAT2004 GSAT2005 Within-school 2.77 2.42 2.76 2.41 Between Schools 0.66 (19.2 %) 0.14 (5.5 %) 0.64 (18.8%) 0.15 (5.9%) Chi-square 1299.33 (p<.001) 535.06 (p<.001) 1246.99 (P<.001) 578.23 (p<.001) Again, the level 1 variables gender and age are predictors of GSAT scores, with substantial gender gaps for both years and analytic samples that are statistically significant for three of the four comparisons (table 5.7). In addition, a statistically significant age gap is observed for the PSM with substitution Control. The NHP has a positive effect on reducing the age gap in this latter group for GSAT2005. The effects of NHP on GSAT Communications Task I are not consistent for GSAT2004 and GSAT2005. For both analytic samples, NHP is associated with statistically significantly higher GSAT2004 scores and statistically significantly lower GSAT2005 scores. On the GSAT2004, the difference amounts to 18 percent of a standard deviation higher than the PSM Control and 20percent of a standard deviation higher than the PSM with substitution Control. On the GSAT2005, the differences are 26 percent and 13 percent of a standard deviation lower, respectively, than the two comparison groups. We are unable to explain this sign switching. Neither school size nor remoteness were consistently associated with GSAT Communications Task I scores. For GSAT2004, remote schools did have higher scores. Since this test is scored holistically, it is possible that standards were higher in urban and peri-urban schools that year, resulting in lower scores being awarded to students in these schools, and that in subsequent years corrections were made. 33 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Table 5.7: Multi-level model predicting Communications Task I outcomes, by comparison group and year PSM w/o replacement PSM with substitution GSAT2004 GSAT2005 GSAT2004 GSAT2005 Fixed effect School mean achievement Intercept 3.15**** 1.64**** 3.11**** 1.99**** NHP 0.28* -0.16* 0.33** -0.24** Enrollment Gr. 1-6 -.000 0.00 -0.000 0.000* Remote school 0.40** 0.13 0.27 0.12 School GSAT1999 0.23 0.22** -0.15 -0.19* Male intercept -0.82*** -0.06 -0.84**** -0.76**** Age intercept -0.01 -0.01 -0.03** -0.03**** NHP on male slope -0.06 -0.05 -0.03 0.12 NHP on age slope 0.001 0.01 0.01 0.01* * p<.10, ** p< .05, ***P<.01, ****p<.001 DISCUSSION The remarkable negative impact of the NHP on achievement on the GSAT2005 merits discussion. We observe that this negative impact was not statistically significant for all content areas, but we do observe sign switching rather consistently between GSAT2004 and GSAT2005. We identified three main explanations for these differences: (a) the scoring of the tests could have changed, (b) the population of those taking the GSAT could have changed, and (c) the population of those taking the GSAT in NHP schools could have included students who did not benefit from the program for all of primary school. We will consider each of these explanations. First, the scoring rubrics for the tests could have changed between GSAT2004 and GSAT2005. There is little evidence that this is the case, however. From all reports, considerable attention is paid to scoring the Math and Language tests objectively and using consistent guidelines for the holistic scoring of the Communications test. Although our analyses show that the GSAT2004 tests were different from the GSAT 2005, there is no reason to suspect this change would have a differential impact in NHP versus matched non￾NHP schools, and indeed we found no evidence for this, as we discuss in Chapter 7. Second, the population of test takers could have changed significantly between 2004 and 2005. The total number of GSAT test takers in 2005 (49196 for mathematics and 49157 for language) exceeded the total for 2004 (47306 for mathematics and 47306 for language) by approximately 1800 students, or approximately 2 students per school on average. However, the number of GSAT test takers in NHP schools increased relatively more, by about 3.5 per school, from a total of 4028 for mathematics and 4034 in language in 2004 to 4265 in mathematics and 4271 in language in 2005. We hypothesize that the changing composition of the test-taking population of students accounts for some of the differential NHP effects, as follows. In 2004, many principals held back some students from taking the GSAT, both 34 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact to provide them another opportunity to take the test and to improve the average scores of the school. This practice, however, meant that the principal was obliged to take on the responsibility of placing the students who did not take the GSAT2004 into a suitable post￾primary educational environment. In 2005, to avoid the additional burden of placing students, all or nearly all Grade 6 students were obliged to take the GSAT2005. This meant that some lower performing students, similar to those who had been excluded from the test the previous year, took the GSAT in 2005. But why would this effect be more pronounced in NHP schools, boosting their average GSAT2004 scores, but lowering their GSAT2005 scores? We hypothesized that the additional attention given to NHP schools created an incentive for the principal to optimize the performance of students in the school in 2004, within the range permissible. Therefore, in NHP schools more students were excluded from the GSAT2004 than were excluded from the matched non-NHP schools, whose identities were not similarly publicly known. These excluded students took the GSAT2005, with a consequent depressing effect on scores in NHP schools. The number of test-takers in NHP schools is approximately 30 percent greater than the number of test takers in the matched non-NHP schools. However, the number of students in NHP schools in 2004 is nearly 40 percent greater than the number of students in the matched non-NHP schools, which means that this cannot be the explanation (table 5.8). Table 5.8. Average number of GSAT2005 test takers per school, NHP and Matched non-NHP schools School type Test takers per school 2005 NHP as % of comparison Students per school 2004 NHP as % of comparison NHP 60 -- 410 PSM without replacement 46 130 298 138 PSM with replacement 47 128 293 140 PSM with substitution 46 130 300 137 Finally, we hypothesized that the population of the NHP schools, in particular, could have changed over time. We explored three ways in which this might have occurred: an overall increase in the number of students enrolled in NHP schools, an increase in the share of (lower performing) boys in NHP schools, and an increase in the average age of students in NHP schools. An increase in the number of students enrolled in NHP schools would suggest that parental choice was playing a role in determining which children attended these schools. In Jamaica, where multiple schools are available, parents are allowed to place their children in the school they perceive as more desirable. If parents were moving students into NHP schools throughout the primary cycle (instead of only at Grade 1), then many students sitting the GSAT could have experienced the NHP only partially. We do not have data that track individual children, but enrollment data from the School Census suggests that parents are selectively moving their children into NHP schools, as the average enrollment growth in these schools was 5 percent from 1999 to 2003, compared with no or even a negative increase in the matched non-NHP schools (table 5.9). We also note through the JASS survey that principals of NHP schools identified significantly more ways the public could have heard about their schools, as compared with non-NHP schools (see Chapter 3 and Lockheed and Jayasundera 2005). 35 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Table 5.9. Average enrollment in Grades 1-6, in NHP and Matched non-NHP schools, 1999 and 2003 School type Observations 1999 2003 % increase (decrease) NHP 72 382 401 5 ** PSM without replacement 70 299 298 0 PSM with replacement 58 298 293 (2) PSM with substitution 66 297 300 1 t = 1.75, **p<.05 We speculated that the increase in enrollments in NHP schools could be due to their greater effectiveness in retaining boys or older students, but available data do not support this hypothesis. Neither the share of boys taking the GSAT nor the average age of GSAT test￾takers differed between 2004 and 2005 (table 5.10). There is one exception, and that is the share of boys in the PSM without replacement control schools was lower in 2004 and slightly higher in 2005 than it was in the other schools. Table 5.10. Percent male and average age in months of GSAT candidates, in NHP and Matched non￾NHP schools, 2004 and 2005 Percent male test-takers Average age of test-takers (months) School type GSAT2004 GSAT2005 GSAT2004 GSAT2005 NHP 51.4 52.0 144.8 144.6 PSM without replacement 48.5 53.8 144.6 144.5 PSM with replacement 50.0 52.0 144.8 144.4 PSM with substitution 51.1 51.4 144.8 144.6 CONCLUSION Students in NHP schools performed no higher on the GSAT than students in matched non￾NHP schools. This absence of NHP effect was consistent across three subjects (math, language and communications) and for both GSAT2004 and GSAT2005. The age gap – that is the poorer GSAT performance of older candidates – was mitigated by the NHP in several cases. That is, on some GSAT2004 and GSAT2005 tests, the performance of older students was enhanced by being in an NHP school. Enrollments increased significantly in NHP schools, while falling or remaining the same in matched non-NHP schools. We speculate that increasing enrollments in NHP schools, caused by parental choice, may have resulted in some students having been exposed to a shorter version of the program than intended, with a consequent reduction in the possibility for impact. 36 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 6. WHAT INTERVENTIONS WERE MORE EFFECTIVE IN BOOSTING LEARNING? Although the NHP as a whole did not appear to boost literacy and numeracy as measured at the school level for Grades 3 and 4 or at the individual level as measured by the GSAT, some of the interventions provided by the NHP may have been effective in improving student performance in weaker schools, such as those participating in the NHP. This chapter examines the effects of NHP-type interventions on improved learning across both NHP and matched non-NHP schools. The results are not as expected, and suggest considerable endogeneity with respect to the inputs and outcomes. ANALYTIC SAMPLES AND METHOD The analysis in this chapter is based on the school average performance on the Grade 4 Literacy test and the GSAT2005, matched with school-level responses on the JASS, for the 71 NHP and 67 statistically matched non-NHP schools for which JASS data were available. We developed one or more indicators representing each of the NHP interventions, based on responses to the JASS. A list of interventions and the indicator intended to capture its degree of implementation, in both NHP and non-NHP schools, is provided in Table 6.1. Table 6.1 Intervention variables, definitions and measures NHP Intervention Definition of variable Measure 1. Innovative math & literacy Q12 school has SIP Did this school develop a School Improvement Plan or School Development Plan for the purpose of improving the reading and math skills of students at the school? Yes = 1 2. In-service teacher training Q22 Gr. 1-3 teacher training About what percent of the current teachers in Grades 1-3 have received more than 10 days of training related to the revised curriculum since it was introduced in this school? Percent Q23 Gr. 4-6 teacher training About what percent of the current teachers in Grades 4-6 have received more than 10 days of training related to the revised curriculum since it was introduced in this school? Percent Q43 clinical supervision - literacy What types of school-based inservice training for literacy in Grades 1-6 have outside specialists provided to regular teachers in this school? __Clinical Supervisory Practice Yes = 1 Q48 training on reading diagnostics With which specific topics in literacy has the resource teacher helped other teachers in this school? __Use of Diagnostic Teaching of Reading Yes = 1 Q41 teacher training by outsiders In the past 5 years, have regular teachers of Grades 1-6 in this school participated in any school-based inservice teacher training provided by outside specialists? Yes = 1 Q42 clinical supervision - math What types of school-based inservice training for mathematics in Grades 1-6 have outside specialists provided to regular teachers in this school? Yes = 1 37 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact NHP Intervention Definition of variable Measure __Clinical Supervisory Practice Q45 training on making math fun With which specific topics in mathematics has the resource teacher helped other teachers in this school? __Making Mathematics Fun Yes = 1 3. Governance, leadership Q37 SIP training What type of training did the principal who served the school during the school year 2003-2004 participate in? __Training in School Improvement Planning Yes = 1 Q37 Community mobilization wkshp. __Workshop on community mobilization Yes = 1 Q37 Principal peer learning __Peer learning with other principals Yes = 1 Q15 Local sponsor In the past 5 years, has any local sponsor (such as Kiwanis, Rotary, Lion’s Club, a local church, Adopt-a-School) helped with resources (cash or in-kind), either regularly or through a special project, for any purpose? Yes = 1 4. Parent training Q53 governance What types of training have been offered to parents and other community members associated with this school, over the past 5 years? __Training in school governance and leadership Yes = 1 Q53 parenting __ Training in parenting Yes = 1 5. Nutrition & health Q62 free meal for any students Does this school provide a free meal (either breakfast or lunch) for any students? Yes = 1 Q63 percent students with free meals Percent of students qualified to receive a free mean Percent 6. Reading & math materials Q30 reading materials During the past 5 years, has this school received any supplementary reading materials for literacy? Yes = 1 Q31 number of materials What sorts of supplementary reading materials did this school receive? Number of materials Q34 math materials During the past 5 years, has this school received any supplementary mathematics materials? Yes = 1 Q35 number of math materials What sorts of supplementary math materials did this school receive? Number of materials 7. Computers Q65 computers for student use Does this school have computers for student use? Yes = 1 Q66 internet access Does this school have internet access? Yes = 1 Q69 computers for administration Is there a functioning computer available for school records and other administrative purposes? Yes = 1 8. Resource teachers Q40 training for math During the last school year, 2004-2005 including the summer, about how many days of training did the math Support Teacher receive? Number of days Q40 training for reading During the last school year, 2004-2005 including the summer, about how many days of training did the reading Support Teacher receive? Number of days 9. & 10 EMIS & Data Bases Q70 JSAS installed Q70. Has the Jamaica School Administrative System (JSAS) software been installed in a computer at this school? Yes = 1 We used ordinary least squares (OLS) multivariate regressions and estimated the effect of the interventions on five outcome variables: Grade 4 Literacy non-mastery, Grade 4 Literacy less-than-mastery (that is, non-mastery plus partial mastery), GSAT2005 mathematics, GSAT2005 language and GSAT2005 communications I scores. The two Grade 4 measures are the percent of students in the school attaining non-mastery and the percent of students in the school attaining less-than-mastery on the 2004 test administration. The three GSAT2005 measures are school average scores on each test. The models vary with respect to the outcome measure, and include as predictors only those inputs directly related to the outcome. Thus, inputs related to literacy are included in the 38 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact literacy and language models and inputs related to math are included in the math model. Specifically, we include “supplementary reading materials” and teacher training for literacy and reading in the literacy, language and communications regressions, but not in the math regression. We include “supplementary math materials” and teacher training for math in the math regression but not in the literacy, language or communications regression. In all regressions, we control for the school’s geographical location (rural or urban) and for the GSAT2005 regressions we control for the school average score on the GSAT1999, for each test separately. The results are presented in columns 2-6 of table 6.2. Table 6.2 Factors affecting school average literacy and numeracy scores, in NHP and matched non￾NHP schools, controlling for urban rural and 1999 test scores (GSAT models only) Grade 4 Literacy GSAT 2005 NHP Intervention Non￾mastery Less than Mastery Math Language Comm1 1. Innovative math & literacy Q12 School has SIP -0.083 -0.167 0.875 1.886 -0.015 2. In-service teacher training Q22 Gr. 1-3 teacher training -0.001 0.00 -- -- -- Q23 Gr. 4-6 teacher training -- -- -0.029* -0.044*** -0.003 Q43 clinical supervision -literacy 0.035 0.008 -- -0.157 -0.177* Q48 training on reading diagnostics -0.035 -0.036 -- -0.165 0.103 Q41 teacher training by outsiders 0.021 0.029 1.487 -0.256 0.021 Q42 clinical supervision - math -- -- -0.311 -- -- Q45 training on making math fun -- -- -1.584 -- -- 3. Governance, Leadership Q37 SIP training -0.012 -0.023 1.3 0.72 0.097 Q37 Community mobilization wkshp. 0.025 0.07 -1.275 -0.301 -0.082 Q37 Principal peer learning 0.024 0.011 -2.67* -2.635** -0.048 Q15 Local sponsor 0.025 -0.002 -2.319 -2.626* -0.1 4. Parent training Q53 governance -0.008 -0.019 2.804 2.448 0.053 Q53 parenting -0.011 -0.047 -0.079 0.026 -0.085 5. Nutrition & Health Q62 free meal for any students 0.031 -0.296 6.761 7.303 0.58 Q63 percent students with free meals -0.002 -0.003 -0.293 -0.131 -0.011 6. Reading & math material Q30 reading materials -0.035 -0.006 -- 1.87 0.348 Q31 number of materials -0.024** -0.03* -- 0.528 0.024 Q34 math materials -- -- 1.767 -- -- Q35 number of math materials -- -- 0.35 -- -- 7. Computers Q65 computers for student use 0.018 0.007 2.447* 1.312 0.06 Q66 internet access -0.019 -0.042 3.641** 4.325*** 0.22 Q69 computers for administration -0.022 -0.116* 5.086** 4.578** 0.411* 8. Resource teachers Q40 training for math -- --- 0.202 -- -- Q40 training for reading -0.005* -0.004 -- 0.322*** 0.02 9. & 10 EMIS & Data Bases Q70 JSAS installed 0.047 0.078 -2.595 -1.943 -0.259 Control variables GSAT 1999 Score -- -- 0.367*** 0.143 0.145 Urban 0.036 0.083 -1.119 -0.585 -0.146 Constant 0.305 1.152*** 17.599** 19.064*** 0.32 Observations 118 118 116 116 116 Adj. R-squared 0.119 0.01 0.195 0.219 0.028 Legend: * p<.10; ** p<.05; *** p<.01 Note: (--) indicate variables not included in model. 39 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CURIOUS RESULTS ON THE EFFECTS OF THE INTERVENTIONS Each regression model includes at least 20 independent variables; with a 5 percent level of statistical significance, we would expect to find by chance at least one “statistically significant” variable in each regression, and with a 10 percent level of statistical significance we would expect to find at least two “significant” variables. To avoid dwelling on spurious findings, therefore, the discussion of results will focus on those variables that show a consistent pattern of statistical significance across the outcome measures. All regression results include some variables with coefficients whose signs are in the opposite direction from that expected, which requires further exploration. Grade 4 Literacy The Grade 4 non-mastery and less-than-mastery regression models explain very little variance in these measures, with adjusted R-squares of .12 and .01 respectively (Table 6.2 columns 1 and 2). Only two variables were statistically significant in the model predicting the percent of students in the school scoring non-mastery; these were the number of types of supplementary reading materials received by the school and resource teacher training for reading. The simple correlations between these measures and Grade 4 non￾mastery was -.28 and -.12, showing that schools with more types of supplemental reading materials and more resource teacher training also had smaller shares of students scoring at the non-mastery level on the Grade 4 Literacy test. At the margin, for each additional type of reading material received by the school, the percent of students in the school scoring at the non-mastery level dropped by nearly 2 percentage points; each additional day of resource teacher training lowered the share even further. Two variables were statistically significant (at the 10 percent level of significance) in the model predicting the percent of students in the school scoring less-than-mastery; these were supplementary reading materials and the availability of a functioning computer for school records. The simple correlations between these measures and less-than-mastery were -.17 and -.07, respectively. At the margin, each additional type of reading material was associated with a 3 percent decline in the number of students performing below mastery level, while schools with a functioning computer for school records had nearly 12 percent fewer students scoring at the less-than-mastery level. Given that these variables could have emerged as statistically significant by chance, these results should not be over￾interpreted. GSAT2005 The GSAT mathematics and language regression models are more robust, with adjusted R￾squares of .19 and .22 respectively (Table 6.2 columns 3 and 4). For GSAT2005 mathematics, controlling for GSAT1999 math, five variables are statistically significant in the model predicting the school average scores, two of them, however, have effects in the direction opposite to that expected. These are teacher training and principal peer learning, 40 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact with simple correlations with GSAT2005 math of -.06 and -.08, respectively. At the margin, for each additional percent of teachers in Grades 4-6 who received more than 10 days of training in the revised curriculum, GSAT2005 school average math scores declined by .03 points, which is not meaningful. Also at the margin, schools whose principals participated in peer learning scored 2.8 points lower on the GSAT2005 math test, or approximately 15 percent of the GSAT2005 standard deviation, which is meaningful. Computers and internet access appear to boost math achievement, with simple correlations with GSAT2005 math of .17 and .29, respectively. At the margin, schools with functioning computers for student use achieve GSAT2005 math scores 2.4 points higher than those without computers. In addition schools with internet access achieve GSAT2005 math scores 3.6 points higher than those without internet access. Finally schools with a functioning computer for administrative purposes also score higher – by over 5 points. Thus, we estimate that schools with all three computer inputs would score over 11 points higher on the GSAT2005 math test, for an effect size of more than 60 percent of the standard deviation. This is both substantial and meaningful. Six variables are statistically significant for boosting GSAT2005 language scores, with three having effects in the opposite direction to the expected. These are teacher training, principal peer learning, and local sponsorship, with simple negative correlations of -.12, - .12 and -.04, respectively. At the margin, for each additional percent of teachers in Grades 4-6 who received more than 10 days of training in the revised curriculum, GSAT2005 school language scores declined by .04 points, which is not meaningful. Also at the margin, schools whose principals participated in peer learning scored 2.8 points lower on the GSAT2005 language test, also approximately 15 percent of the GSAT2005 standard deviation, which is meaningful. Finally, also at the margin, schools having local sponsors that provided resources scored 2.6 points lower on the GSAT2005 language test, also a meaningful difference. Three other variables, however, are associated with higher scores. These are resource teacher training for reading and two of the computer variables discussed above. At the margin, each additional day training received by reading resource teachers boosted GSAT2005 language scores about one-third of a point. And the effects for having computers and internet access are similar to those for math, albeit slightly lower in their combined effect. For the GSAT2005 Communications I test, two variables were statistically significant (at the .10 level), with one in the opposite direction from that expected. Schools reporting more teacher training had lower average scores, although the difference is not meaningful. And schools with computers for administrative purposes scored higher, by about .41 points. Table 6.3 presents the regression coefficients for the six variables that are statistically significant in more than one of the five regression models. Supplementary reading materials, training of reading resource teachers, computers for administrative purposes and internet access are associated with higher scores at the margin, while teacher training and 41 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact principal peer learning are associated with lower GSAT2005 scores at the margin. It appears that certain types of training and provision of resources are helpful, while other types of training are not helpful. What explains these differences? Table 6.3 School input factors related to literacy and numeracy in two or more models Gr. 4 Non￾Mastery (percent students) Gr. 4 Less￾than-Mastery (percent students) GSAT2005 Mathematics (school average) GSAT2005 Language (school average) GSAT2005 Communications I (school average) Teachers in Grades 4-6 who received more than 10 days of training in the revised curriculum -0.029 -0.044 -0.003 Principals participated in peer learning -2.670 -2.635 Resource teacher training in reading -.005 .322 Number of different supplementary reading materials -.024 -.03 Functioning computer for administrative purposes -.12 5.086 4.578 0.411 Internet access 3.641 3.325 Source: Table 6.2 WHAT ACCOUNTS FOR THESE RESULTS? The most plausible explanation for the curious results described in the previous section is endogeneity. That is, schools received resources, from whatever provider, in response to need for intervention, as indicated by lower student performance on the Grade 4 Literacy and GSAT tests. Teachers in poorly performing schools received comparatively more in￾service training, possibly because they teachers in these schools were themselves less well trained, or because the students presented more challenges to the teachers, or because supervisors judged that additional training was needed for other reasons. Similarly, principals of schools where students’ performance was lower may have sought out peer learning opportunities with other principals of similar schools, in an effort to deal with these common challenges. We interpret this to mean that the service providers recognized the need for in-service training in weaker schools, and ensured that these schools received comparatively more of this input. We find a consistent positive effect for computers on achievement. While it is possible that computers and internet access actually helped boost student learning, it is also possible that computer resources were provided to better performing schools because they also had better leadership and initiative. Under the NHP program, for example, schools needed to request computers and demonstrate that they had the capacity to retain them in a secure environment. The schools also needed to have a regular source of electricity. Thus, better schools – including those with electricity and school-level initiative – were more likely to 42 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact receive computers. We included initial student performance as measured by the GSAT1999 in the regression estimates to control for school quality at the outset, but these statistical controls are unlikely to have captured all the quality dimensions of the school. Other, unmeasured “school quality” factors may have affected which type of services the schools received. Specifically, we have no independent measure of instructional leadership at the school that could help explain why some schools, but not others, received computers. We also find a positive effect for reading resource teacher training and variety of supplementary reading materials. Endogeneity does not easily explain these results, unless obtaining these materials and services is also related to leadership at the school level. Resource teachers at the school level seem to help other teacher improve their skills, so that fewer students score at the non-mastery level on the Grade 4 Literacy test and students achieve higher language scores on the GSAT. Supplemental reading materials also reduce the share of children scoring below mastery level on the Grade 4 Literacy test, possibly being used by classroom teachers with support from trained reading resource teachers. CONCLUSIONS In Chapter 3 we noted that the NHP schools were generally more advantaged with respect to NHP type of inputs than were the comparison schools. In this chapter we find that these advantages did not in general translate into higher scores on tests of literacy and numeracy. However, inputs of teacher and principal training in the schools appear to have been targeted at schools in need of such interventions. With respect to computers, resource teacher training and supplementary reading materials, it is plausible that schools with better leadership were more active in securing these resources. 43 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 7: USE OF NATIONAL TESTS TO MONITOR CHANGE OVER TIME This chapter assesses the utility of four Jamaican MOEYC tests as indicators for monitoring change over time, with particular reference to USAID PMP indicators. These are: the Grade 1 Readiness Inventory, the Grade 3 Diagnostic Test, the Grade 4 Literacy Test, and the Grade Six Achievement Test. We conclude that due to anticipated MOEYC policy changes in test administration dates and content6 , it is unlikely that USAID will be able to rely on a year-to-year criterion-based impact analysis. We recommend that USAID monitor project impact by comparing the performance of project schools relative to an empirically derived comparison group of schools, the approach used in the Impact of New Horizons for Primary Schools on Literacy and Numeracy in Jamaica, 1999-2004 (Lockheed, Harris, Gammill and Barrow 2005). GRADE 1 READINESS INVENTORY The projected utility of the Grade 1 Readiness Inventory depends on the outcomes of pending MOEYC decisions regarding the timing and administration of the assessment. If the Inventory remains a low stakes test administered by grade 1 teachers, it could provide a useful baseline for assessing student entry-level skills and assuring the comparability of project and comparison schools. If the MOEYC decides to change current policy and the Inventory administration is conducted before school begins by preschool teachers, there will be pressure for the test to become high stakes; administration quality and reporting response rate may be jeopardized, the need for annual test development will be costly and require equating in order for results to be comparable from year to year; during this transition, year-to-year results will be compromised. Regardless of the MOEYC decision, one limitation of the current procedures is that only mastery level information is available for analysis. Actual scores on each subtest would provide more precise information and a more sensitive assessment of student skills. Actual student scores should be collected from project and comparison schools. With this information, the indicator would refer to average scores on each the 4 subtests and overall score (Visual Motor, Visual Perception, Auditory Perception, and Number/Letter Knowledge, Total). 6 Refer to Technical Support Provided to MOEYC on Pending Decisions for Student Assessment in the October, 2005 status report for a more complete description and analysis of the pending decisions. 44 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact GRADE 3 DIAGNOSTIC TEST In the absence of the proposed MOEYC changes, average subtotal scores in Language Arts, Mathematics and Communication on the Grade 3 test would appear to be good indicators of learning. The test is a low stakes test. The current reporting form includes raw subtest scores, there is a relatively high response rate (85%), and the test blueprint has been revised to reflect the RPC. It should be noted that alignment with the RPC is new and data from past years may not be comparable. In addition, caution should be used in interpreting the math subtests without first taking into account reading proficiency (See Note at end of chapter1). If a change in the timing of the test occurs, the utility of the Grade 3 exam as a PMP indicator will depend upon when the change occurs and how it is implemented. Unless the MOEYC administers the test to two cohorts in the same calendar year, one cohort will not have data. In addition, the comparability of the year-to-year scores will be affected. It is likely that the test content/difficulty would be adjusted and, even if the test remained the same, it is not equivalent to compare average scores from children who took the test at the end of Grade 3 with scores of those who took the test at the beginning of the year. To support its utility as a PMP indicator and to enhance the test interpretation, post-testing scaling of Grade 3 test results is recommended to assure the comparability of year-to-year scores. To do this, the MOEYC would need to systematically collect item data from a sample of schools. External technical support will be needed on a short-term basis. GRADE 4 LITERACY TEST The Grade 4 Literacy could be a useful indicator for monitoring learning of the youth targeted by Strategic Objective 12. It is designed to identify the lowest performing students in relation to stable literacy criteria. The return rate is high. However, the utility of the Grade 4 Literacy exam as an indicator is limited because the reporting sheet only includes mastery information. No subscores or total scores are provided. Students are categorized as mastery/non-mastery in each of the three areas and then given an overall rating of Mastery (mastery on all three areas), Almost Mastery (mastery on 1 or 2 areas), or Non Mastery (non-mastery on all 3 areas). This lack of precision in what is reported significantly limits the sensitivity of the Grade 4 test for evaluation and monitoring purposes. For example, Mastery in word recognition requires at least 75% correct. On the reporting form, a student who had none correct is indistinguishable from a student with 74% correct. In addition, Grade 4 repeaters are not identifiable. In order to use the Grade 4 Literacy test data effectively, USAID would need to support collection of subscale and repeater data. In the short term it is likely that this will need to be 45 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact done on a school-by-school basis for all project and comparison schools. In the long term, it is recommended that resources be made available to support the revision of the score￾reporting sheet to include actual student scores on each of the subtests and repeater information. If the subscore and repeater data are available, average subscores for non￾repeaters (and, if sufficient numbers, average subscores of repeaters) could serve as useful indicators. GRADE 6 ACHIEVEMENT TEST (GSAT) The Grade Six Achievement Test (GSAT) is a high-stakes, curriculum-based exam administered nationally during sixth grade to determine students’ secondary school placement. In the past, average performance on the GSAT has been used by USAID as one of the indicators for monitoring learning. In 1999, the GSAT replaced the Common Entrance Exam as the “gate-keeper” for entry to all secondary school schools including the elite schools; currently, the GSAT “places” virtually all students sitting the exam into some secondary program. The most demanding placement decisions, however, occur at the top of the scale, to distinguish among high scoring students for places into the few prestigious high schools. Consequently, the test has tended to focus on measuring student performance with greater precision at the top of the scale. In one year, additional harder items were added to the GSAT to assure accuracy of decisions affecting high scorers. Effective with the GSAT2005, the test blueprint and content are aligned with the Revised Primary Curriculum (RPC). Test structure and subject areas covered remain the same as earlier versions of the GSAT: mathematics (80 multiple choice items), language (80 multiple choice items), science (60 multiple choice items), social studies (80 multiple choice items), and communication (2 writing tasks). According to test developers from the Student Assessment Unit, the primary differences between the content of past GSAT exams and GSAT2005 are in science and social studies where new topics have been added or rearranged. GSAT as a Monitoring Indicator For the GSAT to be an effective indicator for monitoring changes in learning from one year to the next, it must meet the following criteria: (1) the test content must be stable, (2) the test must be adequately precise for students of various ability levels including low performers, and (3) the scores must be equated or anchored to a common scale. Stability of test content: In recent years the MOEYC has worked to achieve alignment between the RPC and the rest of the educational system. The New Horizons Project included support for the RPC. Teacher training colleges received support to align their 46 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact curriculum and teacher certification process to the RPC. Similarly, the Student Assessment Unit has worked to revise test specifications to the RPC and to develop items that support the revised specifications. Since the test blueprint for the GSAT has been revised to reflect the RPC, it seems likely that the test content specifications will remain fairly stable for the near future. Measurement precision for students of various ability levels: Several strategies were used to evaluate how well the GSAT2004 and GSAT2005 tests of mathematics and language functioned for students of different ability levels. Since USAID interventions are typically targeted at raising the math and literacy achievement of lower performing students, USAID monitoring indicators need to be sensitive to small changes in the performance of low scorers as well as in changes overall. As a first step, the score distributions for GSAT2004 and GSAT2005 were compared separately for math and language tests, for all test-takers in Jamaica. Second, item analyses were conducted to identify those items that functioned better for low ability students. Third, these “easier” items were aggregated into “new” math and language subscores and reanalyzed. A comparison of GSAT2004 scores with GSAT 2005 scores indicates that the test changed dramatically between the two years. Across all test takers in Jamaica, the average score for math jumped over 11 points while the average score for language jumped 4 points. Figures 7.1-7.4 illustrate the score distributions for GSAT2004 and GSAT2005 for mathematics and language. Each bar in the figure indicates the number of students scoring within the score range specified on the horizontal axis. As is evident from the figures, the GSAT2004 results were positively skewed. Low performers tended to be lumped together within a relatively small range of scores whereas high performers were spread out over a wider range of scores (i.e., there was better discrimination between high scorers). Score distributions for GSAT2005 are much more evenly distributed across the scale. While there are still large numbers of students scoring at or just above the change level (students who guess on all of the 80 multiple choice items should get a score of about 20 or about ¼ of the items correct by chance), numbers of students scoring at other points on the scale are more evenly distributed. How well does the test function for low performers? Item analyses using classical test theory and item response theory methodologies confirmed that GSAT2005 was better able to discriminate amongst lower ability students than was GSAT2004. As indicated in Table 1, fewer test-takers scored at the chance level on GSAT2005 than on GSAT2004. In addition, the reliability of the scales for students scoring below the median was quite low for GSAT2004 (coefficient alpha for Math = .58 and for Language = .47) as compared with GSAT2005 (coefficient alpha for Math = .81 and for Language = .68). 47 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Figure 7.1: Distribution of scores 80.0 75.0 70.0 65.0 60.0 55.0 50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 GSAT 2004 Math 8000 6000 4000 2000 0 Std. Dev = 16.52 Mean = 34.9 N = 47396.00 Figure 7.2: Distribution of scores Figure 7.3: Distribution of scores Figure 7.4: Distribution of scores 80.0 75.0 70.0 65.0 60.0 55.0 50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 GSAT 2005 Language 8000 7000 6000 5000 4000 3000 2000 1000 0 Std. Dev = 17.98 Mean = 42.6 N = 49157.00 48 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Although GSAT2005 functioned better for low performers than GSAT2004, there were still sizable proportions of the students scoring close to the chance level. We wondered if the GSAT2005 could be used to measure performance more precisely for those students scoring around the chance level, by constructing “easier” tests from the existing items, even though we did not have access to the actual wording of the items and did not collect additional data. We used a statistical method to identify items that performed better for the lower-performing test-takers. Identified items satisfied the following criteria: (a) adequate overall item indices, and (b) relatively better (>.2 item to total correlations) discrimination for scorers below the median. These items tended to be easier items. Using just these items, we generated “easier” Math and Language tests and rescored the tests, to create “easy” GSAT2005 math and language scores for all test takers. These new scores are based solely on an empirical analysis of the item data from the national administration. As evident in Table 7.1, the new versions of GSAT2005 had slightly higher reliability for lower performing students without significantly reducing the reliability for all students. Curiously, although the new Math test reduced the percentage of students scoring below the chance level, the new Language test did not. Possibly students with questionable literacy will not perform better on the Language test regardless of how easy it is. Table 7.1: Test statistics for GSAT2004 and GSAT2005 Mathematics and Language tests and an empirically generated easier version of the GSAT2005 tests GSAT2004 GSAT2005 GSAT2005 Easier version* Math (Based on 42 items) Mean (SD) 34.91 (16.52) 46.23 (18.82) 28.43 (10.65) Median 30 47 31 Skewness .742 -.054 -.528 N 47396 49196 49196 Reliability (all students) .95 .96 .94 Percentage of students scoring at or below chance (Score<25% correct) 19.5 9.7 6.8 Reliability for students scoring below the median .58 .81 .84 Language (Based on 35 items) Mean (SD) 38.38 (16.38) 42.59 (17.98) 21.67 (9.50) Median 35 41 23 Skewness .451 .166 -.202 N 47416 49157 49157 Reliability (all students) .94 .95 .94 Percentage of students scoring at or below chance (Score<25% correct) 13.1% 11% 14% Reliability for students scoring below the median .47 .68 .73 Scores from these “easier” versions were hypothesized to be more sensitive for identifying smaller differences in NHP and their comparison schools. Thus, Math and Language scores using just the items from the “easier” versions were calculated for all students and then used as dependent measures in supplemental analyses repeating the OLS analyses that had been conducting using the full tests. Findings replicated those reported in Chapter 4 suggesting that even with a more sensitive achievement measure, performance in NHP and matched non-NHP was not significantly different. 49 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Scores must be equated or anchored to a common scale: As is evident from the previous analyses, the GSAT changed dramatically between 2004 and 2005. This is likely due in part to a change in the blueprint and test specifications required to align the test to the RPC. However, at this time, the SAU does not equate scores from one year to the next and over the last several years the scores have fluctuated considerably (see Lockheed, Harris, Gammill and Barrow 2005 for details). Thus, an 11-point increase in average math scores between GSAT2004 and GSAT2005 does not mean that students performed better in 2005. Rather, changes in average performance reflect changes in the test and may also reflect changes in the student candidate pool. Although the SAU follows test development procedures to create parallel forms of the GSAT by using test blueprints, item specifications, and pretested items, they do not conduct analyses following test administration to equate scores for year-to-year comparisons. In past years they have repeated anchor items in order to facilitate equating. At this time it is unclear if these items would be adequate to begin the equating process for GSAT2006. However, in order for the GSAT to be useful eventually for year-to-year monitoring, this equating process would be critical. It would be especially useful to articulate a scale with meaningful score ranges and then to anchor the test each year against this common scale. The SAU is interested in this technical support but they do not currently have the skills in-house to conduct the scaling. CONCLUSION AND RECOMMENDATIONS To be used to monitor change over time, all existing national tests would need to be horizontally equated. Test development processes are in place that would enable this to be done, but post-test equating is not currently practiced for any of the national tests. Given the uncertainty of the possible changes in content and timing of the Grade 3 Diagnostic test, its utility for use as a monitoring indicator is questionable. The Grade 4 Literacy test holds promise as a valuable measure for monitoring the literacy skills of low performing Jamaica youth. It would be necessary to collect actual Grade 4 Literacy test item responses and scores, not just mastery levels as is the current practice. The GSAT holds promise as a potential PMP indicator. The content of the test is likely to be stable over the next few years. The distribution of scores and the item analyses of the GSAT2005 tests of Math and Language suggest that the newly revised test blueprint targets skills across the score range and not just at the top of the range, as was the case for GSAT2004 and before. The major drawback for using the GSAT as a PMP indicator is that the test is not equated from year to year, and thus provides no common scale against which annual fluctuations in performance can be understood and adjusted. Thus, the GSAT can be meaningfully used to compare students and schools within a given year, but GSAT scores cannot be meaningfully compared from one year to the next. It is strongly recommended that the SAU be provided with technical support to facilitate and conduct the equating process. 50 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Note to chapter 7 A review of mathematics items on the 2004 Grade 3 Diagnostic Test indicated that almost all items required reading comprehension (see table below). This is not a critique of the items. Current best practice in numeracy education emphasized mathematics in the context of problem solving and practical applications. Jamaica began implementing this approach as part of its primary curriculum reform in the 1990s. It is appropriate to measure mathematics that is contextualized. However, students who are poor readers will be disadvantaged in reading comprehension should be considered in interpreting the math scores. Area Item numbers Number of items Share of test (percentage) Number of words per item (average) Number 1-30 30 50 10 Estimation and Measurement 31-40 10 17 10 Geometry 41-47 7 12 9 Algebra 48-54 7 12 4 Statistics 55-60 6 10 20 51 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact CHAPTER 8. CONCLUSIONS AND RECOMMENDATIONS The New Horizons for Primary Schools (NHP) was implemented in 72 government schools in Jamaica, from 1998-2005. The program provided support to schools on the basis of needs identified through the preparation of a School Development Plan (also called a School Improvement Plan). This independent evaluation report first compares the schools in the NHP with a statistically matched comparison group of government schools, with respect to (a) support received for literacy and numeracy, (b) learning outcomes as measured by national tests in grades 3, 4 and 6, and (c) impact of NHP on learning comes. Second, the report examines school-level factors related to gains in literacy and numeracy. Third, the report assesses the utility of the existing national tests for monitoring change over time. Finally, the report makes recommendations for better measuring NHP outcomes and for better assessing the impact of the NHP on student learning. CONCLUSIONS Support for literacy and numeracy was greater in NHP schools as compared with a group of statistically matched non-NHP schools. In October and November, 2005, researchers visited 71 extant NHP and 67 statistically matched comparisons schools and carried out group interviews with the principal and experienced teachers. The survey asked about innovative mathematics and literacy programs at the school, in-service teacher training, governance and leadership training, parent education and training, supplementary reading and math materials, computer use in schools and training teachers about computers, training resource teachers, nutrition and health programs, integrating statistical data bases, and linking project schools with national EMIS. In all areas, other than parent education and health and nutrition, NHP schools were more advantaged than matched non-NHP schools. Learning outcomes in NHP schools and matched non-NHP schools fluctuated over time along with national fluctuations in test results and were not consistently higher in NHP schools as compared to matched non-NHP schools. National school-level performance data on the Grade 3 Diagnostic test and the Grade 4 Literacy test and individual student performance data on the Grade 6 Achievement test (GSAT) were obtained and analyzed. No differences were observed in seven of the ten tests analyzed. The performance of students in NHP schools was higher than the performance of students in the statistically matched non-NHP schools on two tests: • 2004 Grade 3 Communications Task I 52 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact • 2004 Grade 6 Communications Task I The performance of students in NHP schools was lower than the performance of students in the statistically matched non-NHP schools on one test: • 2005 Grade 6 Communications Task I The performance of students in NHP schools was no different from the performance of students in the statistically matched non-NHP schools on the following seven tests: • 2004 Grade 3 Math • 2004 Grade 3 Language • 2004 Grade 4 Literacy • 2004 Grade 6 Math • 2004 Grade 6 Language • 2005 Grade 6 Math • 2005 Grade 6 Language It appears that certain types of training and provision of resources are helpful, while other types of training are provided in response to school need. At the margin, supplementary reading materials, training of reading resource teachers, computers for administrative purposes and internet access are associated with higher scores, while teacher training and principal peer learning are associated with lower GSAT2005 scores. The existing national tests have drawbacks that limit their utility as USAID PMP indicators. Each of the national tests was analyzed to assess their utility as monitoring indicators. Drawbacks differ from test to test, and the principal ones are: • Possible changes in the timing and content of the Grade 3 Diagnostic test • Lack of precision in reporting sub-scores and total scores from the reporting sheet of the Grade 4 Literacy test, which reports only non-mastery, near-mastery or mastery levels of performance • Lack of uniformity in forwarding to the MOEYC the Grade 3 and Grade 4 tests results; using these tests for monitoring would require improved data collection, which is ongoing. • Absence of centrally-available student-level test score data for the Grade 3 and Grade 4 assessments. In addition, student-level demographic data, including age and gender, and student-level school history data, such as the student’s attendance, past performance on tests, or classroom assignments, would improve precision of monitoring and evaluation analyses. • Lack of annual, post-test horizontal equating of Grade 3 and Grade 4 tests, through empirical IRT methods, to place test results on a common scale to ensure comparability of measures over time. . 53 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact • As a placement examination, the GSAT assessment is not designed to measure student performance at the lower end of the scale, although the GSAT2005 has greater discrimination at this level. Since NHP schools were selected because the average performance of their students is at the low end of the scale, this is problematic. • The GSAT tests are also not horizontally equated (so there is considerable fluctuation from year to year). As a result, trend data are not meaningful and year to year comparisons are inappropriate. • Waiting five or six years to measure improvement using the GSAT is too long. Using the Grade 3 Diagnostic Test and the Grade 4 Literacy Assessment would provide better interim measures. RECOMMENDATIONS Monitoring and evaluation could be strengthened, to better measure the impact of the NHP and provide guidance for system improvement. For better monitoring of changes over time, the AED Team recommends that the USAID Mission work with the MOEYC and with external technical assistance to ensure that: • The GSAT has fidelity at the low end of the scale and is horizontally equated. • The GSAT, Grade 3 Diagnostic and Grade 4 Literacy tests are horizontally equated. • Actual Grade 4 Literacy test item responses and scores are reported centrally, not just school averages of mastery levels For better measuring the impact of the NHP, we also recommend that the USAID Mission work with the MOEYC and with external technical assistance to ensure that: • Prior to implementing the new NHP, a prospective comparison group is created through propensity score matching (PSM) of all NHP schools with a sample of non￾NHP schools, using the most recent data from the Annual School Census and 2005 national test results to create the propensity scores. The identification of control schools at the outset is essential, given that year-to-year comparisons are not meaningful with the existing, un-equated tests. • The JSAS software is available, functional and staff trained in the use of this software. The software would serve as a school based tool and would also serve the needs of USAID and the MOEYC. This software would need to be installed in all NHP and matched comparison schools, and computers provided as needed to these schools (to replace obsolete computers or to provide computers for administration). 54 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Data from the school based JSAS would need to be collected on a regular basis from all NHP and matched comparison schools. • Data from all NHP and all statistically matched comparison schools are regularly collected. These data would cover the implementation of specific aspects of the NHP or provision of similar inputs to matched comparison schools. • Independent evaluators carry out the program evaluation, including (a) extracting routine administrative data, (b) collecting any additional data as necessary, and (c) using statistical methods to assess the impact of the NHP program as a whole and evaluate the contribution of various interventions to improvements in learning outcomes. 55 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact REFERENCES Bailey, B. (2000). Issues of Gender and Education in Jamaica: What About the Boys? Kingston, UNESCO. Dye, Richard, Joan Jennings, Clement Lambert, Barbara Hunt and Gerald Wein (July 2002). Evaluation and Recommendations for Strengthening and Extending the New Horizons for Primary Schools Project in Jamaica. Washington, DC: Aguirre International. Enge, Kjell and Heather Simpson (October 2003). New Horizons for Primary Schools/Jamaica: Formative Evaluation 2003. Los Angeles, CA: Juarez and Associates, Inc. Harris, A.M. (December 2005). Psychometric analysis of GSAT2004 and GSAT2005 Math and Language tests: Suitability of the Grade Six Achievement Test (GSAT) as a PMP Indicator. Washington, DC: Academy for Educational Development. (Research funded by USAID) Juarez and Associates, Inc. (June 1999). New Horizons for Primary Schools: Diagnostic Report on Project Schools. Los Angeles, CA: Juarez and Associates, Inc. Juarez and Associates, Inc. (December 1999). New Horizons for Primary Schools: Report on the Status of Numeracy and Literacy Programs in Individual Schools. Los Angeles, CA: Juarez and Associates, Inc. Lockheed, M. E., Harris, A. M., Gammill, P., & Barrow, K. (2005). Impact of New Horizons for Primary Schools on literacy and numeracy in Jamaica, 1999-2004. Washington DC: Academy for Educational Development. (Research funded by USAID) Available on the USAID website: http://www.dec.org/search/dexs/index.cfm?fuseaction=Dexs.citation&rec_no=133934 Lockheed, M. and Tamara Jayasundera (November 2005). Did the New Horizons Project Boost Literacy and Numeracy in Grades 3 and 4 in Jamaica? Washington, DC: Academy for Educational Development. (Research funded by USAID) Ministry of Education, Youth and Culture, Planning and Development Division, Statistics Unit (October 2003). Annual School Census Questionnaire 2003/2004: Public Schools. Kingston, Jamaica: GOJ MOEYC New Horizons Activity Project Implementation Unit (December 1998). Final report: Project Schools Selection, New Horizons Activity. 56 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact No Author (Spring 2003). Summary Evaluation of School Improvement Plans (SIP) of NHP Schools. O’Neil, Ernest (October 2003). New Horizons for Primary Schools: Special Report to Private Sector Organizations of Jamaica. Kingston, Jamaica: New Horizons for Primary Schools Russell, Fitz-Albert R. (1996). Issues of validity related to the Jamaican Grade Six Achievement Test. Unpublished doctoral dissertation, Florida State University, Tallahassee. 57 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact ANNEX A. PROPENSITY SCORE MATCHING In the earlier report (Lockheed et al 2005), propensity score matching (PSM) was carried out using a data set in which four errors of coding were subsequently identified.7 Corrections were made in the identification of four schools with respect to their status as NHP schools, in the matching of School Census data with GSAT data at the school level for one school and in estimating a 1999 GSAT score for one school, based on a regression where GSAT1999 = f (GSAT2000). One consequence of these corrections was that the PSM needed to be re-estimated for the 72 NHP schools. We did this, using nearest-neighbor matching with calipers of .1, .05 and .01, with replacement. We also did one-to-one matching with non-replacement. In the process, we found that one of the variables used in the original PSM exercise – “orig195”, whether or not the school was in the group of schools initially identified as qualified for the NHP – could no longer be used in the matching logistic regression, as all 72 NHP cases were in this group. We instrumented “orig195” but the results were not promising: the logit correctly predicted only 2 of 72 NHP cases and the pseudo r-square for the stage one regression was .06. We then redid the PSM without the variable “orig195”. The logit correctly predicted 2 of the 72 NHP cases with a pseudo r-square of .0904. A literature review of recent applications of PSM to small samples yielded consensus that the matching should be on the basis of “replacement” – meaning that, once matched to a treatment school, a matched non-NHP school would be “replaced” in the pool of matched non-NHP schools eligible for subsequent matching to another treatment school. On the basis of this procedure, we identified 66 unique matched non-NHP schools matched to the 72 treatment schools on propensity scores that differed less than .05. Since the NHP schools were selected across Jamaica, we added dummy variables for the six (one omitted) regions to the NHP prediction equation, and obtained matched non-NHP schools for all but one NHP school (with replacement) and 70 one-to-one matched non￾NHP schools (without replacement). However, the logit correctly predicted only 3 of the 72 NHP cases with a pseudo r2 of .1069 (Regression results are presented below in Table A.1.). Variables associated with NHP at p<.10 were shift, number of primary students (-), number of primary teachers, and 1999 GSAT math (-). 7 Two schools were incorrectly identified as NHP, one school did not have a GSAT match with the School Census data, and one school lacked 1999 GSAT data 58 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Conclusion Although specific criteria were given for the selection of NHP schools, empirical measures related to these criteria were not strongly related to NHP incidence. Table A1. Logistic regression, likelihood of school being selected as NHP school, Jamaica 1999. Odds Ratio Std. Err. z P>|z| [95% Conf. Interval] Remote rural location 0.604811 0.275006 -1.11 0.269 0.248074 1.474545 Rural location 0.586479 0.221142 -1.42 0.157 0.280087 1.228041 Region 2 1.752805 0.878127 1.12 0.263 0.656592 4.679202 Region 3 0.831429 0.437815 -0.35 0.726 0.296209 2.333737 Region 4 0.742698 0.40079 -0.55 0.581 0.257911 2.138724 Region 5 0.995249 0.463149 -0.01 0.992 0.399776 2.477688 Region 6 1.820952 0.872953 1.25 0.211 0.711603 4.659713 Board and/or Scope 0.786287 0.320577 -0.59 0.555 0.35362 1.748336 Active PTA 2.416165 1.909774 1.12 0.264 0.513244 11.37442 Breakfast program 1.239283 0.47391 0.56 0.575 0.585686 2.622261 Shift 2.482168 1.252968 1.8 0.072 0.922901 6.67586 Enrolment in 1-6 0.996907 0.001545 -2 0.046 0.993883 0.999941 Teachers in 1-6 1.109875 0.058616 1.97 0.048 1.000736 1.230917 Mathematics 1999 0.886221 0.046916 -2.28 0.023 0.798878 0.983114 Language 1999 0.97441 0.044895 -0.56 0.574 0.890274 1.066497 Communication1 1999 1.594825 0.649552 1.15 0.252 0.717843 3.543206 Communication2 1999 0.778728 0.444161 -0.44 0.661 0.254617 2.381682 Years in service at 1_6 0.987943 0.024733 -0.48 0.628 0.940638 1.037627 Years in service at school 0.999549 0.006535 -0.07 0.945 0.986823 1.012439 % with CXC qualificat. 10.03836 18.32729 1.26 0.206 0.280287 359.5199 % with Certificate 8.388521 14.19605 1.26 0.209 0.304216 231.3074 % with Masters 1.197632 0.205942 1.05 0.294 0.854974 1.677623 Note: Omitted region is Brown’s Town. Number of obs = 786 LR chi2(22) = 51.48 Prob > chi2 = 0.0004 Log likelihood = -214.95952 Pseudo R2 = 0.1069 59 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact ANNEX B. THREE NON-NHP COMPARISON GROUPS The present report discusses findings related to three comparison groups. This Annex describes these three groups, all of which are based on PSM matching exercises carried out in October 2005 using the logit regression presented in Annex A. Group March 2005: This group is the comparison group that appears in the initial evaluation report. When four recoding corrections were made, the logit regression that predicted this group could no longer be used and a new logit regression was estimated. New PSM scores from this new (October 2005 with dummy variables for regions) regression were generated for each of the 70 non-NHP schools in this group. Group A: PSM with replacement: This group was identified through the October 2005 logit regression (with dummy variables for regions), in which matches were returned to the pool of potential comparison schools. Matches were found for all but 1 NHP school, with 7 schools used more than once. Six of these schools were used twice and one was used 4 times. Group B: PSM with non-replacement. This group was also identified through the October 2005 logit regression (with dummy variables for regions), but successful matches were not returned to the pool of potential comparison schools. Matches were found for all but 2 NHP schools. Group C: PSM with replacement and substitution. Group A is the basis for this group. However, selected schools in the non-NHP list were replaced by any school that was initially identified as a non-NHP match school in March 2005 and whose October 2005 PSM score was the same as that of the school being replaced. A total of 40 schools from the March 2005 list had propensity scores within .05 of both the NHP school to which it was being matched and of the empirically generated match. In this case, the replacement was made. Interestingly, 65 percent of the Group C schools came from the initial list of pre-qualified schools, compared with fewer than 20 percent of either Group A or Group B schools. 60 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Table B.1 Propensity Scores of the NHP schools and the Matching Schools from the Three Comparison Groups Pair NHP Group A Group B Group C March 2005 1 0.375424 0.3820437 0.411826 0.3820437 2 0.35954 0.3643285 0.3643285 0.33168 0.3643285 3 0.244051 0.2438581 0.2438581 0.243858 0.2438581 4 0.286995 0.2856985 0.2856985 0.285699 0.2856985 5 0.343706 0.340642 0.340642 0.340642 0.340642 6 0.145774 0.1451638 0.1451638 0.145098 0.1451638 7 0.527593 0.5162771 0.5162771 0.516277 0.5162771 8 0.195518 0.1961126 0.1981155 0.198116 0.1961126 9 0.293068 0.2961921 0.2961921 0.297488 0.2961921 10 0.156024 0.1570321 0.1570321 0.157032 0.1570321 11 0.457256 0.4554726 0.4554726 0.440619 0.4554726 12 0.204965 0.2047709 0.2047709 0.207169 0.2047709 13 0.599017 14 0.240704 0.2398642 0.2398642 0.239864 0.2398642 15 0.157793 0.157407 0.157407 0.157407 0.157407 16 0.14722 0.1492377 0.1514389 0.149238 0.1492377 17 0.265 0.2652515 0.2652515 0.268492 0.2652515 18 0.224852 0.2249335 0.2249335 0.228803 0.2249335 19 0.146338 0.1451638 0.1511369 0.145164 0.1451638 20 0.049221 0.049116 0.049116 0.048299 0.049116 21 0.152299 0.1520756 0.1520756 0.152076 0.1520756 22 0.126812 0.1272792 0.1272792 0.125175 0.1272792 23 0.14602 0.1451638 0.1492377 0.145164 0.1451638 24 0.037552 0.0373183 0.0373183 0.039193 0.0373183 25 0.119864 0.1198781 0.1202041 0.120953 0.1198781 26 0.366163 0.3643285 0.4116485 0.364329 0.3643285 27 0.017604 0.0175418 0.0175418 0.02134 0.0175418 28 0.052105 0.0521836 0.0521836 0.052184 0.0521836 29 0.182433 0.182601 0.182601 0.182601 0.182601 30 0.137381 0.1378225 0.1378225 0.137865 0.1378225 31 0.171599 0.1708269 0.1708269 0.175833 0.1708269 32 0.370348 0.3643285 0.4118264 0.323061 0.3643285 33 0.087431 0.0874422 0.088293 0.089899 0.0874422 34 0.085475 0.0855861 0.0855861 0.085359 0.0855861 35 0.097525 0.0978607 0.0978607 0.097861 0.0978607 36 0.140917 0.1401056 0.1418864 0.140106 0.1401056 37 0.098196 0.0981866 0.0981866 0.098187 0.0981866 38 0.036256 0.0361684 0.0361684 0.037038 0.0361684 39 0.030572 0.0305642 0.0305642 0.031783 0.0305642 40 0.087294 0.0874422 0.0874422 0.088522 0.0874422 41 0.296246 0.2961921 0.2974878 0.323061 0.2961921 42 0.023784 0.0238217 0.0238217 0.023822 0.0238217 43 0.069631 0.0697395 0.0697395 0.067822 0.0697395 44 0.132974 0.1330859 0.1330859 0.132622 0.1330859 45 0.02821 0.0282001 0.0282001 0.0282 0.0282001 46 0.0457 0.0456606 0.0456606 0.044159 0.0456606 47 0.048523 0.0484426 0.0484426 0.047609 0.0484426 48 0.095819 0.0957442 0.0957442 0.095744 0.0957442 49 0.121404 0.1214606 0.1214606 0.121788 0.1214606 50 0.062639 0.0627135 0.0627135 0.062714 0.0627135 51 0.162218 0.1625419 0.1625419 0.162542 0.1625419 52 0.104255 0.1043026 0.1043026 0.103187 0.1043026 53 0.010005 0.0093991 0.0093991 0.009399 0.0093991 54 0.163886 0.16362 0.16362 0.168216 0.16362 55 0.094261 0.0942395 0.0942395 0.09424 0.0942395 56 0.101796 0.1017858 0.1017858 0.101786 0.1017858 57 0.085549 0.0855861 0.085767 0.086832 0.0855861 58 0.082087 0.0819854 0.0819854 0.081985 0.0819854 59 0.14004 0.1399766 0.1399766 0.139977 0.1399766 60 0.119571 0.1198781 0.1198781 0.120204 0.1198781 61 0.363211 0.3643285 0.3820437 0.364329 0.3643285 62 0.148985 0.1492377 0.1517204 0.149238 0.1492377 61 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact Pair NHP Group A Group B Group C March 2005 63 0.140839 0.1401056 0.1401056 0.140106 0.1401056 64 0.061841 0.0619062 0.0619062 0.061374 0.0619062 65 0.196829 0.1961126 0.2006823 0.202354 0.1961126 66 0.069026 0.0690925 0.0690925 0.066765 0.0690925 67 0.193181 0.1919115 0.1961126 0.191912 0.1919115 68 0.094822 0.0947867 0.0947867 0.094787 0.0947867 69 0.072284 0.0724146 0.0724146 0.072955 0.0724146 70 0.191439 0.1919115 0.1919115 0.191912 0.1919115 71 0.12774 0.1275111 0.1275111 0.125659 0.1275111 72 0.278472 0.2849856 0.2849856 0.284986 0.2849856 62 Lockheed, Harris, Gammill, Barrow and Jayasundera NHP Inputs, Outcomes and Impact ANNEX C: JAMAICA AED SCHOOL SURVEY 2005 63 Jamaica AED School Survey ID __ __ __ __ __ __ Page 1 Jamaica AED School Survey 2005 FOR PRINCIPAL AND TEACHERS The Jamaica AED School Survey (JASS) was administered in October and November of 2005 to 139 NHP schools and their matched pairs by the Caribbean Consulting Group (CCG). CCG completed this work under a contact with The Academy for Educational Development (AED), The survey consists of 83 questions. In all, 246 discrete data elements were collected from these 83 questions. The survey is divided into twelve sections covering questions on the following topics: SECTION I: SCHOOL INFORMATION SECTION II: SCHOOL IMPROVEMENT PLANS AND RESOURCES SECTION III: INTRODUCTION OF THE REVISED PRIMARY CURRICULUM SECTION IV. SUPPLEMENTARY MATERIALS SECTION V: PRINCIPAL TRAINING SECTION VI: RESOURCE TEACHER TRAINING SECTION VII: SCHOOL-BASED TRAINING BY OUTSIDE SPECIALISTS SECTION VIII: TRAINING PROVIDED BY SUPPORT TEACHERS SECTION IX. COMMUNITY AND PARENT TRAINING SECTION X. HEALTH AND NUTRITION SECTION XI. COMPUTER USE IN THE SCHOOL SECTION XII GRADE 4 LITERACY Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 2 Jamaica AED School Survey 2005 INTERVIEWER QUESTIONNAIRE Interviewer Name: _____________________________________________________ Date of interview: Month_______Day ______ 2005 Time interview started: _________________ Time interview ended: _________________ Directions for selecting respondents: Choose three respondents: the principal, one teacher of grades 1-3 and one teacher of grades 4-6. Choose resource teachers/ coordinators if they are available. Conduct the interview with this GROUP, not individually. The teachers should have worked in the school for at least the past 3 years. Get information about what year each of the respondents started working at this school. SECTION I: SCHOOL INFORMATION Q1. School Name (in block letters)__________________________________________ Q2. School ID (from list provided) __ __ __ __ __ __Place on top of every page Q3 School address_________________________________________________________ Q4. School telephone number________________________________________________ Q5. School email address___________________________________________________ Q6. School type Circle ONE only (a) primary, (b) primary and infant, (c) all age, (d) primary and junior high, (e) other Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 3 School Type NHP Comp Missing 4.2% 3.0% All Age 37.5% 29.9% Other 2.8% 6.0% Primary 43.1% 49.3% Primary & Infant 6.9% 6.0% Primary & Junior High 5.6% 6.0% Total 100.0% 100.0% Name of Principal_________________________________Q7. In school since ________ Name of Teacher for Grades 1-3 _____________________Q8 In school since ________ Name of Teacher for Grades 4-6______________________Q9.In school since ________ NHP NHP Comp Mean Mean Principal_in_school_since 1986 1987 Teacher #1 In_school_since 1990 1991 Teacher #2 in_school_since 1990 1993 Q10. How might other schools have heard about this school? Has it been designated with a special term, such as a Professional Development Center, an IT Pilot school, a “best practice” for teaching and learning Demonstration School? CHECK ALL that respondents mention but DO NOT READ list __ Professional Development Center __ Demonstration School (“best practice”) __ Learning Center __ Lighthouse School __ Pilot School for the Revised Primary Curriculum __ Pilot School for Instructional Technology (IT) __ Adult Literacy Center __Other________ __ None Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 4 # of Designations School Type NHP Comp 0 13.9% 49.3% 1 54.2% 46.3% 2 20.8% 1.5% 3 5.6% 1.5% 5 1.4% Missing 4.2% 1.5% Total 100.0% 100.0% SECTION II: SCHOOL IMPROVEMENT PLANS AND RESOURCES SAY: School Improvement Plans (or School Development Plans) are documents that schools develop; they analyze the problems of the school and recommend solutions. In some cases, a special group (the School Improvement Action Group) works on these plans. In some cases, schools prepare Annual Action Plans that specify what the school will do, over a 3 or 5-year period. And some of these School Improvement Plans focus on improving students’ reading and math skills. The Government of Jamaica and donors to education have provided support to primary education in Jamaica. This support includes many different types of inputs to schools. Not all schools have received the same inputs. Some schools have received small cash grants, while other schools have received resources from the Ministry of Education, Youth and Culture (MOEYC) or donations from local sponsors. This section asks about School Improvement Plans and donations received by the school to support these plans. Q11. Does this school have a School Improvement Action group that was trained to analyze school problems and contribute to school decision-making? Circle ONE only (a) Yes, there is a trained group (b) Yes, there is a group but it has not been trained (c) No School Type NHP Comp Missing 4.2% 4.5% No 6.9% 17.9% Yes but not trained 37.5% 52.2% Yes Trained Group 51.4% 25.4% Total 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 5 Q12. Did this school develop a School Improvement Plan or School Development Plan for the purpose of improving the reading and math skills of students at the school? Circle ONE only (a) Yes, the school has a School Improvement Plan that includes math and reading (b) Yes, the school has a SIP/SDP, but it does not include math and reading (c) Yes, the school is in the process of developing a School Improvement Plan (d) No If No, Skip to Q 15 School Type NHP Comp Total Missing 4.2% 1.5% 2.9% No 1.4% 3.0% 2.2% Yes in Process of SIP 4.5% 2.2% Yes School has a SIP 94.4% 85.1% 89.9% Yes SIP but not Math & reading 6.0% 2.9% Total 100.0% 100.0% 100.0% Q13. Does the School Improvement Plan include a school-wide Annual Action Plan for improving the reading and math skills of students at the school? Circle ONE only (a) Yes for a 3-year plan period (give dates)______________ (b) Yes for a 5-year plan period (give dates)______________ (c) No If No, Skip to Q15 School Type NHP Comp Total Missing 6.9% 4.5% 5.8% I year plan 1.4% .7% No 1.4% 9.0% 5.0% Yes 3 Yr Plan 63.9% 61.2% 62.6% Yes 5 Yr Plan 26.4% 25.4% 25.9% Total 100.0% 100.0% 100.0% Q14. Who developed the Annual Action Plan? That is, who were the members of the School Improvement Plan/School Development Plan Action Group? CHECK ALL that respondents mention but DO NOT READ list __ Board Chairman __ Other Board members __ Parent representatives from the PTA __ Other parents __ Community members not from PTA Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 6 __ Principal __ Teachers __ Former students (alumni) __Other________________________ School Type # of Groups mentioned NHP Comp Total 0 2.8% 10.4% 6.5% 1 2.8% 4.5% 3.6% 2 22.2% 22.4% 22.3% 3 18.1% 10.4% 14.4% 4 19.4% 17.9% 18.7% 5 12.5% 16.4% 14.4% 6 11.1% 13.4% 12.2% 7 5.6% 3.0% 4.3% 8 1.4% .7% Missing 4.2% 1.5% 2.9% Total 100.0% 100.0% 100.0% Q15. In the past 5 years, has any local sponsor (such as Kiwanis, Rotary, Lion’s Club, a local church, Adopt-a-School) helped with resources (cash or in-kind), either regularly or through a special project, for any purpose? Circle ONE only (a) Yes. Who is the sponsor?_________________________________________ (b) No School Type NHP Comp No 17.4% 24.2% Yes 82.6% 75.8% Total 100.0% 100.0% Q16. In the past 5 years, has the school received additional resources (from any source) for the purpose of improving students’ reading and math skills? By additional resources, we mean resources that are not those regularly provided by the MOEYC. Circle ONE only (a) Yes (b) No If No, Skip to Q 19 School Type NHP Comp Total No 4.3% 21.2% 12.6% Yes 95.7% 78.8% 87.4% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 7 Q17. What sort of additional resources has this school received over the past 5 years? Please look over the following list, and indicate all the types of additional resources this school has received. ? CHECK ALL that respondents mention. __ In-kind donations of hard goods for instructional purposes, such as computers or books (Skip to Q19) __ In-kind donations of hard goods to improve school facility, such as adding fences, repairing desks, paving driveway (Skip to Q19) __ Technical assistance, such as teacher training (Skip to Q19) __ Funds raised for special instructional projects (Ask Q18) __ Funds raised for special non-instructional projects, such as for school meals (Ask Q18) __ Other actual cash donations (Ask Q18) School Type Number indicated NHP Comp Total 0 4.2% 20.9% 12.2% 1 6.9% 13.4% 10.1% 2 20.8% 16.4% 18.7% 3 20.8% 22.4% 21.6% 4 19.4% 14.9% 17.3% 5 16.7% 7.5% 12.2% 6 6.9% 3.0% 5.0% Missing 4.2% 1.5% 2.9% Total 100.0% 100.0% 100.0% Q18. Who provided these funds or cash donations? CHECK ALL that respondents mention but DO NOT READ list __ GOJ/MOEYC (in addition to regular grant) __ A local charitable foundation or sponsor __ Teachers, through their fund raising efforts __ The school’s PTA __ Former students of the school (alumnae) __ The local community or parents of students __ A local company __ A local politician/ member of Parliament __ Adopt-a-school __ Tuck shop __Other_________ Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 8 School Type Number inidcated NHP Comp Total 0 29.2% 40.3% 34.5% 1 15.3% 19.4% 17.3% 2 22.2% 20.9% 21.6% 3 11.1% 11.9% 11.5% 4 12.5% 6.0% 9.4% 5 2.8% 1.4% 6 1.4% .7% 8 1.4% .7% Missing 4.2% 1.5% 2.9% Total 100.0% 100.0% 100.0% Q19. Did any of the resources that the school received directly support a School Improvement or School Development Plan to improve students’ reading and math skills? Circle ONE only. (a) Yes (b) No (c) This school does not have a School Improvement or School Development Plan School Type NHP Comp Total Missing 5.6% 4.5% 5.0% No 1.4% 19.4% 10.1% School Does not have an SIP 3.0% 1.4% Yes 93.1% 73.1% 83.5% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 9 SECTION III: INTRODUCTION OF THE REVISED PRIMARY CURRICULUM SAY: I’m now going to ask some questions about the Revised Primary Curriculum. Q20. In some schools the Revised Primary Curriculum was introduced all at once, and in other schools it was introduced over several years, by grade. Please look at the following options and choose the one that most closely describes how the revised primary curriculum was introduced in this school. ? Circle ONE only. (a) Introduced in all grades during school years 1997/98 and 1998/99 (b) Introduced in all grades during school year 1999/2000 (c) Phased introduction: School year 2000/2001 for Grades 1 and 4 School year 2001/2002 for Grades 2 and 5 School year 2002/2003 for Grades 3 and 6 (d) Introduced, but in a different way than above (e.g. the school obtained a photocopy of the revised curriculum before it was formally introduced) (e) Not yet introduced School Type NHP Comp Missing 5.6% 1.5% Introduced but in a different way 6.9% 4.5% Introduced in all grades during 1999/2000. 25.0% 1.5% Introduced in all grades during 97/98 & 98/99 54.2% Not yet introduced 1.4% Phased introduction 6.9% 92.5% Total 100.0% 100.0% Q21. We are aware that teachers received special training when the revised primary curriculum was introduced. Where did the training of teachers in this school take place, and who conducted the training? Please look at the following list and tell me all the ways teachers in this school were trained for the Revised Primary Curriculum. CHECK ALL that respondents mention __Initial 4-day training workshops, by grade, in large centers __Cluster training workshops with teachers from other schools __Workshops in the school, given by education officers __Other. Please describe___________________________________________ Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 10 School Type Number of selections NHP Comp Total 0 1.4% .7% 1 9.7% 22.4% 15.8% 2 36.1% 46.3% 41.0% 3 36.1% 23.9% 30.2% 4 12.5% 6.0% 9.4% Missing 4.2% 1.5% 2.9% Total 100.0% 100.0% 100.0% SAY: The next two questions ask you to estimate the amount of training in the revised primary curriculum that teachers in this school have received. This training includes summer training in large groups, cluster training during the school year, training in the school during the school year and any other type of training related to the revised primary curriculum Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 11 Q22. About what percent of the current teachers in Grades 1-3 have received more than 10 days of training related to the revised curriculum since it was introduced in this school?_________ percent School Type NHP Comp Total 0 4.4% 38.5% 21.1% 5 1.5% .8% 9 1.5% .8% 10 1.5% .8% 16 1.5% .8% 17 1.5% .8% 25 1.5% 3.1% 2.3% 33 7.4% 9.2% 8.3% 40 1.5% .8% 44 1.5% 1.5% 1.5% 50 16.2% 12.3% 14.3% 57 3.1% 1.5% 60 1.5% .8% 66 7.4% 1.5% 4.5% 67 5.9% 3.1% 4.5% 70 1.5% 1.5% 1.5% 71 2.9% 1.5% 75 2.9% 3.1% 3.0% 78 1.5% .8% 80 1.5% 1.5% 1.5% 83 7.4% 3.8% 85 1.5% .8% 90 2.9% 1.5% 94 1.5% .8% 100 27.9% 15.4% 21.8% Total 100.0% 100.0% 100.0% School Type NHP Comp Mean Mean 68 37 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 12 Q23. About what percent of the current teachers in Grades 4-6 have received more than 10 days of training related to the revised curriculum since it was introduced in this school?_________ percent School Type NHP Comp Total 0 5.9% 50.0% 27.6% 16 1.5% .7% 20 1.5% .7% 25 1.5% 1.5% 1.5% 30 1.5% .7% 33 4.4% 3.0% 3.7% 40 1.5% .7% 42 1.5% .7% 43 1.5% .7% 50 17.6% 7.6% 12.7% 56 1.5% .7% 60 1.5% .7% 66 2.9% 1.5% 2.2% 67 4.4% 4.5% 4.5% 71 1.5% .7% 75 1.5% 1.5% 1.5% 80 4.5% 2.2% 81 1.5% .7% 83 2.9% 1.5% 85 2.9% 1.5% 86 1.5% 1.5% 1.5% 89 1.5% .7% 90 3.0% 1.5% 92 2.9% 1.5% 100 39.7% 16.7% 28.4% Total 100.0% 100.0% 100.0% School Type NHP Comp Mean Mean 71 37 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 13 SAY: Some schools have had special activities that support the revised primary curriculum. The next questions ask about these activities. Q24. Does this school have a Drop Everything and Read program? Circle ONE only. (a) Yes, for about 15 minutes per day (b) Yes, for more than 15 minutes per day (c) No School Type NHP Comp Total Missing 5.6% 4.5% 5.0% No 20.8% 29.9% 25.2% Yes about 15 Min/day 40.3% 44.8% 42.4% Yes more than 15 Min/day 33.3% 20.9% 27.3% Total 100.0% 100.0% 100.0% Q25. Does this school have a Family Literacy (such as Reading Starts with Us) program? Circle ONE only. (a) Yes (b) No School Type NHP Comp Total no 92.8% 89.4% 91.1% yes 7.2% 10.6% 8.9% Total 100.0% 100.0% 100.0% Q26. Does this school have afternoon or evening programs for parents to help them understand the Revised Primary Curriculum so they can help their children? Circle ONE only. (a) Yes (b) No School Type NHP Comp Total No 91.3% 83.3% 87.4% Yes 8.7% 16.7% 12.6% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 14 Q27. Does this school have a Competency Shelter? Circle ONE only. (a) Yes. Please describe what it is:__________________________________ (b) No School Type NHP Comp Total No 75.4% 84.8% 80.0% Yes 24.6% 15.2% 20.0% Total 100.0% 100.0% 100.0% Q28. Has this school been the venue for a Literacy Summer Camp? Circle ONE only. (a) Yes, for five (5) or more summers (b) Yes, for two (2) to four (4) summers (c) Yes, for one summer (d) No, this school has never been the venue of a Literacy Summer Camp School Type NHP Comp Total Missing 5.6% 4.5% 5.0% No 8.3% 14.9% 11.5% Yes 2 to 4 Summers 31.9% 35.8% 33.8% Yes 5 or more summers 47.2% 41.8% 44.6% Yes one summer 6.9% 3.0% 5.0% Total 100.0% 100.0% 100.0% Q29. Are there any other special activities or programs that this school has implemented in support of the Revised Primary Curriculum? Circle ONE only. (a) Yes. Please describe:________________________________________ (b) No School Type NHP Comp Total No 15.9% 36.4% 25.9% Yes 84.1% 63.6% 74.1% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 15 SECTION IV. SUPPLEMENTARY MATERIALS SAY: This section refers to materials that are not regularly provided to the school by the MOEYC, but are in addition to the regular materials. We refer to these as “supplementary” materials. We are interested in supplementary materials that support the Revised Primary Curriculum. Q30. During the past 5 years, has this school received any supplementary reading materials for literacy? Circle ONE only. (a) Yes (b) No If no, skip to Q32 School Type NHP Comp Total No 7.2% 21.2% 14.1% Yes 92.8% 78.8% 85.9% Total 100.0% 100.0% 100.0% Q31. What sorts of supplementary reading materials did this school receive? CHECK ALL that respondents mention but DO NOT READ list __ Textbooks related to the curriculum __“Big books” for shared reading __Story books for the teacher to read to children __A set of identical student books for shared reading __Books for student independent reading __Other______________________________________________________ School Type Number of responses NHP Comp Total 0 7.2% 21.2% 14.1% 1 11.6% 12.1% 11.9% 2 27.5% 31.8% 29.6% 3 34.8% 24.2% 29.6% 4 14.5% 10.6% 12.6% 5 4.3% 2.2% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 16 Q32. How many classrooms have reading corners or classroom libraries (shelves) ________ School Type NHP Comp Total 0 1.5% 3.0% 2.2% 2 1.5% .7% 3 7.4% 12.1% 9.7% 4 4.4% 16.7% 10.4% 5 4.4% 3.0% 3.7% 6 19.1% 22.7% 20.9% 7 2.9% 4.5% 3.7% 8 5.9% 3.0% 4.5% 9 2.9% 4.5% 3.7% 10 1.5% 3.0% 2.2% 11 1.5% .7% 12 4.4% 1.5% 3.0% 13 2.9% 1.5% 14 4.4% 1.5% 3.0% 15 4.4% 2.2% 16 2.9% 1.5% 18 2.9% 4.5% 3.7% 19 1.5% .7% 20 1.5% .7% 21 1.5% 1.5% 1.5% 24 2.9% 1.5% 25 1.5% .7% 27 1.5% .7% 30 1.5% .7% 32 1.5% .7% 37 1.5% .7% 100 16.2% 12.1% 14.2% Total 100.0% 100.0% 100.0% NHP NHP Comp Mean Mean 25 19 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 17 Q33. Is there a school library? Circle ONE only. (a) Yes (b) No School Type NHP Comp Total No 13.0% 13.6% 13.3% Yes 87.0% 86.4% 86.7% Total 100.0% 100.0% 100.0% Q34. During the past 5 years, has this school received any supplementary mathematics materials? Circle ONE only. (a) Yes (b) No If No, Skip to Section V School Type NHP Comp Total No 11.6% 45.5% 28.1% Yes 88.4% 54.5% 71.9% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 18 Q35. What sorts of supplementary math materials did this school receive? CHECK ALL that respondents mention but DO NOT READ list __Textbooks related to the curriculum __ Math activity books __Graph books for students __Sets of teacher guides (e.g. Nelson Primary Maths for Caribbean Schools) __Mathematics software on CDs (e.g. Math Workshop Deluxe) __Professional periodicals for teachers (e.g. NCTM’s Teaching Children Math) __Calculators for students to use __Manipulatives such as blocks, rods, geometric shapes __Measurement tools such as rulers, protractors, compasses __Other_____________________________________________ School Type Number of Responses NHP Comp Total 0 10.1% 43.9% 26.7% 1 12.1% 5.9% 2 10.1% 13.6% 11.9% 3 15.9% 18.2% 17.0% 4 18.8% 7.6% 13.3% 5 15.9% 3.0% 9.6% 6 13.0% 1.5% 7.4% 7 2.9% 1.5% 8 4.3% 2.2% 9 4.3% 2.2% Q35n 10 4.3% 2.2% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 19 Section V: PRINCIPAL TRAINING SAY: The next questions ask about the person who was the principal of the school during the school year 2003-2004. This could be the current principal, or it could be someone who is no longer at this school. Q36. Did the principal who served during 2003-2004 participate in specialized training for principals at any time since the school year 2000? Circle ONE only. (a) Yes (b) No If No, skip to Section VI (c) Don’t know If Don’t know, skip to Section VI School Type NHP Comp Total Missing 6.9% 3.0% 5.0% Don't Know 1.4% 3.0% 2.2% No 1.4% 9.0% 5.0% Yes 90.3% 85.1% 87.8% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 20 Q37. What type of training did the principal who served the school during the school year 2003-2004 participate in? We realize that not all principals participated in all types of training, and we are interested only in the types of training received by the principal of this school. Please look over the list of possible training opportunities, and tell me which ones the principal of this school participated in. CHECK ALL that respondents mention __Overseas summer training at Mt. St. Vincent University in Canada __Training provided by Mt. St. Vincent University in Jamaica (at Bethlehem, Case, Sam Sharpe, or St. Joseph Teachers College) __Training in School Improvement Planning __Workshop on community mobilization __Workshop on parent participation facilitation skills __Training in leadership and management skills for peer mentoring __Peer learning with other principals (e.g. cluster school meetings, NHP principal’s network) __Performance management training __Implementing the revised primary curriculum __Beginning principal training __Training occurred, but not sure of content __Other. Please describe _____________________________________________ NHP NHP Comp Sum Sum Overseas Summer Training 13 9 Training MtStVincent 29 31 Training in SIP 55 36 Workshop Community 21 9 Workshop Parent 28 9 Training Leadership 33 22 Peer Learning 51 29 Performance 38 22 Implementing 55 39 Principal Training 20 19 Training Occurred 9 9 Other 10 7 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 21 SECTION VI: RESOURCE TEACHER TRAINING SAY: In some primary schools, experienced teachers have been identified as persons who can provide support to other teachers. These support teachers are called Resource Teachers or Literacy/Numeracy/ Assessment Coordinators. Some larger schools have Support Teachers that help other teachers on a grade by grade basis, called Grade Coordinators. In the next section we call all of these types of teachers Support Teachers. In some cases, these support teachers have received specialized training. This section asks about these school-based support teachers and the types of training they have received. Q38. First,in the last school year, 2004-2005, did you have any Support Teachers for Grades 1-6? Circle ONE only. (a) Yes. How many? _______________ teachers (b) No. If No, Skip to Section VII School Type NHP Comp Total No 14.5% 33.3% 23.7% Yes 85.5% 66.7% 76.3% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 22 Q39. What types of special training did these Support Teachers receive over the past 5 years to help them provide assistance to other teachers in this school? Please review the list of topics for courses or workshops for Support Teachers, and tell me any that Support Teachers in this school participated in. Check ALL that respondents mention. __How to teach other teachers (e.g. how to teach adult peers e.g.) __How to do peer to peer observation (e.g. Reciprocal Teaching Technique) __How to teach the Revised Curriculum for grades 1-6 __The integrated approach to teaching grades 1-3 __How to carry out Continuous Assessment __How to teach Cooperative Learning __Visualization in Participative Programs (VIPP) Methodology __Communication/Listening __Motivation __None NHP NHP Comp Yes Yes Teach teachers 36 13 Peer to peer 40 10 Teach Revised 54 31 Integrated 59 27 Continuous 59 42 Cooperative 59 24 Visualization 31 7 Communication 34 10 Motivation 35 12 none 0 1 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 23 Q40. During the last school year, 2004-2005 including the summer, about how many days of training did each Support Teacher receive? Answer only for Support Teachers for Grades 1-6 who were in this school during 2004-2005. Math Support Teacher _________________ days Reading Support Teacher _______________days Assessment Coordinator ________________days Other Support Teacher _________________days Math Supp School Type ort Teacher NHP Comp Total 0 12.2% 38.5% 18.5% 1 2.4% 7.7% 3.7% 2 7.3% 7.7% 7.4% 3 9.8% 15.4% 11.1% 4 9.8% 7.7% 9.3% 5 34.1% 15.4% 29.6% 6 2.4% 1.9% 8 4.9% 3.7% 10 7.3% 7.7% 7.4% 15 4.9% 3.7% 25 2.4% 1.9% 35 2.4% 1.9% Total 100.0% 100.0% 100.0% School Type Reading Support NHP Comp Total 0 6.8% 31.3% 13.3% 1 2.3% 6.3% 3.3% 2 6.8% 31.3% 13.3% 3 20.5% 6.3% 16.7% 4 9.1% 6.3% 8.3% 5 25.0% 6.3% 20.0% 6 2.3% 1.7% 8 6.8% 5.0% 10 11.4% 6.3% 10.0% 15 6.8% 5.0% 16 6.3% 1.7% 35 2.3% 1.7% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 24 School Type Assessment Coordinator NHP Comp Total 0 13.8% 6.2% 1 17.2% 7.7% 2 19.4% 10.3% 15.4% 3 25.0% 10.3% 18.5% 4 13.9% 20.7% 16.9% 5 25.0% 10.3% 18.5% 6 5.6% 3.4% 4.6% 8 3.4% 1.5% 10 8.3% 10.3% 9.2% 15 2.8% 1.5% Total 100.0% 100.0% 100.0% School Type Other Support NHP Comp Total 0 14.3% 37.5% 24.3% 1 4.8% 12.5% 8.1% 2 4.8% 12.5% 8.1% 3 9.5% 12.5% 10.8% 4 9.5% 12.5% 10.8% 5 33.3% 18.9% 6 6.3% 2.7% 8 9.5% 5.4% 10 9.5% 6.3% 8.1% 15 4.8% 2.7% Total 100.0% 100.0% 100.0% SECTION VII: SCHOOL-BASED TRAINING BY OUTSIDE SPECIALISTS SAY: Several different types of in-service teacher training have been offered to teachers of primary grades in Jamaica. This question refers to training that takes place in the school, or school-based training. In-service training can include supervised practice and feedback, demonstrations, workshops and much more. We are asking about training that was provided by specialists who came from outside the school, such as regional or district office teacher trainers, project teacher trainers, resource teachers from other schools or other specialists who were not directly teaching at this school. For these questions, we are asking about training opportunities for regular classroom teachers, not for Support Teachers. Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 25 Q41. In the past 5 years, have regular teachers of Grades 1-6 in this school participated in any school-based inservice teacher training provided by outside specialists? Circle ONE only. (a) Yes (b) No. If no, skip to Section VIII School Type NHP Comp Total No 10.1% 39.4% 24.4% Yes 89.9% 60.6% 75.6% Total 100.0% 100.0% 100.0% Q42. What types of school-based inservice training for mathematics in Grades 1-6 have outside specialists provided to regular teachers in this school? Please review the list of types of training and indicate any that teachers in Grades 1-6 have received. Check ALL that respondents mention. __Clinical Supervisory Practice (specialist consults with teacher, observes teaching and provides verbal/written/printed feedback) __Demonstration of teaching strategies by mathematics specialist __Workshops conducted by mathematics specialist __Unannounced observation of teaching and feedback by MOEYC education officer (territorial, district or regional) __Continuous Assessment __Cooperative Learning __Other_______________________________________________________ __ None for primary teachers NHP NHP Comp Yes Tes Clinical Supervisory 48 3 Demonstration 56 8 Workshops 46 14 Observations 50 14 Continuous 40 16 Cooperative 36 11 Other 6 2 None 2 10 Q43. What types of school-based inservice training for literacy in Grades 1-6 have outside specialists provided to regular teachers in this school? Please review the list of types of training and indicate any that teachers in Grades 1-6 have received. Check ALL that respondents mention. __Clinical Supervisory Practice (specialist consults with teacher, observes teaching and provides verbal/written/printed feedback) Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 26 __Demonstration of teaching strategies by literacy specialist __Workshops conducted by literacy specialist __Unannounced observation of teaching and feedback by MOEYC education officer (territorial, district or regional) __Continuous Assessment __Cooperative Learning __Other__________________________________________________________ __ None for primary teachers NHP NHP Comp Sum Sum Clinical Supervisory 50 7 Demonstration 51 22 Workshops 39 24 Observations 48 16 Continuous Assessment 37 18 Cooperative 40 12 other 5 4 None 2 10 SECTION VIII: TRAINING PROVIDED BY SUPPORT TEACHERS SAY: This section asks about the types of support that Support Teachers (Resource Teachers, Coordinators) in this school provide to other teachers. It asks about two kinds of support: first, to teachers in this school, and second, about support to teachers in other schools. Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 27 Q44. Is there a Mathematics Resource Teacher/Math Coordinator in this school? Circle ONE only. (a) Yes (b) No. If No, skip to Q47 School Type NHP Comp Total No 11.6% 56.1% 33.3% Yes 88.4% 43.9% 66.7% Total 100.0% 100.0% 100.0% Q45. With which specific topics in mathematics has this teacher helped other teachers in this school? Please look at this list and tell me all the topics that the Math support teacher has helped other teachers with. CHECK ALL topics that respondents mention __Making Mathematics Fun __Geometry __Statistics/Probability __Measurement __Numbers and Algebra __Other___________________________________________________________ NHP NHP Comp Yes Yes Math Fun 51 17 Geometry 48 20 Statistics 34 18 Measurement 47 22 Numbers & Algrbra 37 15 Other 10 2 Q46. Has this teacher provided any workshop for teachers in other schools? Circle ONE only. (a) Yes. Please describe the most recent workshop_________________________ (b) No School Type NHP Comp Total No 69.6% 90.9% 80.0% Yes 30.4% 9.1% 20.0% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 28 Q47. Is there a Literacy Resource Teacher/Literacy Coordinator in this school? Circle ONE only. (a) Yes (b) No. If No, skip to Q50 School Type NHP Comp Total No 8.7% 53.0% 30.4% Yes 91.3% 47.0% 69.6% Total 100.0% 100.0% 100.0% Q48. With which specific topics in literacy has this teacher helped other teachers in this school? Please look at this list and tell me all the topics that the Literacy support teacher has helped other teachers with. CHECK ALL topics that respondents mention __Emergent Literacy __Use of Diagnostic Teaching of Reading __Whole Language __Phonemic Awareness __Miscue Analysis __Language Experience Approach __Reading Writing Connection __Writing Assessment __Portfolios and Journals __Literacy Window __Other___________________________________________________________ NHP NHP Comp Yes Yes Emergent 23 8 Diagnostic 42 16 Language 33 14 Phonemic 42 21 Miscue 21 10 Language exp 46 21 Reading 52 20 Writing 45 19 Portfolios 57 26 Literacy 50 15 other 5 1 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 29 Q49. Has this teacher provided any workshop for teachers in other schools? Circle ONE only. (a) Yes. Please describe the most recent workshop_________________________ (b) No School Type NHP Comp Total No 55.1% 92.4% 73.3% Yes 44.9% 7.6% 26.7% Total 100.0% 100.0% 100.0% Q50. Is there an Assessment Coordinator in this school? Circle ONE only. (a) Yes (b) No. If No, Skip to Q 52 School Type NHP Comp Total No 17.4% 13.6% 15.6% Yes 82.6% 86.4% 84.4% Total 100.0% 100.0% 100.0% Q51. Has this teacher provided any inservice training on assessment for other teachers in this school? Circle ONE only. (a) Yes (b) No School Type NHP Comp Total No 30.4% 25.8% 28.1% Yes 69.6% 74.2% 71.9% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 30 Q52. How do teachers in this school help each other improve classroom skills? CHECK ALL that respondents mention but DO NOT READ list __ Observing other teachers in this school (Intra school classroom observation) __ Observing teachers in another school (Inter- school classroom observation) __ Sharing of best practices through workshops __ Common planning time __Other: Please describe___________________________________________ School Type Number of responses NHP Comp Total 1 23.2% 27.3% 25.2% 2 34.8% 47.0% 40.7% 3 31.9% 18.2% 25.2% 4 8.7% 7.6% 8.1% 5 1.4% .7% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 31 SECTION IX. COMMUNITY AND PARENT TRAINING SAY: This section asks about training that has been provided to parents of children in this school and to other community members who are associated with this school. Q53. What types of training have been offered to parents and other community members associated with this school, over the past 5 years? CHECK ALL that respondents mention but DO NOT READ list __Training in school governance and leadership __Parents trained as “parent training coordinators __Parents trained in school governance and leadership __Parents attending the National Parenting Conferences __Parents participating in Parent Participation Facilitation Skills workshop __Other: Please describe__________________________________________ School Type Number of responses NHP Comp Total 0 2.9% 9.1% 5.9% 1 44.9% 77.3% 60.7% 2 26.1% 10.6% 18.5% 3 21.7% 3.0% 12.6% 4 4.3% 2.2% Total 100.0% 100.0% 100.0% Q54. Has anyone been recruited from the local community and trained to sensitize and mobilize the community to support the school ? This could be a Community Participation Facilitator, Parent Representative or Parent Grade Coordinator. Circle ONE only. (a) Yes (b) Yes, but not specially trained (c) No School Type NHP Comp Total Missing 6.9% 1.5% 4.3% No 12.5% 46.3% 28.8% YES 61.1% 19.4% 41.0% Yes but not trained 19.4% 32.8% 25.9% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 32 Q55. Has anyone from the local community (not a school official) made home visits to help develop relationships between parents and the school? Circle ONE only. (a) Yes (b) No School Type NHP Comp Total No 52.2% 65.2% 58.5% Yes 47.8% 34.8% 41.5% Total 100.0% 100.0% 100.0% Q56. How has the local community supported children’s learning in this school, over the past 5 years? Circle ONE only. (a) Some type of competition, such as a math competition, art competition, public speaking competition (b) Other_________________________________________________________ (c) No support has been provided by local community School Type NHP Comp Total Missing 4.2% 3.0% 3.6% No support 16.7% 31.3% 23.7% Other 65.3% 46.3% 56.1% Some 13.9% 19.4% 16.5% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 33 Q57. What sort of training has the School Board received? CHECK ALL that respondents mention but DO NOT READ list __ Workshops from the MOEYC __ Workshops from the National Council for Education __ Other__________________________________ __ None School Type NHP Comp Total MOEYC Workshops No 59.4% 51.5% 55.6% Yes 40.6% 48.5% 44.4% Total 100.0% 100.0% 100.0% W School Type orkshops from NCE NHP Comp Total NCE No 52.2% 74.2% 63.0% Yes 47.8% 25.8% 37.0% Total 100.0% 100.0% 100.0% Other School Type Total NHP Comp No 78.3% 87.9% 83.0% Yes 21.7% 12.1% 17.0% Total 100.0% 100.0% 100.0% School Type None NHP Comp Total No 81.2% 74.2% 77.8% Yes 18.8% 25.8% 22.2% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 34 SECTION X. HEALTH AND NUTRITION Q58. During the past 5 years, what method has been used to identify “special needs” students in this school? CHECK ALL that respondents mention but DO NOT READ list __Grade 1 students screened at National Health Fair __Child’s doctor filled out a medical form __Students screened professionally by psychologist/nurse __Classroom teachers observed and identified students __Don’t know __Other___________________________________________________________ School Type Number of responses NHP Comp Total 1 39.1% 50.0% 44.4% 2 55.1% 45.5% 50.4% 3 5.8% 4.5% 5.2% Total 100.0% 100.0% 100.0% Q59. What sorts of programs have special needs students participated in? School Type Total NHP Comp Breakfast & lunch programme 22.2% 17.9% 20.1% Breakfast programme 12.5% 3.0% 7.9% Feeding programme 15.3% 23.9% 19.4% Instructional programme 6.9% 3.0% 5.0% Lunch programme 13.9% 14.9% 14.4% Missing 6.9% 7.5% 7.2% None 8.3% 10.4% 9.4% Other 4.5% 2.2% PATH programme 13.9% 14.9% 14.4% Total 100.0% 100.0% 100.0% Q60. What types of school meals were available in this school last year (2004/2005)? CHECK ALL that respondents mention but DO NOT READ list __Nutribun and drink __Cooked meal (breakfast or lunch) __ Tuck shop snacks (e.g. patties) __Other ______________________ Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 35 School Type Nutribun & Drink NHP Comp Total No 43.5% 47.0% 45.2% Yes 56.5% 53.0% 54.8% Total 100.0% 100.0% 100.0% School Type Cooked Meal NHP Comp Total No 5.8% 6.1% 5.9% Yes 94.2% 93.9% 94.1% Total 100.0% 100.0% 100.0% School Type Tuck Shop NHP Comp Total No 65.2% 69.7% 67.4% Yes 34.8% 30.3% 32.6% Total 100.0% 100.0% 100.0% School Type Other NHP Comp Total No 94.2% 90.9% 92.6% Yes 5.8% 9.1% 7.4% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 36 Q61. On a daily basis, how regularly were meals served last year (2004/2005)? Please choose one option only. Circle ONE only. (a) Every day or nearly every day (b) About half the time (2-3 days per week) (c) Periodically, depending on resource availability School Type NHP Comp Total Missing 5.6% 3.0% 4.3% 35 1.5% .7% Every Day 90.3% 83.6% 87.1% Half the Time 7.5% 3.6% Periodically 4.2% 4.5% 4.3% Total 100.0% 100.0% 100.0% Q62. Does this school provide a free meal (either breakfast or lunch) for any students? Circle ONE only. (a) Yes (b) No. If No, skip to Section XI School Type NHP Comp Total No 2.9% 3.0% 3.0% Yes 97.1% 97.0% 97.0% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 37 Q63. On average, how many children qualified for a free school meal (breakfast or lunch) each day during the last school year (2004/2005)? _____________________ School Type Total NHP Comp 1-10 21.2% 41.5% 31.3% 11-20 24.2% 27.7% 26.0% 21-30 22.7% 9.2% 16.0% 31-40 6.1% 4.6% 5.3% 41-50 12.1% 1.5% 6.9% 51-60 3.0% 3.1% 3.1% 61-70 3.1% 1.5% 71-80 3.0% 1.5% 2.3% 91-100 3.0% 4.6% 3.8% > 100 4.5% 3.1% 3.8% Total 100.0% 100.0% 100.0% Q64. Has anyone monitored the growth (weight for height, BMI) of any children participating in a free school meal program? Circle ONE only. (a) Yes, someone in the school (such as a school nurse) (b) Yes, someone from the Ministry of Health (c) No School Type NHP Comp Total Missing 6.9% 3.0% 5.0% No 63.9% 83.6% 73.4% Yes in School 15.3% 11.9% 13.7% Yes Ministry of Health 12.5% 1.5% 7.2% Yes New Horizon Project's Nutritionist 1.4% .7% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 38 SECTION XI. COMPUTER USE IN THE SCHOOL SAY: These questions ask about computers in this school. Is there someone else (such as a computer or Information Technology (IT) coordinator) who would be better able to answer these questions? If so, could this person be invited to join the discussion now? Q65. Does this school have computers for student use? Circle ONE only. (a) Yes, and all or most of them are working (b) Yes, but they are down now (c) No School Type NHP Comp Total Missing 4.2% 1.5% 2.9% No 4.2% 50.7% 26.6% Yes But Down 18.1% 19.4% 18.7% yes but only one computer 1.4% .7% Yes Working 72.2% 28.4% 51.1% Total 100.0% 100.0% 100.0% Q66. Does this school have internet access? Circle ONE only. (a) Yes (b) Yes, but it is down now (c) No School Type NHP Comp Total Missing 4.2% 1.5% 2.9% No 47.2% 80.6% 63.3% Yes But Down 9.7% 6.0% 7.9% Yes 38.9% 11.9% 25.9% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 39 Q67. Is there a computer Resource Teacher in this school? Circle ONE only. (a) Yes (b) Yes, there should be, but the position is vacant right now (c) No. If No, Skip to Q69 School Type NHP Comp Total Missing 5.6% 1.5% 3.6% No 19.4% 61.2% 39.6% Yes 65.3% 35.8% 51.1% Yes But Vacant 9.7% 1.5% 5.8% Total 100.0% 100.0% 100.0% Q68. What sort of training does the computer Resource Teacher provide for other teachers? Circle ONE only. (a) Workshops on how to use computers to teach (b) Other.____________________________________________________ (c) No training provided by computer Resource Teacher School Type NHP Comp Total Missing 27.8% 62.7% 44.6% No Training 8.3% 11.9% 10.1% Other 11.1% 11.9% 11.5% Workshops 52.8% 13.4% 33.8% Total 100.0% 100.0% 100.0% Q69. Is there a functioning computer available for school records and other administrative purposes? Circle ONE only. (a) Yes (b) Not functioning now, but it was working in school year 2004-5 (c) No School Type NHP Comp Total Missing 5.6% 1.5% 3.6% No 2.8% 17.9% 10.1% Not Functioning now 9.7% 13.4% 11.5% Yes 81.9% 65.7% 74.1% Yes, not for school record 1.5% .7% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 40 Q70. Has the Jamaica School Administrative System (JSAS) software been installed in a computer at this school, and if so, when? Circle ONE only. (a) Yes. If Yes, Ask “when was it installed”______________________ (b) No. If No Skip to Section XII School Type NHP Comp Total No 7.2% 90.9% 48.1% Yes 92.8% 9.1% 51.9% Total 100.0% 100.0% 100.0% Q71. Is there a JSAS support guide (user’s guide, manual) in this school? Circle ONE only. (a) Yes. If “yes”, ask to see to see the guide (b) No School Type NHP Comp Total No 29.0% 97.0% 62.2% Yes 71.0% 3.0% 37.8% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 41 Q72. Who in this school has received JSAS training? CHECK ALL respondents mention but DO NOT READ list __Principal __Specialist teacher __Other specialist __ Other: Please describe____________________________ __ No one has received JSAS training School Type Principal NHP Comp Total No 66.7% 93.9% 80.0% Yes 33.3% 6.1% 20.0% Total 100.0% 100.0% 100.0% School Type Specialist Teacher NHP Comp Total No 62.3% 100.0% 80.7% Yes 37.7% 19.3% Total 100.0% 100.0% 100.0% School Type Other Specialist NHP Comp Total No 88.4% 100.0% 94.1% Yes 11.6% 5.9% Total 100.0% 100.0% 100.0% School Type Other NHP Comp Total No 40.6% 97.0% 68.1% Yes 59.4% 3.0% 31.9% Total 100.0% 100.0% 100.0% School Type No one NHP Comp Total No 89.9% 97.0% 93.3% Yes 10.1% 3.0% 6.7% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 42 SECTION XII GRADE 4 LITERACY SAY: This last section is specifically about assessment, Grade 4 literacy and the Literacy Summer Camps offered for students after Grade 4. Q73. How many children were enrolled in Grade 4 last year (2004-5)? ____________children NHP NHP Comp Mean Mean 68 53 Q74. How many Grade 4 children in this school were recommended for Literacy Summer Camp last year? ____________children NHP NHP Comp Mean Mean 25 22 Q75. How many of these children actually attended a Literacy Summer Camp (either a day camp or a residential camp) in 2005? ____________children NHP NHP Comp Mean Mean 16 13 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 43 Q76. In what years have Grade 4 students from this school attended a Residential Literacy Summer Camp? CHECK ALL that respondents mention __Summer 2004 __Summer 2005 __No Grade 4 students have attended a Residential Literacy Summer Camp School Type NHP Comp Total Missing 4.2% 1.5% 2.9% No Grade 4 68.1% 67.2% 67.6% Summer 2004 6.9% 9.0% 7.9% Summer 2005 20.8% 22.4% 21.6% Total 100.0% 100.0% 100.0% Q77. How many children took the Grade 4 literacy posttest after Literacy Summer Camp? ______________children NHP NHP Comp Mean Mean 16 12 Q78. How many children will repeat Grade 4 this school year (2005-6)? ______________children NHP NHP Comp Mean Mean 4 3 Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 44 Q79. How have teachers used the tests and results of the Grade 4 Literacy test in planning to teach? CHECK ALL that respondents mention __to group students by ability within classrooms __to help with streaming in 5th grade __Other: _____________________________________ School Type NHP Comp Total Missing 5.6% 3.0% 4.3% Other 25.0% 20.9% 23.0% To Group students 62.5% 65.7% 64.0% To help with 5th Grade 6.9% 10.4% 8.6% Total 100.0% 100.0% 100.0% Q80. How have Grade 4 teachers used the tests and results of the Grade 3 Diagnostic test in planning to teach? School Type NHP Comp Total Assess Strength & Weakness 62.5% 53.7% 58.3% Did not use the test 2.8% 6.0% 4.3% Group for instruction 26.4% 31.3% 28.8% Missing 4.2% 1.5% 2.9% Not used 2.8% 6.0% 4.3% Used to adjust teaching 1.4% 1.5% 1.4% Total 100.0% 100.0% 100.0% Q81. Do 4th Grade teachers receive their students’ individual scores from Grade 3 tests? Circle ONE only. (a) Yes (b) No School Type NHP Comp Total No 17.4% 19.7% 18.5% Yes 82.6% 80.3% 81.5% Total 100.0% 100.0% 100.0% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 45 Q82. How have teachers used the tests and results of the Grade 1 Readiness Assessment in planning to teach? School Type NHP Comp number Percent number Percent Assess Strength and Weakness 42 58.3% 29 43.3% Do not use data 1 1.4% 1 1.5% Group for Instruction 15 20.8% 23 34.3% Missing 5 6.9% 1 1.5% Used to adjust teaching 9 12.5% 13 19.4% Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute. Jamaica AED School Survey ID __ __ __ __ __ __ Page 46 Question 83 Do you have a copy of the 2005 Grade 4 Literacy Test Class Report form(s) for this school? A. Yes B. No School Type NHP Comp Total No 10.1% 7.6% 8.9% Yes 89.9% 92.4% 91.1% Total 100.0% 100.0% 100.0% SAY: Do you have a copy of the results of the 2004/5 Grade 4 Literacy test, that is, a copy of the form that was sent to the MOEYC? If so, may I please see it, so that I can complete the attached form. COPY NUMBERS FROM THE FORM INTO THE ATTACHED BLANK FORM. SAY: Thank you very much. We will send you a summary of the results from this survey when it is finished. Jamaica MOEYC, USAID and Academy for Educational Development School Survey. October 2005. All rights reserved. Do not cite, duplicate or distribute.