EVALUATION AND RECOMMENDATIONS FOR STRENGTHENING AND EXTENDING THE NEW HORIZONS FOR PRIMARY SCHOOLS PROJECT IN JAMAICA Submitted to USAID/JAMAICA OFFICE OF GENERAL DEVELOPMENT KINGSTON, JAMAICA Submitted by AGUIRRE INTERNATIONAL 1156 15TH STREET, NW, SUITE 1000 WASHINGTON, DC 20005 Prepared by Richard Dye, Team Leader Joan Jennings, Ph.D. Clement Lambert, Ph.D. Barbara Hunt, Ed.D. Gerald Wein This project assessment was conducted by Aguirre International under GEM Task Order #805 with USAID/Jamaica, Contract Number FAO-1-99-00-00010-00, dated March 2, 2000. The findings, conclusions, and recommendations are those of its authors and do not necessarily reflect the views and/or policies of USAID/Jamaica. July, 2002 Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Acknowledgments · ii ACKNOWLEDGMENTS The success of this review of the New Horizons for Primary Schools Project for USAID/Jamaica is due to the cooperation and collaboration of a great many dedicated people. · Mission Director, Mosina Jordan, who gave generously of her time and expressed strong support for the project. · The Mission staff who directed and supported the project, Ms. Claire Spence and Ms. Joan Davis, deserve special thanks for the guidance, insight, and information they so freely provided, as well as for their assistance with myriad practical arrangements. · Ministry officials and staff who very graciously made themselves available for interviews and questions. · The Chief of Party for Juarez & Associates, Dr. Ernest O’Neil, and his excellent staff who cooperated fully with information, suggestions, and other assistance as needed. · The NHP principals, teachers, and staff, who are on the front lines of the project, were most helpful in sharing their experiences and their expertise. · Our thanks also to the students, parents, school board and PTA members, and others in the various school communities who participated in this study. May their dreams of providing quality education to current and future generations of the community’s children become reality. · Last, but not least, very grateful appreciation is given to all the people of Jamaica who received us so graciously and whose dedication, hard work, and desire for educational improvement and reform inspires us all. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Table of Contents · iii TABLE OF CONTENTS Acknowledgments.................................................................................................................ii Acronyms ..............................................................................................................................iv New Horizons Interventions.................................................................................................v Executive Summary ..............................................................................................................vi I. Introduction.......................................................................................................................1 II. USAID’s Assistance Approach......................................................................................6 III. Findings..........................................................................................................................9 IV. Results ...........................................................................................................................30 V. Lessons Learned.............................................................................................................36 VI. Conclusions and Recommendations for New Horizons Project...................................38 VII. Follow-on Recommendations.......................................................................................42 VIII. Unresolved Issues......................................................................................................45 Annexes Annex 1: List of Contacts....................................................................................................1 Annex 2: Site Visit Reports.................................................................................................6 Annex 3: Focus Group Reports...........................................................................................28 Annex 4: Interview Protocols..............................................................................................33 Annex 5: Health and Nutrition............................................................................................69 Annex 6: Education Management Information System.....................................................76 Annex 7: Parent and Community Involvement...................................................................82 Figures and Tables Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Acronyms · iv ACRONYMS CARICOM Caribbean Community CBP Coalition for Better Parenting CEE Common Entrance Exam CETT Center for Excellence in Teacher Training (USAID) CFNI Caribbean Food and Nutrition Institute CLIN Contract Line Item Number DFID Department for International Development (DFID/UK) EMIS Education Management Information System GSAT Grade Six Achievement Test IDB Inter-American Development Bank IRA International Reading Associa tion GOJ Government of Jamaica J&A Juarez and Associates JAASP Jamaica All Age Schools Project (DFID) JAMAL Jamaica’s Movement for the Advancement of Literacy JSAS Jamaica School Administrative System JBTE Joint Board of Teacher Education JPL Jamaica Labour Party JSIF Jamaica Social Investment Fund KWL Literacy Strategy (What do I know, what do I want to know, and what have I learned?) LRT Resource Teacher in Language Arts MIS Management Information Systems MOEYC Ministry of Education, Youth and Culture MRT Resource Teacher in Mathematics NAP National Assessment Program NCE National Council on Education NHP New Horizons for Primary Schools Project PEAP Primary Education Assistance Project (USAID) PEIP Primary Education Improvement Project (IDB) PESP Primary Education Support Project (IDB) PDU Professional Development Unit (MOEYC) PIU Project Implementation Unit (NHP) PNP People’s National Party PTA Parent Teacher Association RT(s) Resource Teacher(s) ROSE Reform of Secondary Education Project (WB) SDP School Development Plan SDT School Development Team TEO Territorial Education Officer (MOEYC Regional Offices) UAP Uplifting Adolescents Project USAID US Agency for International Development UWI University of the West Indies WB World Bank (International Bank for Reconstruction and Development) WFP World Food Programme Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International NHP Interventions · v NEW HORIZONS INTERVENTIONS 1. Developing innovative mathematics and literacy programs 2. Providing in-service teacher training in reading and mathematics 3. Providing governance and leadership training for schools, communities, parents 4. Offering parent education and training 5. Facilitating selective nutrition and health programs 6. Providing reading and mathematics materials 7. Establishing computer use in school and training teachers in educational technology 8. Training resource teachers 9. Integrating databases using MIS 10. Improved school management through EMIS Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Executive Summary · vi EXECUTIVE SUMMARY This report presents the findings of the Aguirre International New Horizons for Primary Schools Project (NHP) evaluation team, which visited Jamaica from June 9 to July 6, 2002. The team’s charge was to evaluate the project from its inception to date and to present recommendations to USAID regarding possible follow-on activity. The NHP is a 7-year, $10.2 million1 project designed to increase the quality of teaching, raise attendance rates, and improve management in 72 low-performing, at-risk primary schools throughout the country. Ten interventions were developed to accomplish project goals. (See list of interventions on preceding page.) The project began in mid-1998 and is now completing its fourth year. Key Findings The team’s overall assessment of the project is positive. In spite of funding delays and other start-up complications, clear evidence of the project’s impact was found in the 16 NHP schools (more than 20% of the total) that the team either visited or contacted through focus group meetings. The learning environment in NHP schools was also found to be superior to that of the three non-NHP schools visited. Analysis of test scores and formative and process evaluations to date is also positive, and the team is confident that, if the effort is sustained and strengthened, significantly better results will be seen in the future. Project management by the various partners was also generally strong, and working relationships have been excellent. On the other hand, much work remains to be done to meet project goals, and completion of the job will require a second phase. Key Recommendations The team recommends the current project’s continuation through September 2004. During that period, the evaluation team strongly supports the priority that the project apparently plans to give to increasing technical support to NHP resource and regular teachers and to implementing the long-delayed educational technology component. With respect to project support to RTs and teachers, the idea of creating a new cadre of NHP associates appears to be a sound approach, assuming that selection, training and supervision are sound. With respect to educational technology, the proposed creation of a multi-media center and pilot project is a reasonable approach to the establishment of a longer-term capacity to support the appropriate use of technology in the classroom. However, the team recommends that NHP not abandon or further delay the intended provision of computers to classrooms so that NHP teachers and students have access to them. The development of a complementary, phased, multi￾year strategy to accomplish this is recommended, with the first phase to take place during the 1 This figure includes USAID’s management costs and funds provided to the Jamaica Ministry of Education, Youth, and Culture and the technical assistance contractor, Juarez & Associates. Two other, related activities have also been funded by the project. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Executive Summary · vii remaining life of the current project. The team also offers a number of recommendations about other aspects of the program. The team further recommends that USAID support a second phase of NHP, in the form of a 5- year follow-on project, starting in FY 2005. NHP II would continue and build on NHP I but would incorporate important new elements: ÿ An increased emphasis on the early grades; ÿ A focus on two critical “transition” years, that is, from pre-school to first grade and from sixth to seventh grade; ÿ Development of a new category of leadership schools drawn from the best of the current schools and their incorporation in the management model, as mentors for continuing and new schools; ÿ An increase in the total number of schools to approximately 100, a level that would give the project greater leverage on GOJ educational policy and practice; ÿ Expansion of the partnership model to include teacher training colleges and MOEYC regional offices; ÿ A new teacher-centered research component; and ÿ The creation of a small grants competition to assist with modest facilities improvement projects targeting improved learning conditions. In addition, the implementation of the complementary educational technology strategy proposed for NHP I would continue under NHP II. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter I: Introduction · 1 CHAPTER I INTRODUCTION A. Overview of Jamaican Primary and Basic Education 1. State of Primary and Basic Education Primary and basic education have existed since the early years of post-emancipation Jamaica. Currently, pre-school enrollment is 78% of the eligible population. Primary schools enroll most children (98%) at the K-6 level. The New Horizons for Primary Schools Project (NHP) was designed in a context where the dominant concern was the low quality of primary education, specifically: · Large numbers of children completing primary school below a grade six attainment standard. · Children from less fortunate economic circumstances failing to meet minimal literacy and numeracy standards at the end of their primary schooling. · Inability of primary schools to meet the individual learning needs and styles of children. Public and private schools exist in the Jamaican context at all levels of education. The top private primary schools (preparatory schools) are the most successful in terms of government examination results, while at the secondary level, public schools are the best performers and most in demand. Despite the greater level of success in the preparatory schools, public primary schools have two major strengths, which are a greater number of trained teachers and better salary and fringe benefit structures. The majority of Jamaicans cannot afford fees charged by preparatory schools. Fees are not charged in public primary schools. Primary education has long been a priority of governments in Jamaica. Compulsory attendance at the primary level was instituted more than two decades ago, and performance targets for increased numbers of literate and numerate primary school leavers have been established. Despite the attention given to primary education, successive policymakers and educators have highlighted the need for increased levels of performance of primary school leavers. In a response to this need, primary education had benefited from bilateral and multilateral contributions, designed to improve the achievement levels of primary schoolchildren. It is within this framework that the New Horizons Project was conceptualized and implemented. 2. Reform Efforts Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter I: Introduction · 2 Aguirre International Jamaica’s Ministry of Education, Youth and Culture (MOEYC) has introduced a number of reforms in the education sector since the design and implementation of the New Horizons Project. These changes, embodied in the Ministry’s February 2001 White Paper, include a new national primary school curriculum and the national literacy improvement initiative. The White Paper also outlines a number of new policies and procedures for achieving quality education in Jamaica. Notable among these policies is the possibility for retention of students based on their performance on a grade four literacy test and the requirement that each school prepare a school development plan (SDP), in which it is required to formulate and execute a plan in which annual targets are set and matched to resources, demand and potential. The White Paper established new targets of 100% pre-school enrollment of children aged 4-5 by 2003, 90% average daily attendance in primary schools, and a teacher/student ratio of 1:35 (Grades 1-2 at 1:30) at the primary level by 2005. Other targets include the creation of a nation-wide program of early childhood care and stimulation for children between birth and age 4 by August 2001; demonstration of full literacy by 80% of all Grade 6 school leavers by 2003, and five years of secondary education for all students entering Grade 7 by 2003. The new national primary curriculum replaces one that was in use in primary schools for over twenty years. This new curriculum (currently in various stages of implementation) demands changes in teaching methodologies and in the role of the teacher and learners. There is greater emphasis on integration; the grade 1-3 curriculum espouses the teaching of topics in an integrated way, while Grades 4-6 are divided into separate subject areas with an emphasis on inter- and intra-subject integration. In response to national concerns of substandard literacy performance, the MOEYC has also introduced a new literacy improvement initiative based on broad-based consultation with stakeholders. The NHP started in a period in which national assessment policies and structures were also in transition. The National Assessment Program (NAP), funded by the Inter-American Development Bank (IDB), piloted and implemented new diagnostic and achievement measures for Jamaican primary schoolchildren at three levels. These were the Grade One Readiness Inventory (a test of children’s pre-reading competencies), the Grade Three Diagnostic Test and the Grade Six Achievement Test (GSAT). Like the Common Entrance Exam (CEE) that it replaced, the GSAT determines placement of children into post-primary schools. A grade four literacy test was later developed to determine whether students needed additional assistance before proceeding to grade five. The tests were institutionalized nationally between 1999 and 2000. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter I: Introduction · 3 These reforms occurred in the context of a changing policy direction towards assessment as a process rather than a product. As a result there is now a school-based input in these tests – teachers administer and score the tests from grades 1 – 4 and provide school continuous assessment grades that are incorporated into students’ GSAT scores. These Jamaican education development goals and policies for early childhood, primary, and secondary education generally reflect the Caribbean Community (CARICOM) consensus and are consistent with policies in effect in Central and South America. Current goals and policies enjoy wide support in the country from, among others, the current People’s National Party (PNP) Administration, the Minister of Education, MOEYC staff, the major opposition party (the Jamaica Labour Party or JLP), the universities, and the Jamaica Teacher’s Association (JTA). The chances are good that the fundamental principles and current major projects would not be seriously affected by a change of government, should that be the result of new elections widely expected to take place in the late summer or early fall. Under Jamaican civil service regulations, the great majority of MOEYC personnel would remain in their posts. 3. Challenges Jamaica’s challenge is to translate these sound goals and policies into reality through a combination of adequate funding, sound program and project planning, availability of the needed people and skills, competent project management, and strong, sustained leadership. With the economy struggling, the prospects for any significant increase in the education budget in the foreseeable future are not good. Productivity increases in the education sector, while a worthwhile goal, will at best take time to achieve. The Ministry’s quality and capacity are uneven. There are many highly productive staff and also quite a number of low performers. As elsewhere in the developing world, the most competent people are over-worked and spread thinly over many tasks. Regulations and procedures are cumbersome and time-consuming, especially in the financial area. The regional and territorial education offices, which are assigned a critical role in the oversight and technical backup of the schools, do not have the resources to respond comprehensibly and effectively to the many needs of the schools in their areas. The teacher training colleges are of mixed quality and have many staffing, academic, and management needs. Education, teaching and research capacity outside the Ministry is concentrated in the School of Education at the University of the West Indies (UWI) at Mona. The School’s programs are of good quality, and its staff participate extensively in MOEYC activities. Other than the university, the team has not been able to identify any significant private sector or civil society educational research, technical assistance, or advocacy institutions. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter I: Introduction · 4 Aguirre International B. Principal Education Donors The IDB, and to a lesser extent the World Bank (WB), are the largest education donors, by a wide margin, with the IDB concentrating on primary education and the WB on secondary education. Currently, a large, 5-year, $31.5 million IDB loan (plus $8 million of Government of Jamaica (GOJ) counterpart) is supporting a comprehensive primary education improvement effort, concentrated on capacity building at the central and regional levels. An extensive training program for principals and selected pilot school projects is also included. The WB’s principal effort is the ongoing Reform of Secondary Education (ROSE) project. USAID, with its major focus on primary education and the educational needs of out-of-school, at￾risk youth, is the largest bilateral education donor. Other important bilateral contributors are the British Department of International Development (DFID), which is working with All Age schools, primarily in remote, rural areas, and the Canadian International Development Agency (CIDA), which, together with USAID, is supporting an important new UNICEF initiative at the early childhood level to improve the readiness of children entering the first grade. Coordination among education donors, and between donors and the GOJ and the MOEYC, is spotty and uneven. So far as the team has been able to ascertain, currently there is no primary or basic education consultative or working group that meets on a regular basis. C. Background to the Evaluation USAID Jamaica has had a long history of interest in improving basic education, with emphasis on the primary years, Grades 1-6. Prior to the launching of NHP, the Mission funded two Primary Education Assistance Projects (PEAP I and PEAP II), which strengthened important capacities in the primary education sector. New Horizons built on these efforts, applying these and other experiences to develop and implement strategies for improving literacy and numeracy learning in 72 at-risk schools throughout the country (9% of all primary schools). NHP began in 1998 and is now finishing its fourth year. The project expires on September 30, 2004. The technical assistance contract with Juarez and Associates is scheduled to end in June 2003. Aguirre International was asked to conduct an evaluation of NHP’s work and results to date. The Aguirre scope of work also included responsibility for providing USAID with recommendations regarding follow-on to NHP. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter I: Introduction · 5 D. Evaluation Team Aguirre International’s efforts on this evaluation were managed by Tom Judy, a senior staff member in the firm’s Washington office. Team members included: · Richard Dye, Education Administration Specialist/Chief of Party · Barbara Hunt, Evaluation/Instructional Systems/Education Specialist · Joan Jennings, Evaluation/Community Participation/Education Specialist · Clement Lambert, Evaluation/Instructional Systems/Education Specialist · Gerald Wein, Evaluation Specialist E. Methodology The means used by the Aguirre team for gathering information and insight into the various components of NHP included: · Review of documents, NHP formative evaluations, and reports; · Interviews with participants and stakeholders; · Site visits; and · Focus groups. A sample of eight NHP and three non-NHP schools was selected for intensive on-site investigation, often involving more than one visit to each site. (See Annex 2, Site Visit Reports.) A ninth NHP school subsequently was added to the breakfast program review. Included were inner city, urban, and rural schools exhibiting a range of sizes, type, and performance levels (though all NHP schools are low-performing, by definition). To broaden the base of the team’s school contacts, focus group meetings were held in Regions 1 and 6, at which comments and suggestions were obtained from representatives of seven additional NHP schools, again of a range of sizes and types. Participants included: Principals, Resource Teachers (RTs), School Board and PTA members, and parents. In addition to the site visits and focus groups, meetings were held with USAID staff, MOEYC personnel in the relevant units, contractor staff, parents, community members, school administrators, teachers, and students. Contacts were made as well with other key donors and selected projects addressing primary education. In total the team met with and obtained information from approximately 250 people. (See List of Contacts, Annex 1.) Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter II: USAID’s Assistance Approach · 6 Aguirre International CHAPTER II USAID’s Assistance Approach A. Country Strategy The goals of USAID’s multi-year strategy in Jamaica (FY 2000 through FY 2004) are promoting economic growth, protecting Jamaica’s unique environment, on which much of the economy depends, better preparing Jamaica’s youth for the future through improved adolescent health and improved basic education, and promoting good governance. Funding is approximately US$13.5 million per year (of which some $1.5 to $2.0 million goes to education). Strategic objectives include: · SO 1: Improving the business environment for developing the small, medium, and micro-enterprise sectors; · SO 2: Improved quality of natural resources; · SO 3: Improved reproductive health of youth; and · SO 4: Increased literacy and numeracy among targeted Jamaican youth. Improved basic education is viewed by both USAID and GOJ as essential to the success of Jamaica’s efforts to compete in a global economy. Significant improvements in quality educational attainment, especially among poorer Jamaicans, are needed if economic goals are to be met. B. Education Strategy and Program The USAID Jamaica’s education sector program is targeted on children ages 6 to 14. The decision to focus on this group is based on the fact that many of the issues of poor performance, absenteeism, dropout, and equity that affect Jamaican children arise in the pre-school years and become critical in the primary school setting. Taking these facts into account, a five-year strategy for improving learning was developed for FY 2000 to FY 2004. It addressed two top priority basic education areas – literacy and numeracy. The major project is NHP. A separate but closely related project assists the GOJ with a revision of the primary teacher education curriculum. Another activity is the Uplifting Adolescents Project (UAP II), through which a package of remedial education services is being provided to at-risk, out-of-school youth, with the assistance of a group of Jamaican social action NGOs. A complementary USAID initiative is the planned Caribbean Center for Excellence in Teacher Training (CETT). This center, one of three established in the western hemisphere by presidential initiative, will Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter II: USAID’s Assistance Approach · 7 focus on improvement of reading instruction in the first three grades. The Caribbean CETT is to be hosted by the Joint Board of Teacher Education (JBTE), at the UWI in Kingston. It will have five principal components: a) Teacher training to improve pedagogical skills, especially in reading; b) Development of diagnostic tools for assessing student performance in reading; c) Development of teaching and learning materials focused on improving reading instruction in the classroom; d) Applied research to enhance teacher training, tools, and materials; and e) Information and communication technology (designed to support the other four). This Center will provide obvious opportunities for collaboration and interchange of ideas with NHP. As a school-based project focused on disadvantaged and low performing children, NHP is serving to bring the accumulated experience with Jamaican basic education to bear on real problems in the real world. While it was designed to develop, test, and evaluate innovative teaching, management, and community strategies at the local level, it is not just another pilot effort. The project is directly assisting a significant number of Jamaica’s problem primary schools (9%) and at-risk students and, if the effort is sustained, has excellent potential for expansion and replication. It is the logical culmination of the Mission’s basic education improvement strategy. C. USAID’s Partners in the Education Sector USAID’s principal Jamaican education partner is the MOEYC. In most cases, including NHP, projects are jointly managed, with shared responsibilities and funding. The Ministry is fully consulted on project plans and proposals and participates in project design. Projects, thus, respond to Jamaican, as well as USAID priorities, and, for the most part, operate within MOEYC policies and procedures. This mode of operation increases the likelihood that successful projects will be sustained and replicated. Other Jamaican partners are brought into projects in various ways. One is to provide technical and management support for specific project components, such as the assistance provided through the Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter II: USAID’s Assistance Approach · 8 Aguirre International MOEYC by the National Council for Education (NCE) for NHP governance and leadership training and parent education and training. Another is contracting with a local partner to manage a project, as in the case of the umbrella NGO People’s Action for Community Transformation (PACT), which is implementing UAP II. Jamaican partners also may be involved in evaluation. In all cases, stakeholders are consulted and, as appropriate, involved in project design and evaluation. Outside contractors have been employed to support selected projects, normally those which require unusually strong technical and administrative support, especially during the initial years, rapid access to a wide range of international specialists and training capacities, and expertise in program and process evaluation. J&A and its sub-contractor, Interamerican Development Advisory Services (IDEAS), for example, are assisting NHP, while the American Institutes of Research (AIR) and JBTE are working with the MOEYC on the undergraduate teacher education primary level curriculum revision project. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 9 CHAPTER III Findings A. Overview of NHP Design and Development to Date Within the framework of USAID’s SO 4, to increase literacy and numeracy among targeted Jamaican youth, the NHP was designed to implement a school-based model of intervention that synthesizes and builds on successful experience to improve the language arts and mathematics performance of 72 of Jamaica’s poorer performing schools. The intent was to create a model to be applied eventually to all schools serving needy children. Within this context, a menu of ten interventions was developed. (See list at beginning of the report.) The 72 project schools were to choose those of the project’s ten interventions that best responded to their needs. The project is organized around a partnership model. Specific project activities were assigned to the institutional contractor (Juarez and Associates), the Professional Development Unit (PDU) of the MOEYC, and, under the MOEYC’s direction, the NCE. The following table shows the allocation of responsibilities for the project interventions. Table 1: Lead Institutions for NHP Interventions Implementing Agency Institutional Contractor MOEYC/PDU MOEYC/NCE Intervention # 1, 2, 5, 6, 7, 9, 10 8 3, 4 While the interventions are presented separately, at the school level they are intended to be implemented in an integrated fashion. Moreover, some interventions are inextricably intertwined, such as Numbers 1, 2, 6 and 8, which deal collectively with providing strategies, materials, and teacher training to improve literacy and numeracy. It is not possible to assess the differential impact of each of these interventions separately. The implementation of the 10-intervention strategy has been strongly affected by factors outside the control of the project. Some of the interventions have been able to proceed in timely fashion, whereas others were affected by difficulties in funding encountered during the early stages of the project. Intervention #7, Educational Technology, was the most delayed, and is only now beginning to get under way. Although it was intended that individual schools would select from among the 10 interventions, in practice all of the interventions except the breakfast program have been applied in all 72 schools. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 10 Aguirre International Allocation of responsibilities, timelines, and distribution of resources were modified during the implementation process. These implementation factors will be discussed further in Chapter IV: Results. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 11 B. School Governance and Management The School Development Plan (SDP) is the focal point of NHP’s approach to school governance. The successful development of such a plan requires strong and effective leadership by a principal who involves stakeholders in a thorough needs assessment of the school and in careful planning to reach common goals. NHP has provided principals with training on how to develop the SDP, and all of the NHP schools visited had SDPs, although these were of varying quality. Many plans were simply a list of objectives, with activities, resources and indicators of success provided. The better plans, however, described the diagnostic assessment undertaken, the weaknesses and strengths of the school, and reasons for the priorities established. Some non-NHP schools visited also had SDPs. In these schools, objectives tended to be more focused on infrastructure and rarely mentioned learning goals, whereas NHP plans emphasized goals for student learning. In NHP schools visited, or those whose staff participated in focus groups, the School Development Team (SDT) members included the principal, one or more Resource Teachers (RTs), one or more PTA members, the School Board Chairman, and sometimes other classroom teachers and community members. Students were rarely included. Principals mentioned that at first many teachers and parents did not understand the importance of the SDP process and found it difficult and time-consuming. However, most interviewees indicated that the SDP is now considered important for the progress of their school; many stated that the SDP provided them with a focus for their efforts, or a “vision,” and a way to know when they have achieved their goals. Many cited specific achievements that were direct results of the SDP. For example, one school had been able to improve attendance on Fridays by the introduction of special Friday afternoon clubs. Parents who were not members of the SDT were generally not aware of the specifics of the SDPs. Many principals and teachers stated that in the future they think it will be important to include more Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 12 Aguirre International stakeholders in the design of the plan. Some who in the past had only consulted parents said they now think parents should participate actively on the team. It is all too easy for an SDP to become just another piece of required paperwork that has no real meaning in the life of a school. Overall, it appears that NHP schools have made considerable progress towards the development of meaningful SDPs that truly make a difference for the learning of their students. Continued efforts will be needed to assist principals in this important process. C. Instructional Strategies for Literacy and Numeracy NHP specialists have introduced a “Menu of Instructional Strategies.” The concept is an important one, as it places decision-making in the hands of the teacher, who should have available not just one method of teaching, but a variety of strategies from which to draw as appropriate for different instructional goals and diverse groups of children. Some of the strategies are specifically designed to improve reading comprehension or problem solving in mathematics. Others cross curricular lines, such as cooperative learning and methods of assessing children’s learning. The strategies reflect current educational research and have been well chosen to stimulate children’s thinking and active involvement in their learning. Classroom visits indicated varying degrees of progress in implementation of the strategies, which were generally consistent with the findings of recent formative evaluations. These evaluations have noted that most of the classrooms observed remain teacher-centered, although improvements in classroom environment have been noted since the inception of the project. The formative evaluations have also found some improvement on measures of teaching quality such as the provision of explanation or expansion of ideas. However, the evaluation team noted considerable variation between teachers and between schools; such variations tend necessarily to be obscured by the averages reported in the formative evaluations. The team’s observations indicated that many teachers are incorporating some of the desired strategies into their teaching, with the result that in many classrooms, the teaching, while still largely traditional in style, now occasionally incorporates such strategies as questioning at higher levels, grouping, and other techniques that encourage student interest and involvement. Many interviewees (parents, teachers, principals, School Board members) expressed the opinion that students are now more interested in school, are more actively involved in their schoolwork, and are more interested in reading. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 13 It is important to note that the physical conditions in which many teachers are working severely limit the extent to which they can apply some of the recommended strategies. The use of chalkboard partitions to divide rooms into 3 or 4 overcrowded classrooms (as many as 60 students per class) provides an environment in which teachers and students cannot hear each other and have little or no space for project work or interest centers. A considerable difference was noted between the NHP and non-NHP schools observed. In the non-NHP schools, most teachers were still using traditional rote-learning methods, and teacher-student affect frequently had a harsher tone. The leadership of the principal was noted as a key factor in implementation of the desired strategies. Those principals who were generally proactive about the project and who understood the strategies appeared to have been more successful in leading their teachers towards the desired changes. D. Other Issues Affecting Learning Use of Time. Time management within the classroom is an important issue, given that the amount of time children spend actively involved in learning is a key factor in their school achievement. Site visits suggested several areas in which children’s time is not being well used. In some schools, considerable time is spent on early morning exercises. Some students are arriving late; during one early morning class, students came trickling in throughout the hour. During classroom time, one teacher spent 15 minutes rearranging desks and reassigning student seating for group work. Within both group work and whole class teaching, many students were observed doing nothing, apparently paying no attention to the lesson. During the week of review prior to testing and the week of testing at the end of the year, it appeared that many students did nothing for much of the time. Although time appeared to be better used in the NHP schools than in the non-NHP schools visited, an increased focus on effective use of time would be a beneficial addition to the project. Classroom Affect and Discipline. The successful implementation of NHP strategies should significantly change the nature of teacher-student interactions. Although teacher-student relationships were more positive in NHP schools than in non-NHP schools visited, there was evidence in some NHP schools of the use of a strap for discipline. Most teachers used students’ names, but some addressed children anonymously (“You there”). Many teachers mentioned the problem of children with emotional or behavioral problems. It would appear that teachers would benefit from training in positive classroom interaction and management procedures, such as the Assertive Discipline system. End of Year Testing. The custom of end of year testing, in which teachers administer tests they themselves have prepared, is well entrenched in Jamaica. Observations of review classes revealed many activities in which children were filling in blanks with memorized material. It appears that teachers are not generally integrating the new strategies and assessment techniques into their end of year testing. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 14 Aguirre International Preschool. As in other countries, Jamaican children enter primary school with varying levels of preparation. Some students enter NHP schools without having any preschool experience. However, even in the case of students who had attended a preschool, there was little evidence of efforts to facilitate the transition between preschool and first grade. Some first grade teachers mentioned that as many as 75% of their students lacked the readiness skills assessed by the first grade inventory, and that they find it extremely difficult to meet the needs of those children. It would appear that an effective linkage is needed between the preschool and first grade curricula, and that more communication and collaboration is needed between the two levels. E. Training Interviewees most often mentioned training and the provision of materials as the two most successful aspects of the project. This section describes the several types of training that have been provided by NHP and its partners. 1. In-Service Training for Classroom Teachers All primary schools in Jamaica are benefiting from the new curriculum. The MOEYC began to introduce the new elementary curriculum to teachers in three-day summer workshops for two grades per year, starting in 2000. In contrast, NHP took on the commitment to pilot the entire Grade 1-6 curriculum from the inception of the project. Therefore, NHP introduced the new curriculum in an initial training session for all its Grade 1-6 classroom teachers at the start of the project. Subsequent NHP training for classroom teachers has been provided in two ways: · Resource Teachers are responsible for providing school workshops on material they have learned, and also for providing advice, support, materials or teaching demonstrations as requested by classroom teachers. · NHP has four subject area specialists, two in language arts and two in mathematics (one of each is an NHP staff person, while the other two are MOEYC staff members). Each is responsible for a certain number of schools (the Project Director also serves in this role for some schools). Each school receives a visit approximately four times per year, during which the specialist may offer workshops, observe classes and provide suggestions, do teaching demonstrations, or offer support in other ways. In general, it appears that training and support of classroom teachers has not been adequate to enable many of them to implement successfully the new teaching techniques sought. The subject area specialists are spread thinly, and there is little opportunity for continuity between the visits to a given school. In some instances, they give classroom demonstrations just for one teacher, because of the difficulty of providing coverage for other teachers’ classes. Several RTs and principals stated that they wished the classroom teachers could all receive the more intensive training the RTs have received both in subject area content and in the menu of teaching strategies. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 15 The specialists commented that they perceive the Territorial Education Officers (TEOs) as allies. They are assigned to visit schools and are also invited to attend NHP workshops. Although they have extensive duties and many schools to cover, TEOs are an important avenue of support for future sustainability of the project. NHP is planning to make a significant addition to its teacher support team by adding 24 “Associates.” These would be highly trained teachers who could serve to assist the specialists to provide significantly more on-site and in-classroom support. 2. In-Service Training for Resource Teachers RTs are full-time classroom teachers who have received special training provided by the MOEYC’s PDU, in collaboration with the NHP Project Implementation Unit (PIU). The training of RTs has been provided during weeklong summer workshops or in workshops in regional clusters. Frequently, international specialists are brought in to participate in the larger training events. Interviewees were extremely positive about the RT training, particularly the recent training in cooperative learning and assessment. A review of the workshop material and evaluation comments provided on NHP workshop evaluations shows that a great deal of material is packed into some of these sessions. For example, in August 2001, RTs received training for one week that included cooperative learning, assessment, and use of the Encarta encyclopedia CD-ROM. A number of workshop participants indicated on the evaluation forms that they did not feel confident about going back and teaching the newly-learned material. These comments suggest the need to provide continued, follow-up training on such topics. Of course, RTs also receive the benefits of the on-site training and support provided by the NHP subject area specialists. Interviewees frequently requested more training in the following areas: · Strategies for handling multi-grade classrooms; · Diagnostic and teaching techniques for handling students of varying abilities, most especially lower achievers; · Further training in use of materials and AV equipment such as overhead projectors; · Training in computers for instructional use with children; · More in-depth training in strategies such as cooperative learning and assessment; and · More specific content training in mathematics. Some principals and RTs urged that classroom teachers be sensitized to the role of the RT. It appears also that RTs might benefit from more training and reflection on their roles and from further training on strategies for training adults (i.e., other teachers). 3. In-Service Training for Principals Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 16 Aguirre International NHP has provided workshops for principals on the development of the SDP. Principals interviewed stated that this training was extremely useful. Several stated that although they initially did not understand either the point of the SDP or how to develop one, they now understand both the purpose and the process of developing an SDP. In addition, the principals, along with some staff members, have received training on the use of the JSAS software. Most reported finding this training useful. (See Section L, Education Management Information System, below for additional comments.) Principals are members of their School Boards, so they will be included in training to be offered by the NCE for School Boards. The NCE is also allocating two staff to visit schools and provide individual support to principals in the SDP process. Process evaluations conducted by NHP staff recommend the provision of more training to principals on pedagogical leadership and supervision. Findings from site visit observations are in accord with the need for training of this type. 4. Training for School Boards, Parents and Community Members The NCE is planning training for School Board members and has prepared a manual to assist principals and School Board members to understand their roles and the relevant legislation and regulations. NCE staff are presently engaged in making the manual more “user-friendly.” Training for parents has been conducted by NCE, and interviewees who attended workshops found the training very useful. Although NCE staff indicated that parent training is now complete, interviews and focus groups emphasized that continued parent training and involvement is necessary and important. No separate training has been offered to community members, although many of them are either PTA members or School Board members. F. Parent Involvement Interviewees in most schools stated that parent participation has increased since the inception of NHP. The 2001 Formative Evaluation reported on the basis of student surveys that parent participation in learning had increased from 36% in 1999 to 54% in 2001. Furthermore, the percentage of schools with PTAs increased from 89% in 1999 to 100% in 2001. Even more important was the finding that the percent of schools in which PTAs were meeting regularly had risen from 33% in 1999 to 94% in 2001. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 17 Parents in the eight NHP schools providing breakfast programs received training in fund raising; interviewees reported that this training was useful to the schools both in increasing parent participation in general, and in raising funds not only for breakfast programs, but for other school purposes. (See Section K, Health and Nutrition.) In spite of these positive findings, many school staff expressed the opinion that more parent involvement is still necessary, and that they must continue to work towards this goal, with a special focus on the contribution parents can make to their children’s learning. An effort was made in the NCE training to establish parent Study Circles, but these were not successful. This effort was considered by some interviewees to have been the least successful aspect of the project. G. Role, Use, and Effectiveness of Resource Teachers Much of the success of the NHP program hinges on the effectiveness of the RTs. Theirs is a difficult role, since they are expected not only to teach full-time, but also to train others in material they have only recently learned themselves, as well as to serve as mentors and supporters to the regular classroom teachers. RTs are specialized either in Mathematics or Language Arts. The smaller schools have one Math RT (MRT) and one Language Arts RT (LRT), the medium-size schools have two of each, and the largest schools now have three of each. Site observations and interviews suggest that the RTs vary in their dedication, level of confidence, and effectiveness. They also vary in their level of training, and a few are pretrained teachers (teachers without teacher training or certification). One NHP document evaluating an RT training session noted that more pretrained than trained RTs expressed a lack of confidence in their ability to return to their schools and train the other teachers in the material they had just been taught. The report recommended that every effort be made to ensure that future RTs are trained teachers. On the other hand, there are individual cases of excellent pretrained teachers. (It should be noted that in the previous USAID project PEAP II, RTs were selected from the national pool of teachers, without regard to which school they taught in. In contrast, the NHP selection process has been constrained by the need to select RTs from among those already teaching in each selected project school.) Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 18 Aguirre International Several RTs interviewed mentioned that they encountered initial resistance from teachers as they tried to step into their new role, and it also appears that they receive varying support from their principals. A few mentioned their feelings of inadequacy or fear in presenting workshops for their colleagues, especially when strategies are new to them and they themselves are not yet sure how to implement them. However, in the best of cases, the RTs are a resource much appreciated in their schools. Some principals stated that they rely heavily on their RTs and turn to them for support. Some RTs are asked to provide workshops at NHP training sessions, and many have also been invited to present workshops at nearby non-NHP schools. Some have been requested by MOEYC to assist in the training on the new curriculum for non-NHP teachers. The problem mentioned most often in connection with the role of RTs is the difficulty they experience in leaving their classes to provide the services requested of them. No substitutes are provided by the MOEYC or by NHP. Sometimes parents fill in for them; sometimes other teachers take their students, and sometimes Youth Service Workers serve as substitutes. No school mentioned using student volunteers from nearby Teachers Colleges. RTs receive no extra pay for their services, and several suggestions were received regarding the need to give them special certification or recognition when they fulfill certain criteria. It was also suggested that they receive official credit for the training received, and that they be paid when asked to provide workshops at other schools or training events. H. Supplementary Materials and Equipment The NHP has provided supplementary materials and equipment to the 72 schools under Intervention #6, which mandates the contractor to “Provide reading and mathematics materials.” The fact that research has proven that “children become literate and numerate by interacting with different types of materials” is used as a rationale for this component. Language arts materials acquired for schools include storybooks, reference materials, and reading methods texts for literacy improvement, while supplementary math materials include content and methodology texts, measuring instruments, calculators Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 19 and manipulatives. Technological devices are also considered to be supplemental materials. (These are discussed below in Section I, Educational Technology.) The project report on supplementary materials (2001) outlines an inclusive decision-making process that involves a range of relevant stakeholders (e.g., the MOEYC’s Core Curriculum Unit and Education Management Information System (EMIS) specialists). Teachers also stated that they were consulted on the kinds of materials they would like for their classes. While this consultative process is highlighted in project documents, some teachers complained that they did not receive the materials they had selected/recommended. Evidence of existence and use of these materials varied in the evaluation team’s classroom observation visits. In some classes, books were effectively used to develop critical comprehension skills. Students were seen as active participants in the reading process, making predictions and corroborating them through their reading. In other instances, minimal use of materials was evident. The limited use of materials in some classroom contexts confirms the formative evaluators’ concern that the materials provided by the NHP were present in classrooms but were underutilized in instructional activities. However, all teachers interviewed reported that they used the materials a lot and found them very helpful. These materials were often complimented for the excitement they generated when compared with the black and white basal readers provided by the MOEYC. Particular reference was made to the benefits of the children’s literature made available by NHP. Children in NHP schools talked freely about their favorite stories, and the excitement generated by the books was apparent. This excitement and knowledge of books was not strongly evidenced in discussions with their non-NHP counterparts. Some parents also stated that they enjoyed reading the books when their children took them home. Policy on whether children were allowed to take home these books varied. One clear distinction between NHP and non-NHP classrooms was the abundance and variety of books available at NHP schools, while other schools had only books provided by the MOEYC and the Jamaica Library Service. One concern was that some teachers were not aware that the project also had distributed materials for their professional development. In addition, most teachers indicated the need for more training in the use of the materials. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 20 Aguirre International Teachers often commented that they did not receive the materials on time. However, they were aware that these materials could be used in subsequent years with other students. There is an obvious need for more local materials in the schools. While observations identified Jamaican and Caribbean literature, books depicting American culture are in much greater abundance. Teachers have also identified the linguistic differences between Jamaican Standard English and American Standard English as a justification for getting more local materials. Providing a prudent mix of local and foreign materials may be problematic since many of the materials are donated. The NHP has distributed materials based largely on school size. Schools with large population are allocated more materials than their smaller counterparts. Stakeholders expressed satisfaction with this process. However, they also expressed the need for more materials, especially math manipulatives and educational software. In addition to the materials procured and distributed by the NHP, schools are also provided with raw materials (e.g., paper, markers, and crayons). Respondents spoke favorably on the role of this input in helping to generate teacher-made and student-made instructional materials. I. Educational Technology NHP has provided audio-visual equipment to all schools; however, only in a few instances did the team see evidence of use of equipment such as overhead projectors or tape recorders. When these devices were incorporated, teachers seldom displayed confidence or efficiency in their use. Interviewees frequently mentioned that they use the large screen provided that connects to the computer, and that this is very motivational for children. Teachers admitted their shortcomings in using the technological aids, such as the overhead projector; they requested more training, and mentioned that an efficient way of training them would be to allow them to use the equipment in presentations at NHP seminars. Intervention #7, “Establishing computer use in schools and training teachers in educational technology,” has suffered most from the delayed funding at the beginning of the project. At this time, all NHP schools have received one computer. In schools visited, this computer was usually located in the office and was most often used for administrative purposes, such as entering attendance in the project-developed Jamaica School Administration System (JSAS) software. However, several principals mentioned that teachers and children also use the single computer for educational purposes. Some schools have additional computers provided from other sources. Computer availability varied little between NHP and non-NHP schools. One non-NHP school had 30 computers donated by a former student. Respondents also emphasized the need for educational software. The Encarta suite provided and installed by the EMIS team is also used as an instructional resource. Teachers and students have highlighted the usefulness of this software. Teachers seemed to view computer use by students as a means of competing for children’s interest with the multimedia entertainment possibilities at their disposal. In addition, they see it as an incentive for Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 21 rewarding the efforts of their children. While they were aware of educational software as an instructional tool, they could not identify specific software to meet their educational needs. This suggests a need for further in-service teacher training in computer-assisted instruction. Some schools now use the Internet as a research tool for both teachers and children. However, there are also NHP schools without telephone service. Therefore, the benefits of Internet access are not available to all NHP schools. All stakeholders asked for more computers for their schools, for training in the use of computers, and for appropriate software to be used in support of the literacy and numeracy goals of the project. Interviewees mentioned that the availability of computers would generate more enthusiasm among the students, be motivating for slower learners, provide additional learning opportunities for gifted learners, and help to interest boys in school. In addition, some suggested that illiterate parents who were reluctant to attend literacy classes might attend computer classes in which they could actually be taught to read. Parents clearly believe that computers are the route to successful futures for their children. A recent proposal for the infusion of educational technology in NHP schools (PIU, 12/01) highlights the fact that to date the educational technology intervention has not been implemented. The proposal outlines a model for beginning to do so that involves creation of a regional multimedia center at one of the Teacher’s Colleges and the launching of a small pilot project, working with two to four schools. The effort to interface with a pre￾service teacher training college is a positive one. However, the proposal raises doubt that any significant increase in educational technology availability and use in all but a few NHP classrooms will take place, as anticipated, during the life of the current project. The evaluators view this as a cause for concern. (See Chapters VI and VII.) J. Incentives NHP has encouraged teachers to give incentives to students by providing certificates that may be used to reward children for good attendance, excellent schoolwork, improved discipline, etc. A survey of the use of incentive strategies prepared by NHP staff in December 2001 found that teachers using a variety of incentives to improve attendance reported that attendance had improved, particularly attendance on Fridays, which has always been a problem. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 22 Aguirre International At a different level, teachers and principals in the evaluation team’s survey suggested incentives they themselves would like to receive, such as study tours and supplementary or resource materials, as rewards for effective performance. In fact, NHP has recently developed a framework for incentive awards to principals, teachers and schools. This framework lists criteria and suggests awards that range from study tours or conference attendance to such items as NHP pins, special certificates or awards, such as NHP School of the Year. Attendance at conferences and study tours was one of the components in the original project affected by the initial delay in funding. However, since May 2001, 23 individuals have been sent to five different professional conferences such as the International Reading Association (IRA) and the National PTA Conference. Although study tours are presented on the list of incentives, obviously they also have an educational purpose. However, no study tours as yet have been made to visit exemplary schools or classrooms, although some participants at conferences have had such opportunities arranged as an adjunct to their conference participation. K. Health and Nutrition Program Based on the potential linkages between health and nutrition and educational performance, the NHP project plan initially included an intervention with both health and nutrition components in 25 schools. Subsequently, the health component, a de-worming program, was eliminated and in its place additional health education was added. The major focus has been on establishing a breakfast program to supplement the on-going school lunch program. The breakfast program has been implemented, but at a reduced level from that anticipated. Partly in response to reduced funding, selection criteria were tightened, and the number of schools to be included was reduced from 25 to eight. The selection of participating schools was based on a needs assessment that NHP conducted, with emphasis given to schools with low attendance (below 75%) and poor academic performance (NAP scores below 35). All eight schools in the original group successfully launched the program and are continuing it. Participating students were and are selected by teachers and guidance counselors, based on their level of alertness or concentration during early classes, their attendance, mention of morning hunger, and/or teachers’ knowledge of their home socioeconomic situation. The percentage of students participating in the program varies from nearly 100% in the smaller rural schools to less than 15% in large urban schools. The program has become a key vehicle for involving parents in the schools. With the assistance of project-hired community development officers, parents have been trained in community development, organizational strengthening, planning and management, finance, fundraising, and income generation. In 2001, parent and teacher education in health and nutrition were included in the program. To date, fundraising efforts have fallen short of goals. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 23 Recently, six additional schools were selected for the program and will begin providing breakfast during the 2002-2003 school year. Sustainability of the breakfast program is to be achieved through community and parental support. Beginning in the 2002-2003 school year, project financial contributions will be decreased, progressively, to zero. Comparisons of average school attendance before and after the breakfast program show increased attendance from 63/64% to 76/73% in rural and remote rural schools respectively. It is not possible to determine whether the improvement was caused by the breakfast program or by other NHP interventions, or by both in combination. No increase was found in urban schools. When students participating in the breakfast program were compared to non-participating students in two schools, Grade 3 diagnostic scores in English and Math were higher for participating students in both schools while Grade 4 literacy scores were higher for participating students in one school. (See Annex 5 for additional information.) L. Education Management Information System (EMIS) The NHP interventions that focus on improved information systems are designed to contribute to the achievement of the Intermediate Results of “Improved Quality of Teaching” and “Improved Management of Schools.” Through a previous USAID project (PEAP II), the MOEYC was assisted in creating a management information system and database in its Planning Unit. Other projects, such as the IDB PEIP Project, helped develop other databases, including for example, a database for the National Assessment Program (NAP) examinations. NHP was designed to build on these earlier efforts through two interventions: · Intervention #9: To integrate and consolidate the various database systems of the MOEYC into a unified system, along with further connections to other inter-related national databases, such as census data and/or information from the Jamaican Social Investment Fund (JSIF); and · Intervention #10: To link schools with a regional and national EMIS network. Because of funding problems, implementation of these interventions was delayed. In the interim, the IDB initiated the new Primary Education Support Project (PESP), which includes a major component to strengthen management information systems in the MOEYC. As a result, the NHP component was re-designed to, in essence, drop Intervention #9 and focus on Intervention #10, school-level software and connectivity. This resulted in the development of a school database system, JSAS, which is being installed in all 72 project schools. Linkage between the schools and the regional offices and from there to the national offices is awaiting the establishment of the MOEYC network system through PESP. The JSAS software package includes data on school enrollment and other school census information (such as the number of grades and teachers, year of school construction, special courses offered, etc.) and information on teacher profiles, student profiles, student attendance records, student assessment Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 24 Aguirre International scores, and classroom grades. JSAS also can be used to generate a wide variety of reports, both MOEYC-mandated and user-selected; for example, class summaries of assessment scores can be used by teachers to adjust their lesson plans based on class competencies and weaknesses. The system is Internet-ready and was developed so that the system can be distributed to schools license- and royalty￾free and requires only a Microsoft Windows 98 or more recent operating system. The principal and two to three teachers or other staff at each school have been trained in the use of JSAS through workshops and on-site visits by the NHP staff on the EMIS team. Schools currently vary in the degree to which they have been able to get the system up and running. Sustained, quality data entry is a problem. Also, schools remain dependent on hardware and software technical support from the NHP EMIS team. The MOEYC, with NHP’s help, plans to make JSAS available to over 200 non-NHP schools across Jamaica by the end of the year. (See Annex 6 for additional information.) M. NHP’s Monitoring and Evaluation Process Quality and extent of analysis. The evaluation team views J&A’s efforts to monitor and evaluate progress to be exemplary. The project measures performance at multiple levels. Like most projects, NHP keeps track of and reports on important process indicators (e.g., instructional materials delivered to each school, the number of training hours provided to RTs and the number and length of site visits provided by project technical staff). The project collects and analyzes data on many of the program’s ten interventions. NHP goes beyond most projects in assessing impact on intermediate results. For example, the project annually assesses the degree to which teachers in NHP schools are utilizing the child-centered teaching strategies that the program recommends and the extent to which the teachers and children are utilizing the NHP￾provided supplemental teaching materials in the classroom. Similarly, the project monitors attendance data in an effort to determine whether the school feeding program or other interventions have yielded results in that area. NHP also monitors a number of other intermediate results (e.g., the degree to which school principals and resource teachers are performing as intended) on a more sporadic basis. Unlike most projects, NHP collects information on indicators of impact at the SO level that can reasonably be attributable to the project – indicators that measure the project’s effect on increasing Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 25 literacy and numeracy among the targeted Jamaican youth. The indicators utilized are the results achieved by NHP school children on nationwide tests given to all Jamaican school children at the third and sixth grade levels. The Performance Monitoring Plan establishes targets for increased test scores in both language arts and mathematics as measures of impact. To take into account exogenous changes that might influence test performance in NHP schools and throughout the Jamaican education system, the project’s Formative Evaluation compares changes in NHP schools to changes in non-NHP schools. It is important to note that NHP’s monitoring of student test performance was intended to give project staff information that would indicate where changes were needed, not to measure impact. In fact, the Mission intended initially to set up a separate system to determine impact. However, the Mission canceled a procurement action for that system, and the NHP formative system has clearly provided data related to impact. NHP has achieved this without additional resources. In brief, the results of these evaluations (discussed in more detail in Chapter V below) indicate that students in NHP schools are, on average, scoring considerably higher on most tests than they did in previous years – better even than the project’s targets. To confirm that these improvements are reasonably attributable to the project and not the result of exogenous factors, NHP compares NHP’s results with other Jamaican schools. In these comparisons, the large improvements in NHP test scores are in many cases no better than the gains made by the general school population. These results, while troubling, must also be interpreted with caution. NHP schools were selected in part because they were below the Jamaican norm, and the project compares their improved results with all other Jamaican schools, most of which started the period at a considerably higher standard of performance. It is impossible to say whether NHP schools, without NHP, would have progressed less rapidly, equally, or more rapidly than those other schools. A second methodological issue relates to NHP’s use of the terms and categories “mastery” and “near mastery” to assess and interpret results on the 6th grade GSAT exam. These are not terms that MOEYC utilizes with respect to that exam. NHP borrowed these terms and the definitions of them from the 3rd grade exam, which defines “mastery” for that exam as 75% or more correct responses and “near mastery” as 50-74% correct responses. NHP uses these concepts rather than simply comparing average test results because this method focuses on the project’s goal of raising the performance of the lowest achieving students up to a minimally acceptable level. Average scores could rise simply because the best students get better, with those not meeting minimal standards remaining where they were. The evaluation team understands NHP’s desire to measure a school’s progress in bringing all students up to acceptable and desirable standards, but it is concerned about the way in which the project set those standards for the sixth grade. To apply the third grade definitions (50% and 75% correct responses) to the GSAT seems arbitrary and inappropriate. Although NHP is providing interesting and useful data showing changes in the percentage of sixth grade students able to answer 50% and 75% of the correct responses on the GSAT, there is no evidence that these scores correspond to what Jamaican educators would define respectively as near mastery and mastery. Further, since the third and sixth grade tests are very different and there is no evidence that scores on the tests are comparable, Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 26 Aguirre International longitudinal studies relating sixth grade performance to third grade performance (such as NHP hopes to carry out in the future) would need to be interpreted with caution. Further, despite the positive aspect of using these categories (as mentioned above, keeping the focus on bringing all students up to acceptable levels), the use of categories that include broad ranges seems also to offer the potential of masking significant changes in student performance within those categories. For example, the improvement in student test scores from 10% to 49%, from 50% to 74% or from 75% to 100% correct responses would surely be very significant, but under the current “category system” might go unrecognized. These methodological concerns are of more than academic interest, since USAID, the MOEYC and project staff would understandably like to see strong evidence that the combination of NHP interventions is having success in bringing low performing schools up to Jamaican norms. The evaluation team believes that consideration should be given to other methodological approaches (e.g., establishing a control group), if they might resolve this ambiguity at a reasonable cost. (The evaluation team’s suggestions in this regard are provided in Chapter VI below.) Utilization of Evaluation Information. The real test of NHP’s evaluation system is its usefulness in decision-making. The evaluation team’s interviews revealed a great deal of anecdotal evidence that suggests that NHP’s monitoring and evaluation data are utilized in rethinking priorities and in making management decisions. The project seems to have utilized evaluation data, for example, to orient training programs and to identify schools that need additional assistance. The current discussion about the creation of a new cadre of “NHP associates” who will reinforce training in child-centered instructional techniques presumably reflects evaluation findings that the use of those techniques has not yet been adequately incorporated into teaching plans within NHP schools. Access to Evaluation Information. The evaluation team found the project’s efforts to make its monitoring and evaluation reports available to the public to be unduly limited. While the list of products/deliverables attached to its Semi-Annual Reports is helpful, that list does not always include the date of publication and does not identify the author. Neither the Formative Evaluation nor the project’s website includes a comprehensive list of project publications. Moreover, the project office in Kingston does not maintain a library where visitors can easily see documents and obtain copies. Development of Jamaican Evaluation Capacity. NHP has done a superior job not only in setting up an evaluation mechanism but also in training Jamaicans, where necessary, to carry out most of the project’s evaluation functions. This capacity development effort has yielded an extra benefit from the program. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 27 N. Cost and Cost-Effectiveness USAID’s NHP budget (exclusive of USAID management costs and Government of Jamaica contributions) for the five year period, 1998-2003, totals $8.826 million. Of this amount, approximately $6.326 million, or 72%, is budgeted for the J&A contract and $2.5 million, 28%, is budgeted for expenses of the Government of Jamaica (including the MOEYC and its agreement with NCE). Table 2 shows planned amounts, obligations to date, and expenditures. Table 2: SO 4. Total Resources For NHP May 2002 PLANNED OBLIGATIONS EXPENDITURES $ thou. % $ thou. % of Plan $ thou. % of Plan % Oblig. J&A 6,326 71.7% 4,819 76.2% 4,372 69.1% 90.7% MOEYC 2,500 28.3% 508 20.3% 415 16.6% 81.7% Totals 8,826 100.0% 5,327 60.4% 4,787 54.2% 89.9% Source: USAID/Jamaica and J&A The table shows that J&A and the Ministry have expended the bulk of the resources, 91% and 82% respectively, that USAID has provided to date. USAID has yet to obligate 24% of J&A’s budget and almost 80% of the Ministry’s planned budget. These implementing institutions have approximately one year and three years respectively to complete the work that they have undertaken under their USAID agreements. Table 3 below provides more detailed information on plans, obligations and expenditures in the J&A contract. The budget plan gives some sense of the relative priorities given to different project interventions. Interventions #3, #4 and #8 are implemented primarily by the MOEYC and related institutions, and the resources provided for them in the contract do not reflect the total amount of resources devoted to those activities. Nevertheless, it is interesting to note that J&A has utilized only 46% of the funds obligated and only 20% of the planned resources in support of interventions #3 and #4. Factors underlying these low expenditure rates include the Mission’s shortage of funds to provide directly to the Ministry and, according to some sources, problems within some parts of the Ministry and one of its partners in utilizing available resources promptly. Table 3 also shows the low level of resources thus far expended on the educational technology component (Intervention #7). The data show that the contractor has received only 23% and expended only 14% of the planned budget under that intervention. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 28 Aguirre International Table 3: NHP Cumulative Obligations and Expenditures (Initiation through 4/30/02) CLIN Interv. No. Plan Obligations Expenditures $ thou. % $ thou. % of Plan $ thou. % of Plan % Oblig. 1 1, 2, 8 1,097 17.3% 1,070 97.5% 1,019 92.9% 95.2% 2 6 779 12.3% 590 75.7% 524 67.3% 88.8% 3 7 749 11.8% 170 22.7% 108 14.4% 63.5% 4 9, 10 805 12.7% 630 78.3% 577 71.7% 91.6% 5 3, 4 271 4.3% 120 44.3% 55 20.3% 45.8% 6 5 601 9.5% 289 48.1% 239 39.8% 82.7% 7 Mgt. 2,024 32.0% 1,950 96.3% 1,850 91.4% 94.9% Total 6,326 100% 4,819 76.2% 4,372 69.1% 90.7% Table 3 suggests the need for the USAID Mission and J&A to revise the budget. It is apparent that Contract Line Item Numbers (CLINs) 1 and 7 will need additional resources. The table shows that almost 93% of the resources for CLIN 1, which includes development of math and reading programs (Intervention #1), in-service teacher training (Intervention #2) and training of resource teachers (Intervention #8), have been expended. The remaining unexpended budget for this CLIN totals only $78,000 for use for the final 13 months of the current contract (i.e., through June 2003). Similarly, there is a dearth of funds remaining for program management. Unless the contractor were to immediately cut back on project implementation, the remaining $100,000 in CLIN 7 will not be adequate for the remaining months of the current contract. At the same time, the amounts budgeted to several other categories appear to be more than can be effectively utilized in the next 12 months. The evaluation team’s observations would support a reassignment of some resources within the J&A budget from health and nutrition and perhaps supplemental educational materials. Some reallocation of planned resources not yet obligated to the MOEYC might also be reprogrammed. Considerable unexpended resources also remain under the Educational Technology line item. USAID and J&A are considering the possibility of moving a major amount of those resources to another line item. However, the team recommends that sufficient funds be retained to permit the provision of computers and other educational technology in the schools, as originally planned (See Chapters VI and VII). With respect to cost-effectiveness, a quantified analysis of individual interventions is not possible. The organization of Table 3 above reflects the fact that USAID has budgeted and asked the contractor to maintain cost data in categories (CLINS) that separate the costs only for interventions #5, #6 and #7. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter III: Findings · 29 More importantly, because nine of the ten interventions have been implemented in all 72 project schools, attribution of the benefits to any one intervention is greatly complicated. Indeed, the contractor’s efforts using regression analysis to determine whether some activities can be shown to yield higher benefits have proven unsuccessful. Although rigorous quantitative analyses of cost-effectiveness are not possible, the evaluation team did observe considerable data that relate to this question. Efforts to control cost, for example, were evident in the contractor’s use of local-hire as opposed to imported expatriate staff and in its very successful efforts to procure instructional materials through donations and discounts. On the “effectiveness side,” interviews, observations, and test scores suggest that interventions are having a positive impact in the classroom and with students. These indicators of effectiveness should increase as teachers and school administrators become more proficient in delivering improved services and as children pass through this improved system not just for one or two years but throughout their primary school years. At the same time, the evaluation team believes that NHP’s cost-effectiveness to date might well have been higher if the program had been implemented as designed, e.g., with educational technology available in the classroom and with supplemental educational materials delivered in a more timely fashion. O. Project Management The Partnership Approach. As explained in Section II above, the leadership for and control of financing of the various program interventions rests either with the contractor, the MOEYC or NCE. With responsibility diffused, coordination and collaboration have become not only desirable, but essential. Although there are sometimes tensions between the parties and less than perfect coordination of efforts, the partnership approach is generally working well as an implementing management strategy. In the longer term, this approach is likely to pay dividends in terms of Jamaican “ownership” of the program and thus to sustainability and replicability. Contractor Management. J &A’s management of its activities seems to be quite good. Resident staff are technically competent and consultants, judging by their reports, appear to be of high quality. The project makes extensive use of Jamaican expertise, which helps to keep the costs down and is likely to make products and services more appropriate to the Jamaican context. J&A has also made effective use of its U.S.-based staff. The evaluations specialist helped to create an unusually good performance monitoring plan and to train Jamaicans to implement that plan. The financial director, in addition to managing the austerity budget issues, worked with other J&A staff to find creative ways to provide a Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter III: Findings · 30 Aguirre International large quantity of supplementary educational materials despite funding shortages. Perhaps the biggest challenge to J&A management has been dealing with funding delays. Funding problems not only slow the procurement of needed goods and services, they inevitably require multiple rounds of re-planning and re-negotiation among the key stakeholders and divert management’s attention away from other priorities. As far as the evaluation team could determine, the contractor seems to have done well in handling these issues and adapting to changes that have been required. The contractor has had only limited professional staff (presumably because of budget constraints), and most of these are subject area specialists. With no specialists in areas such as educational technology and instructional materials, staff members have had “to wear several hats.” This has stretched staff members between competing objectives. Although the staff appear to have adjusted well to this workload, the evaluation team believes that some issues (e.g., the development of a technically sound educational technology plan within budget and acceptable to all of the principal stakeholders) might have been resolved more quickly had the project been more adequately staffed. Finally, the team noted that project documents describe a “bottom-up” planning process in which participating schools develop SDPs and these are aggregated to create a project work plan. This has not occurred and, in the evaluation team’s view, was an unrealistic expectation. Although not “bottom￾up” in that sense, the project is focused on what happens in the classroom, the school and the community – a factor that distinguishes it from many other donor programs and increases its value to Jamaican education. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter IV: Results · 31 Chapter IV Results A. NHP Impact on the Teaching/Learning Environment As mentioned in Chapter III, Findings, NHP’s formative evaluations have consistently found that NHP classrooms remain teacher-centered, although they have also reported improvements in classroom environment as well as small improvements in other measures of teacher quality. Site visits to classrooms made during this evaluation are in accord with those findings. However, there is considerable variation between teachers and between schools, and many classrooms show evidence of improvements in teacher practice such as basing lessons on student experience, involving students more, and encouraging higher levels of thinking. Most teachers are familiar with the idea of integration and are beginning to apply some of the strategies for improvement of literacy and numeracy. Many teachers use both teacher and student portfolios, and some encourage original student writing in student journals. Some are experimenting with cooperative learning, and many have improved the classroom environment, within the limits of what is possible in overcrowded, noisy classrooms divided from each other only by chalkboards. Since many of these changes are noted within a traditional teacher-centered style, it is important to note that they may not show up on the indicator most often cited from the formative evaluations, that which measures whether interactions are teacher or student initiated. Stakeholders most frequently mentioned that the two most successful aspects of the project were training and provision of materials. These have obviously been highly important inputs into the improvement of the teaching/learning environment. One principal mentioned that, in relation to its SDP goals in literacy and numeracy, her school is “half way there.” In essence, the team concurs that many of the teachers are “on their way” but need additional on-site and in-class support; this is especially needed for regular classroom teachers who are not RTs. Stakeholders also reported that children were: · more interested in school, · more actively involved in their schoolwork, and · more interested in learning. Teachers in schools with NHP breakfast programs often commented that the students who receive breakfast now: · seem more alert, · participate more, and · do not fall asleep at their desks as they used to. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter IV: Results · 32 Aguirre International Other likely influences on student attitude and learning, in addition to improvements in the quality of teaching noted above, are the use of incentives by teachers to reward students for good attendance or good schoolwork, the introduction of interesting activities and clubs, and increased participation by parents in their children’s learning, as noted both in formative evaluations and in the interviews and site visits. B. Impact on Attendance and Student Participation Absenteeism has been cited as a major constraint to educational improvement in Jamaica. Factors including the school environment, teaching styles, nutrition, and parenting are considered to influence school attendance. This evaluation does not propose to map NHP interventions as to their effect on attendance. However, a holistic examination of the attendance pattern in NHP schools indicates that there have been gains over the life of project. During the first year of project implementation, for example, there was an average attendance gain of over 6 %, and in the second year, there was a modest gain of 1%. These gains support the MOEYC’s goal to increase average attendance. Enrollment in most NHP schools has also increased. NHP’s school breakfast program is particularly intended to encourage gains in attendance and in student classroom participation. To date, 985 children in eight schools have participated in the breakfast program. It is reasonable to assume some nutritional benefit, and teachers and students report positive effects on student performance. The program has also been instrumental in increasing parent involvement in the schools. (Parents and PTAs have been active in the management of the program and raising money to pay for it.) Although attendance at most of the eight original schools has improved, the breakfast program appears to have been a significant factor in this improvement only in rural and remote schools. Recently, six additional schools were added to the program; it is too soon to evaluate the results in these schools. C. NHP Impact on Student Performance As mentioned previously, NHP set targets for the performance of 6th grade students on the national GSAT exam. NHP has defined Near Mastery as 50% or more correct responses and Mastery as 75% or more correct responses. NHP schools have exceeded the targets of the Performance Monitoring Plan in all years for which data are available. (Test scores for 2002 were not yet available at the time of this evaluation.) The following table summarizes these results. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter IV: Results · 33 Table 4: Percentage of NHP Students Achieving Near Mastery or Better on the GSAT Language Arts and Mathematics Examinations Language Arts Mathematics Year Girls Boys Girls Boys Target Actual Target Actual Target Actual Target Actual 1998 43 17 14 6 1999 45 47 19 20 16 24 7 10 2000 48 64 22 35 18 41 10 25 2001 52 57 25 34 20 42 13 26 2002 56 30 25 20 2003 60 35 30 30 As NHP’s 2001 Performance (Formative) Evaluation acknowledges, NHP has had less success in improving the performance of children in the early primary grades. The performance of NHP children has been particularly disappointing on the third grade language arts exam, on which the percentage of children achieving mastery or near mastery declined slightly over the 1998-2001 period. Interestingly, NHP schools’ scores on the third grade math test, while lower than language arts, increased quite significantly over the same period. (See 2001 Performance Evaluation, Tables 9 and 3, page 10 and 7 respectively.) The explanation of these results is unclear. All schools in Jamaica have improved on the GSAT since its introduction; this may be attributed both to improved teaching as a result of the MOEYC’s introduction of the new integrated curriculum and materials and to the fact that teachers have now learned what is on the new test and are “teaching to the test.” Table 5 below presents a comparison of the average raw scores of NHP and non-NHP students in 1998 and 2001. The non-NHP student average improved somewhat more than the NHP average in math, whereas the two groups improved roughly equal amounts on the language arts section of the test over the three-year period. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter IV: Results · 34 Aguirre International However, it should be remembered that the main criterion for selecting the NHP schools was that they were low performing schools. Since their scores were lower to start with, the percentage gain in their average raw scores is slightly higher than that of the non-NHP group. This is noted because many educators have observed that it is extremely difficult to achieve improvements on test performance with students who have been accustomed to failure, in schools that for many years have had a “culture of failure.” The comparison being made is with all non-NHP schools, including both other low achieving schools and those traditionally achieving at much higher levels. These sixth graders had only received the benefits of NHP for two years, so it is most likely too early to tell if NHP interventions are having a significant effect on learning. However, it is encouraging to note that they did make the improvements indicated. Table 5: Comparison, Average Raw Scores and Percentages GSAT, 1998 and 2001, NHP and Non-NHP Students N 1998 1998 Avg. Score N 2001 2001 Avg. Score Improvement in Average Score Percent Improvement in Avg. Score NHP Math 3584 29.78 (37%) 3987 41.71 (52%) 11.93 40% Non-NHP Math 38355 37.51 (47%) 43901 50.74 (63%) 13.23 35% NHP Lang. Arts 3556 40.51 (51%) 3987 47.46 (59%) 6.95 17% Non-NHP Lang. Arts 38548 49.48 (62%) 43901 56.42 (71%) 6.94 14% An important consideration is that average scores obscure the considerable differences between schools. Table 6 below presents average percentages achieved by the schools that the evaluation team visited (these are placed in random order). In addition, the range of all NHP schools is presented, as well as the range of scores for this sample. The range of all NHP scores suggests that the improvement that NHP schools have registered in language arts might be attributable to the fact that the lower schools have raised their average scores. The range in math scores, however, shows a considerable gain at the top end. The difference in the level of improvement noted in the evaluation team’s small sample below is illustrative of the general differences between schools. School A, for example, has achieved considerable improvement in both language arts and math. This is of interest because School A was Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter IV: Results · 35 notable in the site visit for the commitment and dedication of its principal and resource teachers. These gains have been achieved in spite of the fact that this is an All-Age school with very inadequate infrastructure, including overcrowded and noisy classrooms. On the other hand, in some of the schools visited that appeared to have less instructional leadership by the principal and/or a lower level of staff dedication, gains have been much lower. In general, the data in this small sample suggest a correlation between the level of instructional leadership and staff commitment observed on the site visits and the degree of improvement on a school’s test scores. It would be of interest to study further the differences between the higher and lower achieving schools. Table 6: Range and Average School Percentages on GSAT Comparing National, All NHP Schools and Schools Selected for Site Visits Language Arts Mathematics 1998 2001 Change 1998 2001 Change Nat. Average 47.6 57.4 9.8 35.8 51.6 15.8 Range, All NHP Schools 21.4 - 66.6 36.7 - 66.47 17.5 – 44.4 29.2 – 66.0 Range in Site Visit Sample 34.6 – 43.8 37.2 – 58.1 27.1 – 33.0 36.0 – 50.5 School A 39.9 58.0 18.1 27.0 50.5 23.5 School B 34.6 44.3 9.7 26.2 36.0 9.8 School C 37.5 52.8 15.3 29.8 42.2 12.4 School D 36.7 41.0 4.3 31.8 39.0 7.2 School E 43.6 51.9 8.3 33.0 45.2 12.2 School F 38.9 57.2 18.3 29.9 51.9 22.0 School G 36.9 37.2 .3 28.2 33.8 5.6 School H 43.4 37.9 -5.5 30.0 38.6 8.6 Overall, the test results show that NHP schools have made considerable improvement (except in language arts at the third grade level). The simultaneous improvement in the test scores of non-NHP schools makes it difficult to know the extent to which the improvement in NHP schools is a result of the project. With some uncertainty surrounding the explanation of test results, the evaluation team’s classroom observations (in both NHP and non-NHP schools) and interviews with stakeholders have Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter IV: Results · 36 Aguirre International proven particularly useful; they provide quite convincing evidence that things have changed in the NHP schools more than in non-NHP schools. If these changes continue, the team believes that children’s ability will continue to improve and test scores will continue to rise. At the same time, the team’s observations and the test results show that the NHP schools still have a long way to go to achieve their goal of mastery in literacy and numeracy. Mathematics appears to present the greater challenge: a higher percentage of children are achieving near mastery in language arts. Boys’ performance presents a greater challenge than girls’ performance; girls consistently achieve higher scores on all of the test measures reviewed. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter V: Lessons Learned · 37 Chapter V Lessons Learned Changing Teacher Behavior Takes Time. The change sought in Jamaican schools is far more than a simple change in curriculum; rather, it is a change in a deep-rooted culture. Experiences in other countries suggest that there is a continuum of change that can be noted in teachers trying to make changes from traditional systems based on memorization to more active, participative learning. · In the first stage, there is often a combination of fear and resentment, combined with the fact that the new methods are sophisticated and time-consuming. · At the second stage, many teachers become conversant with the new jargon, and may begin to try some of the new ideas. However, without ample support in the classroom, many find the new methods difficult and may tend to abandon them. · With adequate training and support, including support of principals and fellow teachers, many will move on to a third stage, in which they perceive the advantages of the new methods, find that they enjoy them and that their students are more enthusiastic. Even at this stage, many teachers still have not mastered the new strategies, but they are on their way. Many NHP teachers are in the second and third stages mentioned. Change has begun, but more, continued on-site support is required for this change to become sustainable. Need for Earlier Intervention. System-wide, MOEYC policy places emphasis on testing at the third and fourth grade in order to identify children who need extra assistance. However, international research clearly indicates the importance of early experience in preparing children for success in school. Interventions at the end of fourth grade come after children have experienced several years of failure and are likely be much less effective than earlier interventions. More attention needs to be paid at earlier levels to children in preschool and first grade who may need extra support and assistance in order to experience school success. There is also a need for more attention to the linkage between preschool and first grade curricula and the transition between those levels. All Elements of an Integrated Program Need to be Implemented for Maximum Results. The delay in implementing the educational technology component from the outset of the project has significantly reduced the impact of the other interventions, especially those related to new teaching and learning strategies. In retrospect, it might have been more productive to start it on a small scale and then increase it as savings appeared elsewhere in the budget. A Partnership Strategy Requires Careful Monitoring and Timely Action to Resolve Problems. In the NHP case, the partnerships have generally worked well, but in one important case, outputs appear to have been well below the anticipated level, without, so far as the team could see, being corrected. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter V: Lessons Learned · 38 Aguirre International The Value of an Effective Evaluation Process. A number of factors have contributed to making NHP’s evaluation system effective. Among the factors not often present in USAID programs are (1) the utilization of an experienced, professional educational evaluator, (2) the collection of baseline data, and (3) the identification of indicators that measure impact as well as outputs. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter VI: Conclusions and Recommendations · 39 Chapter VI Conclusions and Recommendations for Current Project A. Conclusions The task that NHP has undertaken is an enormously challenging one. Changing the way teachers teach, the environment that they create, and their whole relationship with students is difficult. Achieving this transition in poorer performing Jamaican schools – those in which the teachers have little if any professional training, instructional materials are lacking, pedagogic and management support is rudimentary, children often come from low income families where print materials are scarce, parental and community involvement in schools is marginal, and physical facilities are noisy and crowded – is a process that will clearly take some years. Despite the many obstacles to change within the system and the added difficulty caused by funding shortfalls at the outset of the program, the evaluation team’s findings show that significant progress has been made in getting this transitional process started. The progress already achieved shows that the program’s goals are attainable and that the design is essentially sound. With the continued commitment and support of the Jamaican Government and with the support of USAID and other donors, the children who attend what have been Jamaica’s poorest performing schools can reach acceptable competency levels in language and mathematics. The evaluation team strongly encourages USAID to continue supporting this endeavor. B. Recommendations for Current Project As previously mentioned, the evaluation team considers NHP to be a project that is moving in the right direction. The following recommendations are intended to assist in the improvement of NHP as it continues in its present course for the remainder of the project. Extend the Juarez contract through the life of the project – to 9/30/04. With funding delays impeding the implementation of several interventions, the first phase of this effort to improve literacy and numeracy in project schools will not be completed by mid-2003. The contractor is performing competent work and should be awarded an extension to complete the task. Strengthen interface with other MOEYC/Donor and USAID programs. There are many fruitful avenues of collaboration, especially with PESP, the new AID-sponsored Early Childhood Transition Project, and the Caribbean CETT now being planned. It is recommended that efforts also be made to strengthen relations with MOEYC regional offices and TEOs by seeking ways to involve TEOs more in Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter VI: Conclusions and Recommendations · 40 Aguirre International training sessions, on-site supervision, and interaction with School Boards and School Development Teams. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter VI: Conclusions and Recommendations · 41 Continue to give priority to strengthening classroom teaching quality: · Use the new Associates for more intensive in-class follow-up and teacher support. It is essential that the Associates be highly qualified, familiar with NHP recommended strategies, and trained to do the on-site support in the manner desired. In addition, lines of responsibility and assignments to regions or schools should be clearly specified. · Increase opportunities for teacher interaction within/between schools by increasing the emphasis on cluster activities and instituting a system of peer observation (as outlined by Louise Thompson and Jon Saphier at Research for Better Teaching, MA). The new Associates could provide valuable assistance in both of these efforts. · Maximize effectiveness of RTs by offering training on the RT role to both RTs and classroom teachers. · Explore additional opportunities for training classroom teachers through a possible link with the new Caribbean CETT, and consider including classroom teachers when possible in RT training or in cluster workshops Continue the Emphasis on Quality Training of Resource Teachers, by considering additional training in: · Strategies for teaching multigrade classes, with CETT as a possible resource. · Use of materials (including math manipulatives, AV, computers) as well as training in how to make materials out of locally available items. · Diagnostic techniques, especially in the early grades and techniques for teaching students of differing abilities, especially lower-achieving students, with CETT as a possible resource. · Methods of using ongoing formative assessment to plan appropriate instruction for the diverse needs of students, with CETT as a possible resource. · Cooperative learning, including grouping strategies. · Techniques for improving the end of year testing program by incorporating effective strategies and authentic assessment methods into the revision and tests prepared by teachers. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter VI: Conclusions and Recommendations · 42 Aguirre International · Use of positive reinforcement and positive methods of discipline such as the Assertive Discipline method. · Mathematics content and teaching strategies. Continue to provide quality teaching materials: · Explore opportunities for providing more materials written by Jamaicans, centered in the Jamaican or West Indian context. · Consider the provision of more professional teaching materials, which might include a small professional library in each school and videos demonstrating successful use of effective strategies in Jamaican classrooms, with CETT as a possible resource. Ensure that education technology is focused on learning and is student-centered by: · Providing monitoring and technical assistance to ensure that the evolving NHP technology plan has as its focus the use of technology in service to the curriculum. · Developing a complementary, phased, multiyear plan to provide computers and educational software in NHP schools, with the first phase to begin during the remaining life of the current project. Continue to provide training and support for principals in instructional leadership by providing structured opportunities for exchange of ideas between principals in nearby schools, opportunities for study tours, and continued training in supervision and pedagogical leadership. Continue to encourage parents to focus on learning, rather than having parents and community members concentrate only on beautification or infrastructure. Work with parents to help them understand the importance of attendance and punctuality for their children’s success in school. Refocus the study tours so that the emphasis is on learning opportunities, providing more opportunities to visit exemplary schools and classroom programs that are using the management and instructional strategies desired. Introduce changes to improve the assessment of project impact. To address the methodological problems described in Section IV above and to provide a more valid basis for the assessment of the impact of NHP, the team proposes that NHP: · Consider establishing and using a control group of Jamaican schools with 1998 grade 3 and 1998 or 1999 grade 6 test scores similar to those of the NHP schools. To the extent possible, NHP could try to select schools with other similar characteristics as well. · Develop alternatives (additional) to NHP’s current analysis of “mastery” and “near mastery.” The evaluation team suggests that a comparison of mean test scores Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter VI: Conclusions and Recommendations · 43 might offer additional, useful perspectives on program impact. These comparisons might look at changes in individual schools over time, at how different NHP schools are progressing, and how NHP schools compare with non-NHP schools. Another useful approach would compare the percentage of children who score in each decile on the exams. This methodology is essentially a refinement of the current methodology, replacing its three categories with ten in order to obtain a better indication of change. · Consider developing additional indicators for use in measuring the quality of teaching and of leadership. Indicators of teaching quality might be expanded to include measures of levels of questioning used and measures of whether teachers are attempting to use NHP strategies. Measures of principals’ performance could be expanded to include indicators such as attendance at teacher training workshops, awareness of what is needed for effective use of the new strategies, use of NHP criteria in teacher supervision and evaluation, and others obtained from the effective schools literature. Provide careful follow-up on the first phase of breakfast programs to determine the level of sustainability. Plan to continue or discontinue such programs in the future on the basis of successful sustainability. Make the JSAS system operational in project schools. EMIS activities during the remainder of the current NHP should focus on technical assistance to schools to put the JSAS into full operation and training in its multiple uses. Separate, hands-on workshops focused on the system’s capabilities to support learning objectives should be developed for principals, teachers, and guidance counselors. Non-EMIS NHP staff, such as the literacy and math project specialists, should be included in the workshops. Begin to prepare for a possible Follow-On Project. The performance of schools and commitment of principals and staff should be carefully monitored and evaluated during the remaining years of the contract. For those schools that are deemed possible “model” or “leadership” schools, provide extra training to help strengthen them, including training in mentoring, training of adults, etc. For those schools that are deemed not to be adequately taking advantage of NHP opportunities, the recommendation would be to discontinue their participation at the end of the existing project. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter VII: Recommendations for Follow-On · 44 Aguirre International Chapter VII Recommendations for Follow-On NHP, on balance, has been a considerable success. However, its work is incomplete. Classroom teachers are only beginning to utilize child-centered, active-learning techniques; school principals will need to do a better job of supporting teachers and managing school and community resources; and the number of parents actively supporting their children and the schools needs to increase many-fold. For these reasons, the goals in terms of language and numeracy and the program’s sustainability are only partially achieved. The team recommends that a new, 5-year project, to be known as NHP II, be developed to finish the job. A. NHP II Objectives ÿ Further strengthen and consolidate NHP I achievements in approximately 60 of the most deserving original schools, especially through human capacity development in the schools and involvement of students and parents. ÿ Extend the project’s benefits to approximately 40 additional schools, thus creating a more critical mass of innovation in the primary school system. ÿ Explore new and more effective models of collaboration between the schools, the regional and territorial offices, and the teacher training colleges. These models might include mentoring of new teachers and project-related action research projects. ÿ Identify and implement innovative strategies for addressing transition issues between early childhood education and Grade 1 and improving performance in Grades 1-3, the subject of a new, integrated curriculum. ÿ Develop and implement innovative strategies for better preparing Grade 6 school leavers for high school. B. Proposed Approach NHP II would continue and build on NHP I but would incorporate important new elements, as follows: Increase the focus on the early grades and transitions. Educational objectives would be modified to include increased emphasis on the early grades (1-3), on transitions between pre-school and Grade 1 and Grades 6 and 7, on boys, and on bringing computers and other educational technology into the classroom; Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter VII: Recommendations for Follow-On · 45 Increase emphasis on educational technology in classrooms and at the service of the curriculum. If the team’s recommendations for the initiation of a phased, multiyear plan to enhance the use of educational technology in NHP schools during the remainder of NHP I are accepted, implementing the plan’s subsequent phases would be a high NHP II priority. Expand the partnership model to include other actors. The partnership model would continue but be expanded to include regional and territorial offices and selected teacher training colleges; the proposed new CETT would be another candidate for partnership, working with the colleges; in addition, increased efforts would be made to involve the community, particularly parents. Address facilities problems that inhibit learning. A fund would be established under the PIU, which, through a competitive process, would enable the project to make small grants for modest, learning-centered facilities improvements; besides learning impacts, strong community involvement and cost￾sharing would be important elements in successful applications. Utilize the best of the existing 72 schools as teaching models. The management model would be significantly modified through creation of a new category of leadership schools, selected from among the most successful NHP I schools, to serve as mentors to new NHP schools in their respective areas. Each school might serve as a “cluster center,” providing assistance to surrounding schools. The creative use of this approach might increase the intensity of support for teachers in new NHP schools and lessen the demands on the Kingston-based staff. Expand the program to additional schools. The total number of schools in the project might be increased to approximately 100, including perhaps 10 leadership schools, 50 current NHP schools, and 40 new schools. New schools would be selected based primarily on the basis of need, the presence of a strong principal, teacher interest, and active community support for the school. The majority of current NHP schools would remain in the project but receive a reduced level of services. Schools that have failed to take adequate advantage of the project, however, would be dropped, after an appropriate review process. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter VII: Recommendations for Follow-On · 46 Aguirre International Support project-related research. Two applied research projects that would contribute significantly to measuring NHP’s impact are 1) a longitudinal study, which would follow an NHP cohort from, say, third to seventh grade and 2) a study of the characteristics of the most and least successful NHP schools, based on improvement on the national tests. Coordinate closely with the PESP project. NHP II should collaborate closely with the PESP project, taking maximum advantage of that project’s experience with demonstration and lighthouse schools and sharing the NHP experience. Expand the staff of the technical assistance contractor. A new five-year technical assistance contract would be designed and competed to support the project. A re-designed PIU would be budgeted and staffed appropriately, taking into account the experience in NHP I, which demonstrated clearly that the original project staff was too thinly stretched to be maximally effective. A larger portion of USAID funds, including training funds, would be channeled through the PIU to increase the efficiency and timeliness of project actions. Refine the indicators of changes in classroom activities. In addition to implementing the evaluation recommendations included in Chapter VI above, NHP II might identify indicators that detect more subtle changes in teacher practices and student behavior. C. Sustainability Under the proposed approach, sustainability would be achieved through several inter-related processes: A significant increase in the number of reform-oriented schools, related capacity-building in the schools, and mobilization of parents and other community resources as advocacy groups on behalf of their children’s education. Efforts by USAID, assisted by the PIU, to get the GOJ and MOEYC to buy into the package and commit themselves to making every possible effort to find the resources to progressively extend its application island-wide. Organization of special events to give Jamaican leaders, top Ministry officials, the IDB, and other interested donors opportunities to receive briefings on the project’s progress and periodic opportunities to view it in action. Consideration should be given to encouraging the establishment of a joint private sector/civil society advocacy group to help keep primary education reform high on the nation’s agenda and the organization of a national consultation, similar to the one for the public sector modernization program. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Chapter VIII: Unresolved Issues · 47 Chapter VIII Unresolved Issues The evaluation team believes that the following issues will require additional analysis: Future of breakfast program While it is recommended that the current breakfast program in 14 schools be continued and carefully monitored and evaluated through the life of the current project, it is an open issue whether the program should be continued as a project-supported activity during the follow-on period. So far, its impact on NHP’s core learning objectives appears to have been modest in rural schools and even more limited in inner city and urban schools. Likewise, while considerable effort has been expended on cost sharing and the program has been instrumental in increasing parental involvement in the schools, it is not clear that the program will become self-sustaining. Significant progress on achieving one or both of these impacts is essential, if the program is to be continued. Selection and Role of the New Associates The plan to select 24 new Associates to assist the NHP literacy and math specialists for the remainder of the current project is unclear in several respects. It will be necessary to clarify the selection process, the role and location of the specialists, and lines of responsibility and supervision. The associates, even if highly qualified, will require training to enable them to carry out effectively some of the on-site support now provided by the specialists. The continuation of such a position in the NHP II project should depend on the results of ongoing evaluation of the on-site support model during the remaining years of NHP I. Role of the Resource Teachers in NHP II Several difficulties have been noted surrounding the role of the RTs in NHP I. These include the lack of substitute coverage for their classes on the numerous occasions when they must attend or provide workshops, the lack of any official recognition of their status or the ample training they receive, and the lack of remuneration for their efforts. Evaluation and Recommendations: New Horizons for Primary Schools Project Chapter VIII: Unresolved Issues · 48 Aguirre International With regard to substitute coverage, more might be done to encourage assistance from volunteer student teachers or Youth Service Workers. However, in order to meet its commitment to improving classroom quality, NHP II may wish to consider developing a cadre of qualified substitutes who could be used to provide classroom coverage on selected occasions. It would certainly be appropriate to consider remuneration for RTs who are asked to provide workshops at other schools or in NHP training. In addition, the issue of giving RTs official recognition and credit for their training could be discussed with the Human Resources Division of MOEYC, with the goal of moving RTs with specified qualifications and training to another step on the salary scale. Technology As noted in Chapter III, the current NHP technology plan does not include a strategy for simultaneously putting computers and other educational technology in a significant number of NHP classrooms or for training their teachers in its use, while the proposed multimedia resource center is being established and completes a small pilot study. The team has proposed the immediate development of a complementary, phased, multiyear strategy to accomplish this, with the first phase to be implemented during the remainder of the current project and subsequent phases under NHP II. Leadership and governance training NHP experience to date clearly demonstrates that this component is of great importance. An effective strategy is needed to fully implement it, with emphasis on principals, boards of management, and parents. A review could be undertaken of locally available training institutions, such as the Coalition for Better Parenting (CBP), which might help supplement NCE efforts. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 1 · 1 ANNEX 1 LIST OF CONTACTS Government of Jamaica Ministry of Education, Youth and Culture (MOEYC) Ms. Beverly Lawrence, Director, Project Management & Technical Services Ms. Valerie Been, Director, Planning & Development Ms. Jean Hastings, Project Director, PESP Ms. Diane Brown, Project Manager, Demonstration Schools, PESP Mrs. Helen Robertson, School Feeding Programme Mrs. Elizabeth McLean, School Feeding Programme Ms. Claudette Carter, Director, Media Services Unit Ms. Jean Beaumont, Instructional Technology Manager, PESP (Media Services Unit) Ms. Yvonnette J. Marshall, Senior Education Officer, Tertiary Unit Ms. Phylicia Marshall, Senior Education Officer, Tertiary Unit Ms. Maxine Headlam, Territorial Education Officer, District 1.7 Mr. James Spencer, Senior Education Officer, Region 1 Dr. Fitz Russell, Director, Student Assessment Unit Ms. Charlotte Lewis, Student Assessment Unit Ms. Sharon Neil, Student Assessment Unit Ms. Beverly Thompson, Director, Professional Development Unit New Horizons Project/Project Implementation Unit Dr. Ernest O’Neil, Project Director (J&A) Mr. Samuel Myers, Literacy Specialist (J&A) Ms. Laurel Brent-Harris, Literacy Specialist (MOEYC) Mr. David Morgan, Mathematics Specialist (J&A) Ms. Beatrice Wilson Mathematics Specialist (MEOYC) Ms. Yasmin Yusuf Khalil, Evaluation Specialist (J&A) Ms. Bernice Fearon, Training Coordinator (MOEYC/Professional Development Unit) Ms. Kareen Edwards-Brown, Program Implementation Officer (NCE) Ms. Patricia Thompson, NHP Nutritionist (J&A) Ms. Nika Duncan, EMIS Consultant (J&A) Ms. Leodis Douglas, Administrator (MOEYC) Ms. Beth Vaughan, Executive Assistant (J&A) Ms. Valda Brown, Executive Secretary (J&A) Ms. Oniece Grant, Accountant, Project Accountant (J&A) Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 1 · 2 Aguirre International U.S. Government USAID/Jamaica Ms. Mosina Jordan, Director Ms. Claire Spence, Deputy Director, General Development Office (GDO) Ms. Robin Brinkley, Director, Office of Project Development and Management (OPDM) Mr. Charles Clayton, OPDM Ms. Joan Davis, Assistant to the Deputy Director Ms. Jennifer Knight-Johnson, Project Management Specialist, Adolescent Reproductive Health Project Ms. Tameka Utto, Financial Support Advisor, Health Ms. Margot Francis, Contracts Office Ms. Maxine Johnson, OPDM Other Donors Dr. Everett Allen, Program Officer, IDB Ms. Kerida McDonald, Project Officer, UNICEF Ms. Patricia Johnson, Project Manager, Jamaica All Age Schools Project, DFID Others Ms. Ruth Morris, Executive Director, NCE Professor Errol Miller, Chairman Institute of Education, UWI / Chairman, JBTE Mrs. Marcia Stewart, Secretary, JBTE Secretariat Ms. Florette Blackwood, Programme Director, PACT Dr. Ray Chesterfield, Vice-President and Evaluation Specialist, J&A, Washington, D.C. Ms. Heather Johnson, Program Management Associate, J&A, Washington, D.C. Mr. James MacDonald, Financial Director, J&A, California School Contacts Allman Town Primary Mrs. Beril Jacobs, Principal Mr. Orlando Dillon, LRT (Gr. 1) Miss. Lewis, MRT (Gr. 4) Mrs. Kandi-Lee Crooks-Smith Classroom Teacher (Gr. 6) Mrs. Brown (Gr. 3), Organizer of School Breakfast Program Rev. Selvin G. Watson, Pastor, The Apostolic Worship Center – School Board Tamara Reynolds, PTA president Beverly Clark, PTA Public Relations Officer 13 other parents Students: Rashie Moriah, Chammion Blake, Jamilia Lobban, Richard Thomas, Karim Smith, Shadae Lynch, Dexter Morgan, Morgan Mitchell, Angilique Palmer Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 1 · 3 Bull Bay All Age Mrs. Ellen Mitchell, Principal Mrs. Joan Thomas, LRT (Gr. 1) Mrs. Petrona McDermott, MRT (Gr. 2) Ms. Joan Wilson, PEAP II Resource Teacher (Gr. 5) Ms. Karen Whyte, Guidance Counselor Group of 6 Parents 8 Students Chetolah Park Primary Mr. Harry Chambers, Principal Mrs. Barnett, LRT (Gr. 1) Mrs. Hollinsworth MRT, (Gr. 4) Mrs. Valerie McLeod, Guidance Counselor Mr. Derrick Lewis, PTA President Ms. Paulette Oliver, PTA Secretary Mr. Allan Staples, School Board Chairman Students: Melissa Green, Natoya Pine, Phillip Anglin, Jodi-Ann Yapp, Renee Cunningham, Richard Logan Crescent Primary Ms. Nobia Robinson-Spence, Principal Mrs. Avolda Baghaloo, LRT Mr. Allan White, Chairman, School Board Devon Pen All Age Mrs. Norman, Acting Principal Mrs. Brown, LRT Daphne Ferguson, PTA Vice President Members of PTA at regular meeting (approximately 12 parents and 3 teachers) Students: Jennilyn Burchell, Shallini Stanley, Sheenell Dawkins, Raoul Brown, Ramon Clarke, Adrian Ayre Drew's Avenue Primary Mr. Ray Howel, Principal Mrs. Brown, Head of PTA Committee for School Breakfast program Kellits Primary T.A. Fearon Principal Ms. Olive Bygrane, LRT Linstead Primary and Junior High Mrs. Esmie Hemans, Principal Ms. Loretta Henry, Vice Principal Ms. Lillian Reid, LRT (Gr. 1) Ms. Gillian Ludford, LRT (Gr. 3) Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 1 · 4 Aguirre International Ms. Cynthia Silvera, MRT (Gr.3) Ms. Donnette Jackson, MRT (Gr. 4) Ms. Althea Smith, MRT (Gr. 6) Mr. Levi Croll, President, PTA Ms. Dorothy Outar, Secretary, PTA Ms. Jennifer Walker, parent Students: Alicia Ramsarop, Ayvia Francis, Keron Calvert, Sheena Taylor, Deon Stewart, Johnaire Brown, Rushane Thomas, Delmar Garricks, Alex Archer, Asheika Pryce, Joenelle Gordon, Chantelle Morris Little Bay All Age Ms. Phyllis C. Morris, Principal Miss Miller, Classroom Teacher (Gr. 4-5) Mrs. Thelma Robinson, Classroom Teacher (Gr.1) School Cook Mrs. Dorothy Shoreman, Community Member Students: Stedson Palmer, Roxane Miller, Christine Samms, Kristal Bremmer, Melinda McNeil, Kamar Clayton, Tameka Pearson, Lasana Bremmer Maxfield Park Primary Mr. Noel Watt, Principal Negril All Age Mrs. Beverly Wedderburn, Principal Mrs. Lalline Johnson, LRT (Gr. 2) Mr. Charles MacKenzie, MRT (Gr. 5) Ms. Fiona Vassell, Guidance Counselor Group of 3 parents, including PTA President Students: Leshawn Grant, Jamall Kerr, Rhea Gayle, Chadd-Rick Clarke, Yanique Allen, Crystal Gavin, Kosiano Fraser, Kenneth Heron Norman Gardens All Age Mr. Lloyd Bennett, Principal Ms. Ena Waugh, LRT Ms. Marcia Witter, School Board Member Polly Ground Primary Ms. Beverly R. Jobson-Grant, Principal Mr. Colin Blake, MRT Port Henderson Dr. Eugenie Spence, Principal Mrs. Byfield, Vice Principal Ms. Alma Whyte, Classroom teacher (Gr. 6) Ms. Janet Peyser, Classroom teacher (Gr. 4) Ms. Arlene Small, Classroom teacher (Gr. 1) Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 1 · 5 Group of parents Students: Shelly-Ann McKenzie, Sheldon McKenzie, Christopher Campbell, Gary Coley, Anike Stephenson, Loriston Daley, Krystan Henry Redwood All Age Ms. Olive Walker, Principal Ms. Ivorlyn Samuels, LRT Ms. Felicia Edwards, Parent Rennock Lodge Primary Mr. Hopeton Allman, Principal Mrs. Berley, Teacher (Grades 5/6) Mrs. Beverly Wiggan, Teacher (Grades 1/2) Mrs. Beverly Taylor, Grandparent Mr. Brenton Clarke, Parent Students: Nicholas Brown, Rasheel Pinnock Spring Vale Primary Mr. Lennox Davis, Principal Miss James, LRT (Gr. 2-3) Miss Monica Henry, MRT (Gr. 4-5-6) Mrs. Kelly, PTA President St. Peter Claver Primary Mrs. Margaret Brissett-Bolt Mr. Jennifer Brown, LRT Mr. Orville Rose, Parent Windsor Castle All Age Mr. Harvey, Principal Mrs. Olga Carwin Terrier, Vice Principal Ms. Dorothy Richards, Acting Vice Principal Ms. Joan Willie Senior Teacher Ms. Elaine Brown, Teacher Ms. Ruth Hamilton, MRT (Gr. 1) Mrs. Bryan, LRT (Gr. 2) Librarian/media specialist Mrs. Beryl Phillips, School Board Chairperson Mrs. Andrea Jones, Secretary of the PTA Mr. Melbourne Morgan, Public Relations Officer of the PTA Students: Davion Scott, Janice Ogeare, Dacia Kahn, Oshane Brown, Marsha-Lee Hutchinson, Shayne Rashford Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 6 Aguirre International ANNEX 2 SITE VISIT REPORTS Schools visited are listed in random order and have been identified only with numbers, to help ensure their anonymity. School #1 Date of Visit: June 14, 2002 Enrollment Oct, 2001: 370 Region: 4 Type: Urban/Medium, NHP Parish: Westmoreland School Feeding Program: Phase 1 Background: In one wing of this All Age school, all of the classrooms open onto a courtyard. Another section is a large room divided into several classrooms. All of the classrooms were extremely crowded, with a high noise level simply from the teaching and interactions one could hear from all the neighboring classrooms. The principal mentioned proudly that the classrooms are all ventilated by ceiling fans purchased through school fundraising efforts. All observations and interviews at this school were completed on June 14th. Three classrooms were observed, and the principal, students and a group of parents were interviewed. Interviews were also held with both RTs and with the guidance counselor, who is responsible for the school breakfast program. It was not possible to schedule an interview with the School Development Team; however, several of the people interviewed had participated in the SDP process and were able to provide the necessary information. One evaluation team member attended a gospel concert offered by the school as a fund-raiser for the breakfast program. School Development Plan: The School Development Plan (SDP) has provided a useful focus. Although a 5-year plan had been a requirement of MOEYC, the principal stated that it was only after the special training received through the project that she really understood the purpose and process of developing the plan. The team included parents, teachers, the PTA president and School Board Chairman. Interviewees believe that the project has had a positive impact on the school. They stated that attendance has improved, and attributed this to the breakfast program as well as to the fact that the more active learning styles fostered by the project have encouraged student interest and participation. The principal commented that, in relation to the school’s listed SDP goals, “We are about half way there.” Training: The Resource Teachers routinely provide workshops for the other teachers on strategies they’ve learned at NHP sessions, and they are frequently asked for help by other teachers. They have provided some training to other schools. They would like to have continued training and on-site support, and suggested more training on cooperative learning, how to work with lower achieving children, and how to use materials and technology. One teacher said the strategies seemed hard at first, but that now teaching is much more Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 7 interesting. “I really like this program because the teachers are more rounded and so are the students.” One difficulty they note is the need to leave their students in order to attend NHP activities or to help other teachers. Staff would appreciate a small library of professional materials for their own use. Materials: The school is well-equipped with audiovisual material, with television sets, a VCR, a camera, a photocopier, computers, printers, tape recorders, a CD player, a camcorder, and an overhead projector. Some of the students are allowed to use the computers. Texts and manipulative materials were in evidence and were used in lessons. Supplementary books were out and available to students, and teachers commented on how much the children liked the colorful NHP books. The Guidance Counselor, several teachers and the principal use the JSAS database on a computer in the principal's office. It is considered useful and user-friendly by the staff, but there are concerns about finding time to enter all the data. Classroom Observations: Three classrooms were observed. Most desks were facing forward, and a teacher-centered style was mixed with methods in which learners were actively involved. Within the teacher-centered presentations, several NHP strategies were being used, and the children seemed interested and involved. Lessons were well-planned, with linkages to previous lessons, and questioning at higher levels took place. In spite of extraordinarily crowded conditions, two classrooms had reading centers, and another had several interest centers. Teachers were maintaining both teacher and student portfolios and student journals were in use, with some evidence of original student writing. School Nutrition: Staff and participating parents discussed the selection of 20% of students for participation in the school breakfast program as based on needs defined as: school achievement, health signs such as pallor, and family socio-economic conditions as known by staff. The school breakfast program is seen as part of a larger focus on health, and frequent school events are organized that include a health focus, such as a Sports Day with presentations by staff and students on health issues. Recently, the height and weight of students participating in the school breakfast program was assessed. The parents interviewed who have participated in workshops for the school breakfast program rated the quality of the workshops as high and stated they received excellent and frequent follow-up on-site from both the NHP nutrition support member and the NCE Community Specialist. Teachers and parents interviewed stated that they see a difference in children that they attribute to the effects of the school breakfast. Children who used to fall asleep in class or appeared listless and lethargic now seem to them to be more alert and interested. It is their impression that these children are learning more than they used to, even though they may not have attained the formal literacy goals. The Guidance Counselor also feels that better nutrition has contributed to improvements in self-esteem that she notes through student dress, posture and deportment. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 8 Aguirre International Parent and Community Involvement: Parent and community support has increased somewhat but is still insufficient from the school's perspective. For example, only two parents can be relied upon to prepare the school breakfast. (These two people live at least 4 km. distant and receive no remuneration for transportation.) This is quite a challenge, due to the economic hardship of families of students at this all-age urban school. The school came up with another idea to begin a committee for an income-generating project to support remuneration of the breakfast cooks; however, the committee leader has emigrated recently. They plan to revitalize this committee soon. Parents have learned fund-raising techniques from NHP workshop linked to the breakfast program, and ample evidence of community support in preparation for the next day's fundraising concert was seen. School #2 Date: June 11, 2002 Enrollment: 226 Region: 1 Type: Non-NHP Parish: Kingston Background: This All Age is a Non-New Horizons school selected for comparison in the project evaluation. It is a multi-grade inner-city school located in a noisy industrial area. Three teachers are assigned to teach the primary grades. While the principal reports population growth from forty-eight students in 1997 to the present 226, there is discontent that the school is seen as a last resort for parents who send their children only when they are unable to get them in other schools. The school was selected for comparison since it was among the list of schools that pre￾qualified as a NHP school but was not among the final seventy-two schools selected. The principal has served in the school for six years. The school has not been the beneficiary of any major projects or initiatives. The school is housed in a relatively small concrete building. The fact that there are no fences between the school and the community is cited as a major problem by the principal. Grades 1-6 are housed in a single hall partitioned in three sections by chalkboard and cupboards. The constant noise from local industry prevailed during our observations. At the time of visit there was adequate seating, the rooms were well ventilated and lighting was adequate. School Development Plan: Teachers indicated that the development planning started but was not completed. One resource teacher described the plan as a document that outlined schedules (e.g., what we’ll do for different days, such as devotion). The plan was described as “more like a calendar of events.” This calendar included staff development sessions, workshops and PTA meetings. The principal indicated that staff meet weekly during common planning time—trying to help each other. In these meetings they might talk about reading, lesson planning, and conflict resolution. The big problem earlier was violence at the school. He has a zero tolerance policy for violence now and feels that things have improved. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 9 Training: Teachers indicated that they attended several workshops arranged by the Ministry of Education (e.g., April workshop on assessment). One concern was that more time was needed for workshops. The school has no language or math resource teacher. One member of staff was trained as a religious education resource teacher. Teachers indicated the need for training on multi-grade teaching techniques and strategies, as well as counseling and remedial reading. Materials: The school has an administrative computer that was donated by the Ministry of Education. There are 30 other computers donated by past students. Teachers consented to the conversion of their staff room to a computer room. There is also an overhead projector, television and cassette player in the school. Boxes and cupboards of books provided by MOEYC are also evident in the classrooms. Children indicated that they liked school because they had “a lot of things to play with” and "a lot of books to read." Teachers cited the LMW (a basal published in the early 1980’s) teachers’ guide as resource material for instruction. Teachers have also received teaching manuals for math and science from MOEYC. Classroom Observations: All three primary classrooms were observed for thirty minutes each. Teachers indicated that they were conducting revision for the upcoming end of year tests. In one classroom, students were reviewing letter writing where students were instructed to write friendly letters. The teacher explained that she had previously reviewed the “stages” of writing a letter. On the whole, students were quietly engaged in seatwork. There was little interaction between students and teacher. Although students varied in their literacy competencies, this was not accounted for in the assignment of the task. Children wrote without the use of any reference materials (e.g., dictionary). At the end of the session, some students had completed their letters while others were still struggling to copy the teacher's written instructions on the chalkboard. No whole class interaction occurred for the duration of the observation. A few charts were displayed on the walls. However, there was little evidence of recent efforts to produce these instructional aids. Children’s work was not displayed and no corners or centers of interest were evident. On a whole, the children in the classes observed were struggling readers and writers but little seemed to be in place to address their needs. Teaching was traditional, and classrooms were quiet and orderly. Classroom interaction was highly teacher-centered, with little provision for student differences. In one classroom, students responded to teacher questions in unison. School Nutrition: There was a breakfast program at the school but this has been discontinued. A cooked lunch program exists. Parent and Community Involvement: Parental involvement was described as minimal at the school. A few parents come in from time to time to assist at the school. Very few parents help their children with schoolwork. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 10 Aguirre International School # 3 Dates of Visits: June 10/19, 2002 Enrollment Oct, 2001: 521 Region: 1 Type: Inner-city/Medium size, NHP Parish: Kingston School Feeding Program: Phase 2 Background: This primary school is in an inner-city area with considerable poverty and many unemployed parents. Its SDP (2001-2) mentions “the stigma attached to the school as underachiever, indiscipline and located in an inner-city community known for crime and violence.” However, strengths mentioned include a 100% trained teaching staff and adequate infrastructure (light, telephone, water, ventilation, etc.). This was the only school visited in which the classrooms observed were of adequate size and were separated by concrete walls, so that teachers and children could be heard. This school was visited twice. On the first visit, interviews were held with the LRT and the teacher organizing the new breakfast program. On the second visit, the two RTs held demonstration classes for the team, and interviews were held with the principal, the MRT, 15 parents who were members of the PTA and the SDT, a group of children from grades 1 – 6, and a School Board member. School Development Plan: The principal felt that both parents and teachers initially perceived the SDP as more work. At first, parents were not accustomed to being involved and did not see how this process would benefit them or their children. Now she notes more interest and understanding in the planning process, although it is still difficult to get all groups together in the same meeting so that they can actually exchange ideas in person. Interviewees noted that the SDP goals have provided them with a specific focus and a “vision” that have helped them reach objectives. One specific example was the provision of a Friday Club program that was one of several successful measures taken to improve attendance. The following changes were attributed to the project: teachers now plan together; attendance has improved; improvements are noted among the students in attitude, behavior, and learning; parents and community members are more involved. Training: All of the training provided was perceived as useful, and that provided to the RTs was considered to be a great strength. The RTs always provide workshops for the other teachers on strategies presented in NHP workshops and have also provided workshops for other schools. The training on cooperative learning was particularly appreciated. The principal noted the difficulty of having the RTs removed so frequently from their classrooms, and recommended that an assistant teacher be provided for substitute coverage when RTs are attending workshops or assisting other teachers in their classrooms. She felt that more on-site follow-up was needed, with demonstration lessons or videos of effective practice, so that teachers could see effective use of strategies in classrooms like their own. Two interviewees recommended that all teachers receive training such as that provided to the Resource Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 11 Teachers. Further training was also recommended in the use of materials and equipment, as well as in information technology. Materials: Books and materials were in evidence in the classrooms and showed signs of use. Virtually all interviewees mentioned their appreciation for the materials and mentioned that students enjoyed reading the colorful books. It was suggested, however, that teachers should be allowed to select their own materials. This school has two computers, one provided by NHP. Unfortunately, in order to participate in NHP, the school had to withdraw from its planned participation in another project that would have provided it with a computer lab. They still have a room ready to be turned into a computer lab, and would like to have more computers available for student use. Classroom Observations: Two classrooms were observed. Strategies encouraged by NHP were in evidence, as were interest centers. Rooms were decorated with colorful charts, and in one classroom student work was displayed. One class was largely teacher-centered, but with a high level of student interest and participation. In another class, students worked in groups to create various kinds of graphs. Some cooperative learning strategies were observed, and audio-visual equipment was used effectively in both classrooms. School Nutrition: The school is preparing to initiate a breakfast program for the school term beginning in September 2002. Those interviewed stated that the preparation process through training workshops has been very participatory; for example, parents assisted in developing the weekly menu. At present, at least 15 parents are actively supporting the preparation process by planning fundraising activities, developing a Past Student list to encourage donations and participation, and sending solicitation letters with follow-up visits or phone calls to businesses in the local neighborhood. Selection of students will be need-based, with an expected enrollment of approximately 15% of students in the breakfast program. Parents will serve as volunteer cooks on a rotating basis. Parent and Community Participation: Interviewees felt that there is now more parent and community involvement in the school, as a result of NHP. For example, parents often support the school by substituting for teachers when they attend workshops or on Teachers' Day, and they provide good support for school beautification activities. At present, 30-50 parents regularly attend PTA meetings. The preparations for the school breakfast program mentioned in the previous section have also served as a vehicle for increasing parent and community involvement. A relative weakness has been the introduction of parent Study Circles, which have not really caught on. The PTA representatives perceive the need to improve communication between parents and the school in a positive manner, through conflict negotiation and encouraging parents to use the PTA as a channel to address their needs. Parents mentioned that NHP workshops have provided training on these skills. Workshops also provided information on assistance agencies and available services in Jamaica. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 12 Aguirre International School #4 Dates of Visits: June 12, 18, 2002 Enrollment, Oct., 2001: 411 Region: 2 Type: Urban, medium, NHP Parish: Portland School Feeding Program: Non-NHP Background: This All-Age school operates on a shift system. The school consists of three long buildings closely parallel, with two of the buildings each subdivided into 3 classrooms by chalkboards. The third building is divided by concrete walls into a storage room and canteen, a future computer lab, and a school library. The school enrollment has steadily increased in the past two years. Evaluation activities included 3 classroom observations and interviews with the principal, both RTs, the librarian/media specialist, a group of parents including the PTA secretary and Public Relations Officer, the School Board Chairperson, and a group of 6 students. School Development Plan: The School Development Team (SDT) consists of 10 persons representing the school (including a student representative from this All-Age school) and the community almost equally. The SDP has been a vehicle for disseminating the goals of the New Horizons Project to the community at large. The SDT has evaluated the school’s progress thus far in achieving the goals of the SDP. Achievements include: an increase in attendance by 6% in 2 years (with a target of 10% in 3 years); an increase in literacy of students as shown by Grade 4 assessment results in which only 2 students did not achieve mastery or near-mastery, resulting in no need for a summer literacy camp this year; community-supported improvement in the school environment, including school beautification; and donations obtained from the private sector. It was suggested that the project be expanded to other schools in the future, and that current NHP schools and staff might help in presenting the program to new schools. Training: Teachers interviewed found the strategies taught by NHP to be particularly useful, and mentioned liking particularly the KWL technique (What I Know, Want to know, and want to Learn) and cooperative learning. Interviewees suggested that sometimes it would be useful to be provided with more background information on a topic, and suggested that professional materials for use by teachers would be appreciated. Also suggested was training for how to work with the lower-achieving children—both diagnostic methods and “how to reach them” with diversified materials and techniques. The RTs thought it would be useful to have more training in how to give presentations and workshops for other teachers, and wished the whole school together could receive training. More on-site follow-up is needed. Classroom Observations: The three classes observed each had 24 to 30 students, in classrooms partitioned by blackboards. The noise level was high due to the proximity between buildings and the subdivision of large rooms into classrooms. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 13 The students were seated 2 children per desk or with an individual chair at a dual desk. Most desks were in rows facing the teacher, with one class grouping the students in sets of two desks facing each other; in this classroom, learner sharing was encouraged. There were no centers of interest evident, but some teachers had been quite creative in using mobiles on a clothesline so that materials could be stored and covered against the wall for the other shift class. There was a small amount of children's work displayed. There was some evidence of lesson planning for an integrated curriculum with one teacher linking the lesson to previous lessons. Most teaching was teacher-centered; however, one teacher allowed students to suggest each of the elements of a class exercise, and students used the chalkboard to demonstrate the lesson. Journal writing was encouraged. Encouragement of student questioning was shown in one class; it should be noted that noise level due to infrastructure limitations made teacher-student interactions in the group setting very difficult. The non-NHP trained librarian was observed conducting reading periods with children. The librarian used reading comprehension strategies, modeling reciprocal teaching techniques and also described the use of educational technologies, such as computers or video, for learning. Materials: One class had books accessible, and these were utilized during a reading period that occurred between two planned lessons. The books appeared attractive and culturally appropriate for the most part. A large number of attractive traditional charts were hanging from clotheslines so that they could be pushed to the walls and covered during use of the classroom by another class in the afternoon. Texts were used in another classroom. The school library was attractive, and media materials were in use by students during both evaluation visits. The librarian assists small groups of 2-4 students to use educational software which he has obtained on his own initiative, and also connects the computer to a large-screen TV for video access by a larger number of students. The school is using the JSAS database and finds it to be user-friendly. Several teachers have received on-site training that they describe as appropriate to their individual learning needs. Administrative information and roughly half of student attendance data for the year have been entered. The staff is aware of how and where to enter assessment scores and plan to do this when scores are returned from MOEYC. School Nutrition: Although the school does not participate in the NHP-promoted school breakfast program, the principal has started a small breakfast program for a limited number of children due to a need perceived by the principal and all teachers. This program is operated through occasional community donations from the private sector and a sliding scale fee for participating children. Parent and Community Participation: Parents interviewed stated the importance of their role in supporting their children's scholastic efforts in a variety of ways: ensuring their punctual attendance at school, reviewing their exercise books and test scores, and helping Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 14 Aguirre International them with homework. A parents’ Study Circle was started but failed to get off the ground. There are plans to reinvigorate this component next school term. School #5 Date: June 14, 2002 Enrollment: 52 (Grades 1- 9) Region: 4 Type: Small, rural, non-NHP Parish: Westmoreland Background: This is a small rural all-age school that does not participate in the New Horizons Project. It is located on the coast, and most families are said to rely on fishing for income. Road access is said to be difficult during rains, and the cost of transport to larger towns (and/or other schools) is expensive. There are 4 teachers who currently serve this multi-grade school, from Grades 1 to 9, including the principal, who teaches Grades 2 and 3. Enrollment is said to be decreasing for a variety of reasons, including migration by families to larger towns. This school is in an area visited by tourists, and has received some continuing assistance from a teacher abroad who has a summer home in the area. The school consists of one long building that is divided into 3 classrooms by plywood partition or chalkboard. The building is well-ventilated. The grounds are fenced in and well￾kept. There are sufficient desks for students, and these were somewhat grouped in a non￾traditional manner. There are no centers of interest, as such, but there was evidence of gathered materials, such as seashells, arranged by size and type and accessible to students on a small table. Student work was not displayed on the day of observations. Many of the students have been to Basic School; however, the school occasionally admits first grade children as young as 4 1/2, when no alternative schooling is available for them. In an interesting counterpoint to the worry of the principal over declining enrollment, the students commented on the advantages of being in such a small school. The Principal, 2 teachers, a group of students, the school cook, and an active community member were interviewed. School Development Plan: The school has had an SDP; all of the teachers and the School Board chairman were members of the committee. A teacher commented that having a plan was useful to provide a focus for their actions. The school’s main goal is to get every child to read. The school does not presently have an updated SDP as community interest has waned over the past few years. It is planned to revitalize this, however, during the next school year. Training: Teachers received the training provided by MOEYC on the new integrated curriculum and are somewhat conversant with the new terminology. They have little contact with teachers from other schools and would like more training and follow-up. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 15 Materials: Ministry texts are available, although the school did not receive enough this year. They do have a donated copy machine, and so were able to copy texts for some children. Charts were displayed in classrooms. Student activity books were in use, and storybooks were accessible to students and showed signs of use. The grade 1-2 classroom had a tape recorder and Big Books, donated by a friend overseas. However, interviewees commented that they need more books, stating that parents cannot afford to buy the integrated texts they saw at Ministry workshops. Two donated computers, which were being used with educational software by small groups of students during the lunch period, are housed in a closet-sized room next to the principal's office. Classroom Observations: The evaluation team observed 3 classes in session. There were an average of 10 students (multi-grade) per class. Lessons were planned, with some evidence of linkage to previous lessons, and they incorporated student experiences to some degree. Teaching was predominantly teacher-centered, but teachers used some methods that involved learners. Most teacher questions required only recall, but a few required inferential thinking. Supplementary material was used in one class. Evidence of teacher reading to students and some original student writing was seen. One teacher maintained student portfolios, although these contained only a few pages. School Nutrition: Students pay on a sliding scale, up to J$30 (US$0.65) for a cooked lunch program at the school which is partially supported through rice, cornmeal and flour (with occasional oil, tinned mackerel or beef) provided by MOEYC and through a portion of the school grant (J$8,000 per term). However, the delivery trucks do not extend to such a rural area, and the principal uses her own vehicle to pick-up the commodities in the nearest town. A separate kitchen exists, with a new stove provided by MOEYC one year ago; however, the stove is not functioning, and the school cannot find a skilled repairman to come to this remote area. Parent and Community Involvement: A group of foreigners have invested in camping￾type tourist facilities in the immediate area. They have assisted in revitalizing a Citizen's Association and bring some donated materials from overseas, such as books and computers. Approximately twenty parents regularly meet through the school PTA and the principal states that community response for specific activities, such as Work Day, is acceptable. An adult literacy program (JAMAL) has been initiated through the school and some parents participate. This year's School Board consists of 7 entirely new members. School #6 Dates of Visits: June 10, 18, 2002 Enrollment, Oct., 2001: 120 Region: 1 Type: Small, inner-city, NHP Parish: Kingston School Feeding Program: Phase 2 Background: This is a small urban primary school that has been confronting serious problems within the past year or two. There has been a recent upsurge in violence in the area, Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 16 Aguirre International and this led to the resignation of 5 teachers and the withdrawal of children from the school by many parents. At present, unused classrooms are falling into disrepair, and the school is becoming multi-grade with a total of 5 teachers (including the principal, who also teaches). This school was visited twice. During the first visit, three classrooms were observed. In the second visit, interviews were conducted with the school principal, the 2 RT teachers, the School Board Chairperson, 2 officers of the PTA, the Guidance Counselor and a group of 6 students from grades 3-6. An Education Officer, new to this school, was present during the site visit and was also interviewed. She stated that MOEYC was aware that this school was experiencing serious difficulties and intended to provide close, frequent supervision over the next year. School Development Plan: Mixed information was presented regarding activities of the SDT. The SDP had been completed shortly before the evaluation team visit. One SDT source stated that all of the staff, 4 PTA members and the Chairman of the School Board had participated in the development of the plan, along with community members and past students. However, some other interviewees seemed unaware of the plan. The School Board Chairman stated that he was unaware that any plan had been developed and had never seen one. He quoted from an official evaluation report on the school that stated “there is no documentation of objectives and goals for the school.” He felt these were badly needed. Training: The principal, RTs, and parents have participated in NHP training activities. Reading continues to be perceived as a serious problem, and interviewees requested training on how to deal with multi-grade classes, as well as specific training on how to deal with children having trouble learning to read. They would also like more training in how to use materials such as the overhead projector. Materials: Both classrooms observed were decorated with charts. There were multiple copies of different types of mathematics activities books, spelling and introductory reading activities books, as well as various copies of several types of storybooks for reading. All of these materials showed signs of use and were somewhat accessible in stacked piles along the wall. The school is preparing a library/computer resource room and has a total of four computers. The Guidance Counselor was observed entering information in JSAS on the administrative computer provided by NHP, assisting the principal in updating class attendance. The Guidance Counselor is fully conversant with the capabilities of the system, including recent upgrade changes, and sees it as useful. One suggestion, however, was increased password protection for the teacher profile information. Classroom Observations: Two classrooms were observed. Due to absence of one teacher, one multi-grade 1-2 class also had a pre-school class present the day of the site visit for a total of 46 children in one classroom. This teacher was also checking occasionally on a third grade class whose teacher was absent. One classroom was part of a larger room with separation into two rooms by a blackboard, while the other was a single room with concrete Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 17 walls. The lights were not functioning, due to a storm, so the room was dark. It was also difficult for students to hear, due to the noise level and the storm. The students were seated with 2-3 children per desk; desks were in rows facing the teacher. There were no centers of interest evident in either classroom. One class had a moderate amount of children's work prominently displayed, while the other had one example. As this was the week before exams, both teachers conducted reviews. An integrated curriculum was presented, incorporating language, math and social sciences, and demonstrated previous preparation of a lesson plan. Both lessons were teacher centered, in a traditional question and answer format, with some extension beyond recall to broader thinking. There was some accommodation of learners' experiences. The blackboard was the only educational material utilized. In one classroom, students generally showed enthusiasm to answer the teacher's questions and/or to do their written exercise, whereas in the other, few children were attending to the teacher. A sample of student notebooks showed completed exercises involving simple recall and copied phrases. School Nutrition: The school will participate in the NHP school breakfast program starting in September 2002. The school Guidance Counselor selects the students who will participate and the level of payment expected, with a sliding scale from 0 to J$25 (US$0.54) anticipated, based on knowledge of each student's situation. A cooked school lunch is regularly provided to approximately 70-80 students, with half receiving lunch free and half paying from J$15 to J$25 per lunch. MOEYC regularly provides rice, flour and cooking oil for the school lunch and, occasionally, other items such as tinned mackerel or beef. Student lunch fees are used to purchase additional foods to provide a balanced lunch. The cook and kitchen condition demonstrated a good standard of hygiene. Parent/Community Involvement: At present, the parents/PTA see their role within NHP as assisting in bringing the community and school closer together. Parent workshops received have clearly articulated the need for improved communication between both parents and their children and also between the school, parents and the community at-large. School #7 Dates of Visits: June 11, 17, 2002 Enrollment: 650 Region: 1 Type: Non-NHP Parish: St. Catherine Background: This Primary School is a Non-New Horizons school selected for comparison in the project evaluation. It is a double shift suburban school that serves children of working class parents. Despite the existence of the shift system, overcrowding is still a major problem in the school. The school was selected for comparison since it was among the list of schools that pre￾qualified as a NHP school but was not among the final seventy-two schools selected. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 18 Aguirre International Located next to a major road on the outskirts of Kingston, the school is housed in three small buildings. Rooms are shared with similar grade levels on the other shift. Classrooms are small, and there is little room for creative seating configuration. Two classrooms were observed in this school, and interviews were held with the principal, a group of parents, and three teachers. School Development Plan: Parents, teachers and principal were engaged in the school development planning process. Stakeholders articulated the concept of making an annual plan. However, they did not make a total one-year plan because things “came in as time went by.” They indicated that they met once per month at PTA meetings to plan for everything. The priorities for the school development plan included having more classrooms and improved facilities and environment. Literacy and numeracy gains were not among the stated priorities of parents. However, teachers highlighted literacy and numeracy at the end of Grade 6 as a prime target. Respondents were unable to clearly define a route to achieving the literacy and numeracy goals. They provided responses such as phonics and reading as routes to achieving their learning goals. The notion that the school building was small and housed more than the prescribed number of students seemed to influence the plan most. Parents and children cited students’ disciplinary problems as one of the major challenges facing the school community. Training: Parents and teachers indicated that they received training as a result of their involvement with the school. Teachers participated in the grades one and four curriculum implementation seminar, and a grade five teacher attended a social studies seminar. Parents also indicated that a seminar is held for parents who “do not have the know-how” to help their children. Materials: The school has a television, a videocassette recorder, a cassette recorder, a risograph, and one computer for administrative purposes (without an internet connection). However, there is no overhead projector. Use of these materials was not evident in the classes observed. Parents were not aware that there was a computer in the school. However, their perspective on the use of a computer includes getting their children computer literate, engaging in computer-aided reading and connecting to the Internet. According to one parent, “we are all for technology. This is the 21st Century and it’s all about technology.” The school has a library stocked with storybooks and reference print materials and is housed in a building that was constructed through the parents’ contributions. The school credits the Jamaica Library Service for most of the books. In the classrooms, there was little evidence of teacher-made or pupil-made materials on display. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 19 Classroom Observations: Two classrooms were observed for thirty minutes each. Two teachers “staged” mini lessons for the evaluation team because it was examination time and teaching had virtually ended for the academic year. Because the sessions were unplanned, it was difficult to identify the instructional merits in the observation. Classes were mostly teacher-centered. However, in grade one there was dialogue between teacher and students. It was evident that many of these children were able to read and write, and this facilitated the instruction. In another grade, there was very little participation by students. No books or materials were used, although the teacher did use an instructional chart. School Nutrition: There is a lunch program that provides cooked lunches for children on both shifts. Children interviewed indicated that some students had lunch at school while others ate elsewhere. There was a breakfast program at the school but this was discontinued due to lack of resources. The school continues, on an informal basis, to provide breakfast from the canteen for some children who attend school without eating breakfast. Parents and Community Participation: There is strong evidence of parent and community involvement at this school. Stakeholders interviewed used the construction of a small block that houses the staff room and library as a symbol of parental interest and involvement in the welfare of the school. The focal points of this involvement were largely to improve the school facilities and address disciplinary concerns. School #8 Date: June 13, 21, 2002 Enrollment: 1942 Region: 1 Type: Large Urban, NHP Parish: St. Catherine School Feeding Program: No Background: This Primary & Junior High is an urban New Horizons school that operates on the two shift system. Currently, the grade six to the junior high department attends the school on the morning shift while grades one to five attend on the afternoon shift. The school is in close proximity to a major bauxite plant and is approximately an hour from Kingston. Thirty￾eight teachers serve grades one to six, including a total of six resource teachers. There are 24 teachers with a teacher’s diploma, five with university degrees, seven with a teacher’s certificate and two teachers without professional certification. This school has three blocks of buildings that house 12 classrooms, a library, resource and technology centers (for junior high subjects) canteen, administrative offices, and a staff room. Some classes are partitioned by chalkboards with only grillwork enclosing one side of the building. Overcrowding seems to be a major concern. However, classrooms are well ventilated and lighting is adequate. The principal, 3 of the 6 RTs and a group of students were interviewed during the first site visit. On the second visit, interviews were held with members of the Project Development Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 20 Aguirre International Team, including all of the RTs, the Vice Principal, 2 officers of the PTA and a parent. In addition, a group of students was interviewed. School Development Plan: The SDP provides background information on the school, examines the current circumstances that face the school, and expresses targets of having 90% of the student population literate and numerate by 2004. The planners advocate, as means to achieving new ways of learning for the younger ones, having RTs that are well trained, materials, and practical ways of learning. Respondents describe broad-based participation of stakeholders in the planning process. The merits of the SDP process included providing focus for school activities and an end result to work for. Development planning was also credited for providing the opportunity for members to work together. Respondents cited positive outcomes of planning, including greater parental participation in children’s learning and increased contributions (cash and in-kind) to the school. The interviews revealed that students and teachers were well aware of the New Horizons project and the alignment of the project to the school goals for improved literacy and numeracy. They praised the project for the resources provided and training given to resource teachers. Teachers stated that attendance has improved, attributing this to the fact that children are now more interested in school because of the active methodology. Parent involvement has also increased. Among the problems highlighted by students was the need for a shift system, which mitigated attempts to keep the school clean. In addition, they think that some of their schoolmates “can hardly read and that is a setback for the school.” Classroom Observations: The evaluation team observed the classrooms of three resource teachers. The use of NHP strategies was evident (in varying degrees) in all cases and highly visible in some. Teachers modeled appropriate problem solving strategies and encouraged students to respond critically to literature used in the classes. The teachers were good language models and demonstrated confidence and competence in their use of the teaching strategies. While the classes were teacher oriented, students were given responsibility for predicting and locating information in texts. For example, in one lesson, the teacher modeled reciprocal teaching strategies; although the lesson was teacher-centered, students were actively involved, questioning, predicting and summarizing throughout the lesson. Classroom time management was also efficient. Activities were well timed and transitions in lessons were efficiently executed. Children in the classes observed demonstrated competence in the use of oral and written language. Some were fluent in oral reading and uttered grammatically correct sentences. Training: Teachers, the principal, and a parent received training from the project. The resource teachers commended the strategies that they were exposed to at training sessions. A significant feature of the parent’s response on her training was that her child’s education was not singularly the responsibility of the teachers. The chief suggestion for improvement of training was to have more sessions and get more persons involved. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 21 The RTs routinely provide workshops for the other teachers and assist them in using the strategies. They found the training in cooperative learning to be particularly useful, and they use student journals and student and teacher portfolios. They find it difficult to leave their students for training or project activities and wish a floating substitute teacher could be provided. They would appreciate additional training in working with lower achieving pupils, as well as more training in the use of computers and other AV materials. One observation was that some stakeholders were not aware of the training others received. For example, parents were not sure of the kind of training resource teachers received, and teachers were not sure of the kind of training the principals received. However, they could articulate in vague terms the kind of training they would expect each stakeholder to receive. Materials: Because of the school size, the NHP provided three of each of the standard audio visual aids (TV, VCR, OHP, audio cassette player) to this school. Several of these are used in a separate trailer, which is air conditioned and functions as a media resource center. The administrative computer is not used for the designated purpose due to technical difficulties experienced after loading the JSAS software. Teachers mentioned the need for training in the use of the materials. Acquiring more computers was also high on the respondents’ wish list. Despite the existence of the shift system - which entails sharing of classroom space - books, charts and other instructional materials were displayed in classrooms, and both teacher and pupil made materials were evident. School Nutrition: There is no breakfast program, but there is a cooked lunch program at the school. Parent and Community Involvement: Attendance at PTA meetings is a major strength of parental involvement. Members report as many as 800 parents attending the meetings. Children report that their parents help them with homework and other school related activities. Members of the SDT desired the implementation of study circles to maximize parental awareness of their roles and responsibilities in the school community. School #9 Date of Visit: June 12, 2002 Enrollment, Oct., 2001: 105 Region: 2 Type: Small, rural, NHP Parish: St. Mary School Feeding Program: No Background: This All Age school is a rural New Horizons school with a multi-grade instructional context. The school is located in the mountainous region of St. Mary close to a main road. Four teachers currently serve the school. There has been a recent administrative change at the school, which has had an acting principal since September 2001. This person has been a teacher at the school for many years and has received NHP training as a Resource Teacher Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 22 Aguirre International Although it is officially an All Age school, the school currently hosts students from grades 1- 6 only. This has been attributed to the placement of students who sit the GSAT in other schools. The school is housed in a two-floor concrete building on a lot with a precipitous border that descends to a river. At present, the parents and community members are assisting the school to construct a protective wall. Classes on the lower floor are partitioned by chalkboards while rooms on the upper floor have concrete partitions. There is also a resource room on the upper floor that has books and instructional computers. There is adequate seating. However, the resource room is in disrepair. Classrooms were well ventilated and lighting was adequate. The following groups and individuals were interviewed: the principal, RTs, a group of students, the Vice President of the PTA, and members of the PTA attending a regularly scheduled meeting. School Development Plan: Those interviewed included most of the members of the SDT. In the interviews, the respondents were able to candidly assess their needs and achievements. They felt that the SDP was useful, and stated that they often refer to it and check to see if they’ve reached their goals. They mentioned improvement in parental involvement; this was evident in the interaction with PTA members. In addition, more community involvement has been noted. Improvements have been noted in children’s attitude towards school, and the principal stated that attendance and test scores have improved. Training: NHP training has been perceived as very useful, and a new teacher mentioned how much support she had received from the RTs. Staff would appreciate continued training and on-site visits, and suggested that more training in the use of materials and computers would be helpful. The principal suggested that clusters of schools should meet and interact more frequently, and recommended consideration of special training for new principals and teachers. Materials: Most of the educational materials obtained through NHP are stored in the principal's office, where the administrative computer is also located. A variety of materials are available, such as combination CD player/cassette recorder and overhead projector. The school has not yet located education materials for use with these tools and suggests that workshops on the use of these tools within lesson plans would be useful to all teachers in their school. A large screen TV and video player are set-up in one classroom for use by any teacher; a few educational videos have been obtained through the public library system and used in the past school year. Books, charts and other instructional materials were displayed in classrooms, and both teacher- and pupil-made materials were on display. The NHP-provided computer is used for administrative purposes, and the principal is quite knowledgeable about the capabilities of the JSAS, such as graphic presentation of student assessment scores. She rates the training received in JSAS as excellent. Older donated Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 23 computers in the resource room are used by teachers for some lesson planning and administrative organization and by students for creative writing. Classroom Observations: The evaluation team observed all four classrooms for varying periods of time. The use of NHP strategies was minimal in some cases and visible in others. The presence of student portfolios and displayed children’s work was a notable feature. Some teachers made obvious efforts to base activities on student experience and to involve the students actively. In one classroom, children were working in groups with math manipulatives. Most of the classes were engaged in revision. There is evidence that some teachers’ view of revision reflects a competitive mode where students are tested in oral and written modes and little teaching and assessment occurs. An example of this was the use of a “part of speech” game, which “eliminated” children who could not correctly identify the part of speech. Children in the classes observed had varying degrees of literacy and numeracy competencies. There are unexplored avenues for improving the competencies of struggling readers (e.g., effective grouping, peer coaching). School Nutrition: There is a cooked lunch program at the school. However, there is no established breakfast program. All staff interviewed perceive a great need for school breakfast, stating that some students arrive to school lethargic and faint, primarily due to the instability of some of the family social structures. Parent and Community Involvement: In the interview held during the PTA meeting, very few parents knew of New Horizons, although the PTA Vice President mentioned that the training workshops she had attended had been excellent. However, parents could identify improvements in the school. The principal’s demonstration of the use of the administrative software was also significant since her database was the most up-to-date of all the schools visited. School #10 Dates of Visits: June 11, 20, 2002 Enrollment Oct, 2001: 317 Region: 1 Type: Urban/Medium, NHP Parish: St. Thomas School Feeding Program: Phase 2 Background: Although this All Age school is classified as urban, it appears to be located in an entirely rural area. The school, which is housed in several small buildings, is isolated, with no nearby houses, buildings or commercial areas, and is surrounded by fields and wooded hills. Classrooms are crowded, divided by blackboards, so that the noise level is high, and it is difficult for teachers and students to be heard. This school was visited twice. On the first visit, interviews were held with the principal and both resource teachers. On the second visit, three classrooms were observed, and interviews Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 24 Aguirre International were held with an RT trained by PEAP II as well as with parents and members of the SDT and a group of students. School Development Plan: The SDP has been useful in giving staff members a common focus and clear goals. Teachers, including the RTs, were involved, as were parents. The principal commented that was difficult at first to understand what was expected, but that now she comprehends the process and finds it provides a clearer picture of the way to achieve the goals of the school. Some staff and parents still do not “have the hang” of the SDP. Training provided on this topic was useful, and she has tried to help other principals with their SDP process. The interviews indicated that the project has had a positive impact on the school. Planning is now done together by groups of grades (1, 2, 3 together, etc.). It is perceived that students are better able to express themselves and that their writing has improved. Interviewees feel the project should be expanded to other schools. Training: The strategies provided in training are helpful, especially cooperative learning. RTs commonly provide workshops for the others on the topics of training they have received. They note some resistance from teachers and wish the classroom teachers could have the same training they had. They would appreciate it if some training could be given on the role of the RT, both for the RTs and for the classroom teachers. The principal noted that integrating the curriculum is hard for teachers. Interviewees agreed that there is an overall need for continued training and on-site visits. More training would be useful in cooperative learning, in strategies for early literacy, and in use of materials, including computers and other audio-visual equipment. Teachers would appreciate professional materials for their own use. It would also be useful to have more training for parents. Classroom Observations: Classroom observations indicated some efforts to implement NHP strategies, although the interaction was predominantly teacher-centered. In one classroom approximately 15 minutes of the lesson was spent rearranging desks and students for group work in the second part of a lesson. Tasks and questions were mostly at the recall level, although some called for higher-level thinking. Student questions were not encouraged, but some positive responses were noted to student errors. Materials: Books were present in the classrooms, and interviewees indicated great appreciation for the books and materials provided. Students also commented that they found the books and materials interesting, and that they are allowed to take the books home. The school has a sizeable school library, which includes a large number of books from donations, the Jamaica Library System, and the MOEYC series (Dr. Bird, etc.) Use of the JSAS was reviewed with the principal (other teachers have also received training and assist in entering data). Although the interviewee is only moderately experienced in using computer programs, she has found that the training received has prepared her sufficiently to feel comfortable with the system. She believes the initial time invested up front to establish the database will be paid off by multiple uses (such as easily providing Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 25 school records to community members who need these for other purposes). The computer used for administrative purposes is not used in the educational program. School Nutrition: This school has been preparing to initiate the school breakfast program when the next school term begins in September 2002. The school is notable for having raised a high amount of money in a single fundraising event in support of the breakfast program: J$17,000 (US$370) earned from the "Jamaica Day" event in March 2002. School staff and parents perceive a need for a breakfast program to encourage punctual attendance, to increase student alertness in morning classes, and to provide a balanced hot nutritious meal. The school does not have a school lunch program, but rather provides canteen-prepared food for sale and/or the Nutribun and milk drinks. Teachers have developed a plan for student attendance at breakfast so that classes will not be interrupted. Recently the height and weight of all students was measured, and the school awaits assessment results. The nutrition education on balanced meals provided through NHP training was noted as especially useful to participants (parents and one school staff representative). Parents have participated in designing the breakfast menu and will serve as volunteer cooks on a rotating basis. School staff and parents consider the school nutrition component in a broad sense of student health and nutrition. The school plans to "pool community health resources" through a Health Fair in the next school term, in which community organizations would be invited to come to the school to provide health screening services (vision, hearing, dental, etc.) to visitors for a fee donated to the school breakfast program, while the students and parents would give presentations on health and nutrition topics. Parent and Community Involvement: Interviewees felt that parent and community involvement had increased, as evidenced by the successful Jamaica Day fundraiser mentioned above. It can be noted that the training provided in connection with the School Breakfast Program has served to involve more parents and provide them with skills that will be useful to the school in other areas. The guidance counselor noted that parent Study Circles had been started but had not gathered momentum. One thought for the coming school term was to revitalize these Study Circles through forming a cluster with other schools in the Parish. School #11 Date: June 13, 2002 Enrollment: 86 Region: 6 Type: Small, rural, NHP Parish: St. Catherine School Feeding Program: Phase 1 Background: This is a small rural multi-grade school with 3 teachers. Per interview with the principal and teachers, enrollment has increased 80% since the previous year, due to family outreach by the principal. The school consists of one building subdivided into 3 classrooms and an office and a separate small room with one computer. The school receives a small amount of assistance through a U.S. Baptist church. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 2 · 26 Aguirre International In this school, three classrooms were observed, and interviews were held with the school principal, both RTs and a parent. The only other member of the SDT – the School Board Chairperson – was not available during the two week period of visits by the evaluation team. School Development Plan: The principal, who arrived only a little over a year ago, and staff note that they would like to stimulate and receive greater parental involvement in developing/updating the SDP, as compared to the previous experience. At present the SDT consists of staff, the PTA President and the School Board Chairperson (a community member). Interviewees commented that the SDP is useful for establishing a vision of what they want their school to be. Staff members felt that overall the project has resulted in improved attendance and enrollment, more involvement by parents, and more collaboration among themselves. They plan together weekly. They feel that children are more involved and “like learning more.” They suggest that the project be expanded to other schools and that current NHP schools assist in training the new schools. Training: All staff note that the training provided regarding the process involved in establishing an SDP was very useful. The RTs felt that the training they received in cooperative learning and assessment were particularly useful. They particularly need training in methods for use in multi-grade classrooms. Materials: Books were readily accessible to students in all classrooms and most of these were culturally appropriate and appeared to be cared for yet well used. Teachers use the overhead projector and a variety of manipulative materials in lessons. A computer is used by small groups of students in a closet-sized space, and the principal also permits small groups of older students to use the administrative computer. The principal is very computer-literate, and the school's JSAS database is almost complete. The principal noted that NHP staff requested, welcomed and took into account his comments on potential improvements to the system in the latest system update. He feels the program is well designed and has many ideas of how the reports the system can generate could be used to stimulate parental awareness and involvement in their children's academic needs. Classroom Observations: In all three classes observed, an average of 24 to 30 children were grouped to make an effective teaching environment within the setting, with 2 children per desk. The rooms were separated by chalkboards. There was evidence of an attempt to establish a reading corner; however, there were space constraints. Children's work was prominently displayed in all three classrooms. All teachers shared their lesson plans, which were well-prepared. The lessons were linked to previous lessons. Teachers were using methods that demonstrated a shift to a better balance between teacher and learner involvement and that incorporate student experiences. Both written and oral exercises included at least one question to stimulate critical thinking. Educational materials were used in support of the focus of the lessons. Both teacher and student portfolios were reviewed and contained some original work. There was evidence of Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 2 · 27 teacher reflection and of assessment of individual and multi-grade abilities and some structuring of activities in response. School Nutrition: The school has participated in the NHP-promoted school breakfast program since Phase 1 and, due to the small enrollment, all students participate. Teachers note an improvement in students' attention span in the early morning and increased attendance. The principal has obtained regular weekly donations of a large supply of green bananas from a fruit company with agricultural land near the school. Two fundraising activities have been held this school year in support of the school breakfast program. The food preparation area showed a good standard of hygiene and the school has switched to chlorinated water that is trucked in due to an awareness of contamination in the rain catchment tank. Community Participation: The principal has noted less turnover and a greater number of participants in the school PTA as their awareness of the NHP goals has continuously increased. A teacher estimated that 95% of parents now come to PTA events. Both the participation of 15 to 20 parents in organizing two fundraising events and community response to these events was very positive. There is a smaller group of parents and community members who regularly support school activities (volunteer teach during a teacher's absence, assist in maintenance of school facilities, etc). Teachers note that some parents, but not a majority, assist students with homework. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 3 · 28 Aguirre International ANNEX 3 FOCUS GROUP REPORTS Background Focus groups were held in Regions 1 and 6 with a total of 17 representatives from seven NHP schools that had not been included in the site visits. Persons attending included 7 principals, 6 Resource Teachers, 2 School Board members, and 2 parents. Most of the participants had been in their schools for more than 10 years, with 7 reporting having more than 21 years experience in the same school. Because the viewpoints expressed were very similar in the two groups, the two focus groups are summarized together. (See Focus Group Protocols in Annex 4.) Overall Perceptions of NHP Overall impressions were very positive. Principals commented that they had been re-energized and given a support system. They also appreciated the training provided on the SDP process. The training provided to the RTs was perceived as excellent and the strategies presented seen as effective in enhancing learning. Some felt that at first they expected too much too quickly, but have learned to pace themselves and now are beginning to see results. Positive results are noted both in children’s interest and participation and on the literacy tests. “You now have learning taking place.” The project was also considered to be a boost for parents. A general strength of the project is perceived to be the focus on the child, which is resulting in improved learning. Regarding specific changes in their schools due to the project, staff members mentioned opportunities to exchange ideas and work together, and said they have gotten to know people from other schools. Teachers now plan together, are “team-building,” and feel they are better able to understand and apply the curriculum. Teachers are benefiting from the training and doing less “chalk and talk.” Lessons are more pupil-centered, and the training in cooperative learning has been especially useful in helping children learn from each other. Children show new enthusiasm for books and are interested in doing research. They are also more active and express their opinions more. Participants were pleased with the materials provided and stated that the AV equipment interested the students. The boys especially like some of the hands-on materials, such as play dough. The lack of computers was mentioned frequently as a relative weakness in the project. Other areas mentioned were the need for more diagnosis of student needs, especially at grade 1, and the need to involve parents more. It was also suggested that more emphasis should be given to mathematics and more training provided in math content. Many noted the difficulty of trying to free RTs to do the work requested of them, since substitute coverage is not available. They would like to have a spare teacher, so they can release the RTs or other teachers as needed. It is also perceived as unfortunate that the RTs receive no extra remuneration for all of their extra work. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 3 · 29 School Development Plan Some stated that when they first tried the process “it was a flop.” People had ideas but didn’t know how to go about making a plan. The training provided was very useful. The process has been very participative, and some now feel they’re using real data to make their plans, with input from teachers. The goals used to be primarily infrastructure but now focus on learning, and they list the strategies they’ll use to achieve their goals. It has been particularly useful to target a specific result in a specific time frame because everyone knows what the goal is and works toward it. Some commented that more guidance is still needed, specifically on how to develop the goals and strategies for improving literacy and numeracy and in how to help people see a “vision.” With regard to specific changes brought about by the SDP, participants stated that teachers are now more committed and more focused on achieving the goal of improving literacy. The community has bought in, and there is more parent involvement. They have noted improvement on the grade 4 literacy tests. However, changes are limited by resources available. For example, they have many children with learning difficulties and don’t have the resources to deal with them, so need more guidance on strategies to help these children learn. Training Almost all present had received training, and they considered that one of the greatest strengths of the project was the training, including the ongoing support provided by NHP staff. The parent training was very helpful, both in learning effective parenting skills and in understanding the importance of parent involvement in the school. Principals felt they benefited a great deal from their own training, particularly that on the SDP process and the JSAS. They also mentioned the benefits of sharing best practices with each other. There has been a bonding among NHP schools, and teachers exchange ideas even when they meet in out of school settings. Some have shared NHP ideas and strategies at workshops with non￾NHP colleagues. RTs also found their training extremely useful and especially appreciated the recent training in cooperative learning and the workshops on assessment. The RTs routinely provide workshops for the other teachers on training received. At first, many classroom teachers were not receptive to the RTs, but now they go to them for help. Principals mentioned that they learn from their RTs, whom they perceive as a wonderful resource. One principal commented, “The RT is like my pillar,” and others stated that no decision is taken without them in literacy and numeracy. “They have become part of our administrative staff because they guide us in our SDP.” Regarding how to improve training, it was suggested that training for new principals is needed. Training in computer technology should be offered at different levels, since not all begin with the same basic knowledge. There are often conflicts when workshops are scheduled, so schools need to be given ample advance notice. It is extremely difficult to release all of the RTs at the same time, and assistance is needed in providing coverage for them. Some mentioned that the last week of August should be reserved for school Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 3 · 30 Aguirre International preparation and not be used for workshops. As noted above, more training in math content is desired, as well as more training in diagnosis and strategies for children with special needs. It would be useful to do needs assessments regarding training needs and structure the training to meet the particular needs. More ideas are needed for involving reluctant parents, as well as for involving boys. Materials The materials were “an answer to our prayers.” The books are appreciated for their colorfulness and durability. Since they are at different levels, children who aren’t reading at grade level now have books they can enjoy reading. The computer and big screen and other AV materials are much appreciated. Some commented that they have not received materials they requested, and would like more input into selection, as well as more culturally appropriate materials. More manipulatives are needed, as well as educational software, CD-ROMs, and tapes. More materials are needed for children with learning disabilities. A uniform theme was the desire for more computers for use by students. Computers are perceived as particularly attractive to the boys, and it was stated that provision of more computer software would be an effective way to interest boys in school. Some schools have computer labs, but others feel the need for more computers and more educational software. In most cases, the computer provided by NHP is used for administrative purposes. However, many reported that the single computer is also used by teachers and students. In one school, it is used by the teachers for accessing the internet before 7:45 AM and after 3:30. In another school, the principal has little access to the computer because it is used all day by the teachers and children. Schools are trying to obtain computers, and one school reported having computers in about four classrooms. Participants all expressed a need for more computers, and ideally would have a computer lab as well as some computers in classrooms. Two of the schools have empty rooms ready to become computer labs. Some expressed appreciation for the JSAS software, although one school reported not having received the JSAS training. It was suggested that it would be useful if the office computer had a fax modem attached to it. Teaching Participants stated that because of the training received, teaching is now much more child￾centered, and children’s work is displayed. Language development is improved, and shy children are participating more. There is more incorporation of music and art. Assessment has taken on a new form, as portfolios and journals are used more frequently. Many teachers also keep portfolios now. In general, record keeping is better, since they must keep records in order to know how they’re doing on the SDP. There is a better relationship between teachers and students, and the teachers as well as the students are becoming more creative. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 3 · 31 School Nutrition One school used to have a breakfast program, but switched to a lunch program so children would stay for the afternoon. They questioned NHP’s logic in denying breakfast programs for shift schools, stating that many children come to school at noon with no breakfast. They felt a shift school could monitor a breakfast program adequately. Another school has a program that is locally funded, so they don’t have all the necessary facilities. They had many children who needed breakfast so they raised funds and started a program. They believe children on the breakfast program behave better. The school has increased scores on the grade 4 literacy test, but they don’t know if the scores are attributable specifically to the breakfast program. Another school has stated in its SDP the goal of starting a breakfast program. Parent Involvement Some schools reported excellent support from parents, including more support recently from fathers. The increased support from fathers might be engendered by the increased violence in the inner-city areas, and the desire by fathers to ensure the safety of their children. Parents help with homework and are perceived as more involved in the educational program now. However, many lamented their inability to reach and communicate with the parents they most need to see, and most would like more parental involvement. Recommendations for Improving the Project in Its Final Phase · It would be useful to involve the officers from MOEYC more, training the TEOs and Core Curriculum staff so that they would have available more of the strategies to share with other schools. More public relations would also be useful, so that the Ministry and the public would be aware of the project. The project’s successes should be publicized at the Ministry level. · It would be also be useful to have a link with the teachers’ colleges, and it was suggested that it would be useful to collect more data, doing research on the effectiveness of the project, and find out what is really working. For example, a cohort group entering first grade could be followed through its school career. · It was suggested that an incentive and official recognition be provided for the RTs. · Training in the SDP process should be continuous, providing more assistance in how to evaluate progress or non-progress. More focus and training should be provided in mathematics. · It would be useful to have a standardized diagnostic assessment, especially at grade 1, and training should be provided in diagnosis. More coordination could be provided with the preschool level regarding needs of incoming students. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 3 · 32 Aguirre International · The project should be expanded to other schools. The project should start in those schools by working with the schools and communities to identify existing needs and goals. Existing schools could help with training and implementation. Some current NHP schools could become demonstration schools. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 33 ANNEX 4 INTERVIEW PROTOCOLS Aguirre International Evaluation of USAID/Jamaica New Horizons for Primary Schools Project DISCUSSION GROUP Mixed Group such as Members of School Improvement Project Development Team (Parents, School Board Members, Community Members, Teachers, Principal) Introduction (To be used with all of the questionnaires): We are evaluating the USAID/GOJ New Horizons for Primary Schools project, and we very much appreciate your assistance in this effort. Your input will be very valuable to us. We would like to emphasize that we are not evaluating your school, your teachers, your team, or any of you personally but rather are evaluating the project itself. Our task is to learn which interventions have been most effective and to make suggestions to help improve the program in its final phase. Your comments and suggestions will help us do this job effectively. Everything you say here will be presented anonymously. (Select questions below as appropriate for group) · Could you please tell us how long you’ve been involved with this school, and in what way you’ve been involved in New Horizons Project activities? · What do you perceive as the primary goal or goals of this project? · Please describe the process used to develop the School Development Plan. · How long did it take? · What was the extent of participation by parents? teachers? school board members? community members? students? · What were the most useful aspects of the process? · If you were to engage in this process again, what would you like to see done differently? · Do you see changes in the school that you believe stem from the School Development Plan? Describe them. · Do you perceive that parents have become more involved in the school as a result of the New Horizons project activities? Why? In what way? · Do you think that community members (apart from parents) have become more interested in or involved in the school as results of the New Horizons project activities? Why? In what way? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 34 Aguirre International · (If the school has a school breakfast program) Please comment on the school breakfast program in this school: · How are children selected to participate in the program? What type of meal and type of foods are received? · Has it been well run? (If problems noted, please explain.) · Have children benefited from it? Why do you think so? · What was the situation before the breakfast program was initiated? · Do you think it has improved attendance of children? · Do you think it has affected children's abilities to learn in school? · Do you have suggestions for its improvement? · What other food or health programs are supported in this school? · Do you believe the community will be able to continue funding of the breakfast program in the future? · In what way do parents of children in this school assist in their children’s learning: Have any of these increased since the inception of the NHP project? · Volunteer in classrooms? · Meet with teachers? · Help children with their homework? · Read to their children at home? · Other? · Have you, the principal, the school board members, or teachers in your school received training from the project? If yes, are you aware of what kind of training they received? · What was the most useful aspect of this training? · What suggestions would you make for improving the training you received? · What additional training would be useful? · Have you been able to note changes in classrooms as a result of the training provided to teachers? Describe. · What changes have been easiest for teachers to make? · What changes have been most difficult? · In what way do teachers need additional support? · More in-service classes? In what areas? · More in-class support? · Other? · Do teachers in your school meet together in grade levels or in teams? · To engage in planning? · To work together on implementing changes sought? · To discuss children’s needs? · To discuss test results? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 35 · For other purposes? · Have teachers in your school engaged in peer observations? If yes, please comment. · Re: tests (ask only if appropriate for group): · Do teachers in your school have available diagnostic tests or informal diagnostic measures in each grade, to help them plan for individual children? · Do you believe that the tests used in grades 3 and 6 are testing the skills being promoted by the New Horizons project? (Example: higher level thinking skills) · Please explain the role of the resource teacher(s) in your school. · Do you see this role as useful for improving student learning? · Would you suggest changes in this role? · Have you noticed the use in classrooms or at home of materials, including audio￾visual technologies, provided by the New Horizons Project? · How were teachers trained in the use of these materials? · Have these materials been useful? · Do these materials include professional books or journals for use by parents and/or teachers? If yes, are these useful? If no, would they be useful? · Do you have suggestions regarding the materials or their use? · Have you noted changes in students’ attitudes or behavior that you attribute to this project: · Changes in attitude towards school? Describe. · Improved learning? (Reading/math) · Describe changes in students' health, alertness, or punctuality? · Other? · Has your school received a computer from the project? · How is it used? · What suggestions do you have for its future use? · What suggestions do you have for the incorporation of technology in future activities of the project? · What do you believe have been the most successful aspects of this project? · What have been the least successful? · What recommendations would you make for improving the project in its final phase? For expansion of the project to other schools? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 36 Aguirre International Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for SCHOOL PRINCIPAL · How long have you been principal of this school, and involved in New Horizons Project activities? · What do you perceive as the primary goal or goals of this project? · Please describe the process you used to develop your School Development Plan: · How long did it take? · What was the extent of participation by parents? teachers? school board members? community members? students? · Did you perceive this as a useful process? If yes, what were the most useful aspects of the process? If not, why not? · If you were to engage in this process again, what would you do differently? · Do you see changes in the school or in school-community relations that you believe stem from the School Improvement Plan? Please describe them. · Do you perceive that parents have become more involved in the school as a result of the New Horizons project activities? Why? In what way? · Do you think that community members (apart from parents) have become more interested in or involved in the school as a result of the New Horizons project activities? Why? In what way? · (If the school has a school breakfast program) Please comment on the school breakfast program in this school: · How are children selected to participate in the program? What type of meal and type of foods are received? · Has it been well run? (If problems noted, please explain.) · What was the situation before the breakfast program was initiated? · Have children benefited from it? Why do you think so? · Do you think it has affected children’s abilities to learn in school? · Do you think it has improved attendance of children? · Do you have suggestions for its improvement? · What other food or health programs are supported in this school? · Do you believe the community will be able to continue funding of the breakfast program in the future? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 37 · In what way do parents of children in this school assist in their children’s learning? Have any of these increased since the inception of the NHP project: · Volunteer in classrooms? · Meet with teachers? · Help children with their homework? · Read to their children at home? · Other? · Have you received training from the project? · Please describe the training. · What was most useful about it? · What suggestions would you make for improving the training you received? · What further training would you like to receive? · What suggestions do you have for improving this training for other principals new to the project? · Have you had the opportunity to meet with other principals to discuss the project and its activities? Would you like more such opportunities? · Do you perceive that this project has resulted in a change in your role? · More work? If so, describe. · Sharing of responsibility with others? · Changes in relationships with parents? with teachers? with students? · Other changes? · Please describe your role in supervising and evaluating teachers. Has this role changed as a result of the project? · Do you now look for anything different in classrooms/teacher performance? Please describe. · Do you feel the need of more support or assistance in this new role? · Have teachers in your school received training from the project? If yes, · Did you attend the training? · Are you aware what kind of training they received? · What was the most useful aspect of this training? · In your view, what are the most difficult aspects of the change being sought in teacher behaviors? · What additional training would you recommend for teachers in your school? · More in-service classes? In what areas? · More in-class support? Describe what is needed. · Have you received new teachers in the school since the project was started? · Have they had difficulty in learning of project goals and activities? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 38 Aguirre International · What provisions have been made for incorporating them into the project? By you? By project staff? · Do teachers in your school meet together in grade levels or in teams: · To engage in planning? · To work together on implementing changes sought? · To discuss children’s needs? · To discuss test results? · For other purposes? · Have teachers in your school engaged in peer observations? If yes, please comment. · Re: tests: · Do teachers in your school have available diagnostic tests or informal diagnostic measures in each grade, to help them plan for individual children? · Do you believe that the tests used in grades 3 and 6 are testing the skills being promoted by the New Horizons project? (Example: higher level thinking skills) · Please explain the role of the resource teacher(s) in your school. · Do you see this role as useful for improving student learning? · Would you suggest changes in this role? · Have you noticed the use in classrooms or at home of materials, including audio￾visual technologies, provided by the New Horizons Project? · How were teachers trained in the use of these materials? · Have these materials been useful? · Do you and your teachers have available a small library of professional books and journals. If yes, is it useful? If no, would it be useful? · Do you have suggestions regarding the materials or their use? · Have you noted changes in students’ attitudes or behavior that you attribute to this project: · Changes in attitude towards school? · Improved learning? (Reading/math) · Changes in students’ health, alertness, or punctuality? · Other? · Has your school received a computer from the project? · How is it used? · What suggestions do you have for its future use? · What suggestions do you have for the incorporation of technology in future activities of the project? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 39 · What do you believe have been the most successful aspects of this project? · What have been the least successful? · What recommendations would you make for improving the project in its final phase? For the expansion of the project to other schools? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 40 Aguirre International Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for RESOURCE TEACHERS · How long have you been a Resource Teacher in this school? · What do you perceive as the primary goal or goals of this project? · If you were involved, please describe the process used to develop the School Development Plan? · How long did it take? · What was the extent of participation by parents? teachers? school board members? community members? students? · What were the most useful aspects of the process? · If you were to engage in this process again, what would you like to see done differently? · Do you see changes in the school that you believe stem from the School Improvement Plan? Describe them. · Do you perceive that parents have become more involved in the school as a result of the New Horizons project activities? Why? In what way? · Do you think that community members (apart from parents) have become more interested in or involved in the school as a result of the New Horizons project activities? Why? In what way? · (If the school has a school breakfast program) Please comment on the school breakfast program in this school: · How are children selected to participate in the program? What type of meal and type of foods are received? · Has it been well run? (If problems noted, please explain.) · What was the situation before the breakfast program was initiated? · Have children benefited from it? Why do you think so? · Do you think it has affected children's abilities to learn in school? · Do you think it has improved attendance of children? · Do you have suggestions for its improvement? · What other food or health programs are supported in this school? · Do you believe the community will be able to continue funding of the breakfast program in the future? · In what way do parents of children in this school assist in their children’s learning: Have any of these increased since the inception of the NHP project? · Volunteer in classrooms? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 41 · Meet with teachers? · Help children with their homework? · Read to their children at home? · Other? · Please describe the training you’ve received from the NHP project. · What was the most useful aspect of this training? · Have you been trained in asking questions at different levels? · Are you familiar with the Menu of Instructional Strategies? If so, which of these strategies have you tried and found useful? · What suggestions would you make for improving the training you received? · What additional training would be useful? · Please describe your role in the school. Do you: · Review test results of children with teachers? · Meet with teachers to help them plan? · Meet with parents? · Work in classrooms with teachers? · Provide demonstration lessons for teachers or groups of teachers? · Work out of class with some students who need more help? · Recommend to teachers materials for special students? · Please discuss your role: · What do you find most difficult about it? Why? · Which aspects of your role do you feel are most useful? · What recommendations would you make for changes in your role? · Have you been able to note changes in classrooms as a result of the training provided to teachers? Describe. · What changes have been easiest for teachers to make? · What changes have been most difficult? · In what way do teachers need additional support? · More in-service classes? In what areas? · More in-class support? · Other? · Do teachers in your school meet together in grade levels or in teams: · To engage in planning? · To work together on implementing changes sought? · To discuss children’s needs? · To discuss test results? · For other purposes? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 42 Aguirre International · Have teachers in your school engaged in peer observations? If yes, please comment. · Re: tests: · Do teachers in your school have available diagnostic tests or informal diagnostic measures in each grade, to help them plan for individual children? · Do you believe that the tests used in grades 3 and 6 are testing the skills being promoted by the New Horizons project? (Example: higher level thinking skills) · Please comment on texts and materials, including audio-visual technologies, provided by the New Horizons Project. · How were teachers trained in the use of these materials? · Have these materials been useful? · Has the project provided a small professional library of books and journals for use by the teachers and/or parents? If yes, is this useful? If no, would such a library be useful? · Do you have suggestions regarding the materials or their use? · Has your school received a computer from the project? · How is it used? · What suggestions do you have for its future use? · What suggestions do you have for the incorporation of technology in future activities of the project? · Have you noted changes in students’ attitudes or behavior that you attribute to this project: · Changes in attitude towards school? · Improved learning? (Reading/math) · Changes in students' health, alertness, or punctuality? · Other? · What do you believe have been the most successful aspects of this project? · What have been the least successful? · What recommendations would you make for improving the project in its final phase? For expansion of the project to other schools? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 43 Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for CLASSROOM TEACHERS · Could you please tell us how long you’ve been a teacher in this school, and in what grade you teach (have taught in the past)? (Ask teachers to fill out small form at end of questionnaire) · What do you perceive as the primary goal or goals of this project? · To the best of your knowledge, please describe the process used to develop the School Development Plan. · How long did it take? · What was the extent of participation by parents? teachers? school board members? community members? students? · What were the most useful aspects of the process? · If your school were to engage in this process again, what would you like to see done differently? · Do you see changes in the school that you believe stem from the School Improvement Plan? Describe them. · Do you perceive that parents have become more involved in the school as a result of the New Horizons project activities? Why? In what way? · Do you think that community members (apart from parents) have become more interested in or involved in the school as a result of the New Horizons project activities? Why? In what way? · (If the school has a school breakfast program) Please comment on the school breakfast program in this school: · How are children selected to participate in the program? What type of meal and type of foods are received? · Has it been well run? (If problems noted, please explain.) · What was the situation before the breakfast program was initiated? · Have children benefited from it? Why do you think so? · Do you think it has affected children's abilities to learn in school? · Do you think it has improved attendance of children? · Do you have suggestions for its improvement? · What other food or health programs are supported in this school? · Do you believe the community will be able to continue funding of the breakfast program in the future? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 44 Aguirre International · In what way do the parents of children in your classroom assist in their children’s learning? Have any of these increased since the inception of the NHP project? · Volunteer in classrooms? · Meet with teachers? · Help children with their homework? · Read to their children at home? · Other? · Have you received training from the project? If yes, please describe the training. · What was the most useful aspect of this training? · What suggestions would you make for improving the training you received? · Have you been trained in asking questions at different levels? · Are you familiar with the Menu of Instructional Strategies? If so, which of these strategies have you tried and found useful? · What additional training would you like to receive? Please describe what kind of training or other support would be most useful: · In-service classes · Classroom visits · Classroom demonstrations · Exchanges of ideas with other teachers · Other · Have you changed some of your classroom practices as a result of the training you received? Describe. · What changes did you find easiest to implement? · Which changes have you found most useful? · What changes have been most difficult? · Is it your practice to make different plans for different groups of children? If so: · How do you decide what the different groups need? · How do you arrange your classroom so that different children are doing different things at the same time? · Do you meet together with other teachers in grade levels or in teams (and, if yes, how often?): · To engage in planning? · To work together on implementing changes sought? · To discuss children’s needs? · To discuss test results? · For other purposes? · Have you engaged in an exchange of peer observations with another teacher? If yes, please comment. If no, would you be interested in such exchanges? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 45 · Re: tests: · Do you use diagnostic tests or informal diagnostic measures, to help you plan for individual children? · Do you believe that the tests used in grades 3 and 6 are testing the skills being promoted by the New Horizons project? (Example: higher level thinking skills) · Please explain the role of the resource teacher(s) in your school. · Have the resource teachers helped you to improve the learning of your students? · Would you suggest changes in this role? · Please comment on the texts and materials, including audio-visual technologies, provided by the New Horizons Project? · How were teachers trained in the use of these materials? · Do you use these materials in your classroom? Have you found them useful? · Has the project made available in your school a small professional library of books or journals for use by teachers and/or parents? · Do you have suggestions regarding the materials or their use? · Have you noted changes in students’ attitudes or behavior that you attribute to this project: · Changes in attitude towards school? · Improved learning? (Reading/math) · Changes in students' health, alertness, or punctuality? · Other? · Has your school received a computer from the project? · How is it used? · What suggestions do you have for its future use? · What suggestions do you have for the incorporation of technology in future activities of the project? · What do you believe have been the most successful aspects of this project? · What have been the least successful? · What recommendations would you make for improving the project in its final phase? For expansion of the project to other schools? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 46 Aguirre International Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for PARENTS · Could you please tell us how long you’ve been involved with this school, and in what way you’ve been involved in New Horizons Project activities? · What do you perceive as the primary goal or goals of this project? · If you were involved, please describe the process used to develop the School Development Plan. · How long did it take? · What was the extent of participation by parents? teachers? school board members? community members? students? · What were the most useful aspects of the process? · If your school were to engage in this process again, what would you like to see done differently? · Do you see changes in the school that you believe stem from the School Improvement Plan? Describe them. · Do you perceive that parents have become more involved in the school as a result of the New Horizons project activities? Why? In what way? · Do you think that community members (apart from parents) have become more interested in or involved in the school as a result of the New Horizons project activities? Why? In what way? · (If the school has a school breakfast program) Please comment on the school breakfast program in this school: · How are children selected to participate in the program? What type of meal and type of foods are received? · Has it been well run? (If problems noted, please explain.) · What was the situation before the breakfast program was initiated? · Have children benefited from it? Why do you think so? · Do you think it has improved attendance of children? · What would children eat or do otherwise, if the program did not exist? · Do you have suggestions for its improvement? · What other food or health programs are supported in this school? · Do you believe the community will be able to continue funding of the breakfast program in the future? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 47 · In what way do parents in this school (you and/or others) assist in their children’s learning? Have any of these increased since the inception of the NHP project? · Volunteer in classrooms? (you/others) · Meet with teachers? (you/others) · Help children with their homework? (you/others) · Read to their children at home? (you/others) · Other? · Have you received training from the project? If yes, please describe it. · What was the most useful aspect of this training? · What suggestions would you make for improving the training you received? · What additional training would be useful? · Are you aware of training by the project that has been received by personnel in this school or community? If yes, please comment: · Training provided to teachers · Training provided to resource room teacher · Training provided to principal · Training provided to school board members · Other · Have you been able to note changes in classrooms as a result of the training provided to teachers? Please describe. Do you believe further training is needed? (In what areas and of what types?) · How often do you meet with teachers regarding your child’s progress? · Have you noted changes in students’ attitudes or behavior that you attribute to this project: · Changes in attitude towards school? · Improved learning? (Reading/math) · Other? · Are you aware of the role of the resource teacher(s) in your school? If yes: · Do you see this role as useful for improving student learning? · Would you suggest changes in this role? · Have you noticed the use in classrooms or at home of books or materials provided by the New Horizons Project? · Have these materials been useful? · Has the project made available any books or journals for use by parents? · Do you have suggestions regarding the materials or their use? · Has your school received a computer from the project? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 48 Aguirre International · How is it used? · What suggestions do you have for its future use? · What suggestions do you have for the incorporation of technology in future activities of the project? · What do you believe have been the most successful aspects of this project? · What have been the least successful? · What recommendations would you make for improving the project in its final phase? For expansion of the project to other schools? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 49 Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for STUDENTS · Have you heard of the New Horizons Project? Please tell us what you know about it. · (In the case of older students) Do you think that your school has changed since you first started going here? If so, how? How do you feel about these changes? · Do you have books and materials in your classrooms that you think are interesting? Do you sometimes take them home? · Do you like to read? If so, what kind of books or magazines do you like to read? Do you have enough good books or magazines in your school? · Do you like maths? Why or why not? · Does your teacher show films or use other audio-visual materials? What do you learn from these? · If you don’t understand something or need help with your school work, what do you do? What does your teacher do? (Is it easy for you to ask for help?) · Does anyone else help you when you don’t understand something? (other students, resource room teacher, parents) · Do your mother or father (or someone else from your family) help out at school? (doing what? how often?) · At home, do you discuss your schoolwork with your parents or other members of your family? Do they help you with your homework? Does someone at home read to you? · Do you sometimes work in groups with other students? Do you like that? · (Adapt to the terminology of each site) Do you participate in your school's feeding program? (Breakfast, Nutribun, etc.) Do all of your friends at school participate too? Why or why not? Do you like the food? Do you have any suggestions? What would you eat or do if the program didn't exist? · What do you think is the best thing about your school? · What do you wish you could change about your school? · Is there anything else you’d like us to know about your school? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 50 Aguirre International Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Classroom Observation Checklist School: Grade: Teacher (code): Resource Teacher: (yes/no) Date of observation: Observer: Subject observed: Number of Students A. Classroom setting Configuration ’Children grouped to make the most effective use of physical setting ’Children grouped to make effective use of physical setting ’Little evidence of grouping to make effective use of classroom setting ’All rows facing teacher’s table when more creative possibilities exist Partition ’No partition evident between classes ’Classroom separated by chalkboard ’Classrooms separated by wooden partition ’Classrooms separated by concrete walls Seating ’Four children per seat ’Three children per seat ’Two children per seat ’Children seated on individual chair/desk combination Centers of interest (corners) ’Centers of interest are effectively integrated (presented in themes rather than subjects) ’Reading center of interest evident ’Mathematics center of interest evident ’No corners or centers of interest evident Books ’Seem well used by children and placed in an area for ready student access ’Seem somewhat used by children and placed in an area for ready student access ’Locked away in cupboards with evidence of some use ’Appear to be in pristine (unused) condition Display of children’s work ’Children’s work prominently and purposefully displayed in classroom ’Children’s work prominently displayed in classroom ’Little evidence of children’s work displayed in classroom Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 51 ’No evidence of children’s work displayed in classroom B. Teaching ’Evidence of planning and chronological coherence of lesson (e.g., evidence of present and future linkages, use of students’ knowledge, etc.) ’Lesson well planned with strong evidence of linkages to previous lessons ’Lesson planned and there is evidence of linkages to previous lessons ’Lesson poorly planned with little evidence of linkages to previous lessons ’No evidence of planning and no evidence of linkages to previous lessons Comments: Learner involvement ’Teacher uses methods that predominantly involve learners ’Teacher uses methods that show balance between teacher and learner involvement ’Teacher uses methods that demonstrate little learner involvement ’Teacher uses methods that do not involve learners Comments: Use of Instructional Strategies encouraged by the Project ’Strong evidence of contextually appropriate use of NHP strategies ’Moderate evidence of contextually appropriate use of NHP strategies ’Little evidence of contextually appropriate use of NHP strategies ’No evidence of contextually appropriate use of NHP strategies Comments: Use of resources ’Highly effective use of texts and supplementary materials ’Effective use of text and supplementary materials ’Minimally effective use of text and supplementary materials ’Ineffective/no use of text and supplementary materials Comments: Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 52 Aguirre International Use of technological aids (e.g., computer, television, VCR) ’Highly effective use of technological aids ’Effective use of technological aids ’Minimally effective use of technological aids ’Ineffective/no use of technological aids Comments: Questioning Techniques ’Students are challenged to use judgment, inferences and recall in their responses ’Students are challenged to use inferences and recall in their responses ’Students are challenged to use only recall in their responses ’Students are not challenged to use any of the thinking skills in their responses Comments: Use of learners’ experiences ’Learners’ experiences are highly regarded and accommodated in instruction ’Learners’ experiences are regarded and accommodated in instruction ’Learners’ experiences are marginally regarded and accommodated in instruction ’Learners’ experiences are neither regarded nor accommodated in instruction Comments: Accommodation of learners’ questions ’Learners are encouraged to ask questions ’Learners’ questions are accommodated in instruction ’Learners are rarely allowed to ask questions ’Learners are not allowed to ask questions Comments: Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 53 Teacher response to student errors ’Teacher responds to student error in positive fashion (requesting more information, telling child what question he did answer, etc.) ’Teacher indicates answer is incorrect, but gives child another chance. ’Teacher calls on another child. ’Teacher laughs or makes negative comment about answer. Comments: Assessment Strategies ’Highly appropriate and incorporate traditional & alternative modes ’Appropriate and incorporate traditional & alternative modes ’Marginally appropriate and seldom incorporate traditional & alternative modes ’Inappropriate and do not incorporate traditional & alternative modes Comments: Evidence of use of formative assessment to customize instruction for learners ’Strong evidence of use of formative assessment ’Average evidence of use of formative assessment ’Little evidence of use of formative assessment ’No evidence of use of formative assessment Comments: Use of classroom time (entire lesson) ’Teacher is punctual and all students are constantly engaged in meaningful activities ’Teacher is punctual and students are largely engaged in meaningful activities ’Teacher does not begin class on time and students are barely engaged in meaningful activities ’Teacher is not punctual and students are not engaged in meaningful activities Comments: Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 54 Aguirre International Teacher Reflection ’Manifested in lesson notes, portfolio and interaction with observer ’Manifested in lesson notes and interaction with observer ’Manifested only in interaction with observer ’Not manifested at all Comments: Projects ’Ample evidence of classroom projects that promote student learning ’Adequate evidence of classroom projects that promote student learning ’Little evidence of classroom projects that promote student learning ’No evidence of classroom projects that promote student learning Comments: Teacher modeling (literacy) ’There is strong evidence that teacher frequently reads to the children ’There is evidence that teacher frequently reads to the children ’There is little evidence that teacher frequently reads to the children ’There is no evidence that teacher frequently reads to the children Comments: Student writing ’There is abundant evidence of original student writing ’There is evidence of original student writing ’There is little evidence of original student writing ’There is no evidence of original student writing Comments: Additional observations/notes Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 55 Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 56 Aguirre International Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Materials Checklist School: Grade: Teacher (code): Resource Teacher: (yes/no) Date of observation: Observer: Audio Visual/ Technological equipment in school: (tick the appropriate boxes) ’ Television ’ VCR ’ Camera ’ Photocopier ’ Computer ’ Printer ’ Tape Recorder ’ CD Player ’ Camcorder ’ Overhead Projector ’ Other (specify) Overall Impression ’The materials are highly appropriate/motivating for learners - in terms of age, interests ’The materials are appropriate/motivating for learners - in terms of age, interests ’The materials are minimally appropriate/motivating for learners - in terms of age, interests ’The materials are not appropriate/motivating for learners - in terms of age, interests Comments: Organization ’Materials are creatively selected and organized to facilitate learning ’Materials are selected and organized to facilitate learning ’Some materials are selected and organized to facilitate learning ’Materials are not selected and organized to facilitate learning Comments: Cultural relevance ’Photos and other illustrations have strong cultural relevance ’Photos and other illustrations are culturally relevant ’There is little evidence of cultural relevance in photos and illustrations ’There is no evidence of cultural relevance in photos and illustrations Comments: Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 57 Innovation ’Teacher & student created materials are highly innovative ’Teacher & student created materials are innovative ’Teacher & student created materials are minimally innovative ’Teacher & student created materials are not innovative Comments: Avenues for learning ’The materials present generate maximum learning opportunities ’The materials present generate learning opportunities ’The materials present generate little learning opportunities ’The materials present generate no learning opportunities Comments: Avenues for student reflection ’Materials provide abundant opportunities for student reflection on their learning ’The materials provide opportunities for student reflection on their learning ’The materials provide few opportunities for student reflection on their learning ’The materials provide no opportunities for student reflection on their learning Comments: Evidence of integration ’Materials strongly highlight cross curricular links ’Materials highlight cross curricular links ’Materials minimally highlight cross curricular links ’Materials do not highlight cross curricular links Comments: Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 58 Aguirre International Assessment materials ’Highly effective/linked to the National Curriculum ’Effective/linked to the National Curriculum ’Barely effective/linked to the National Curriculum ’Not effective/linked to the National Curriculum Comments: Assessment materials ’Strongly cater to differences in learners’ performance/achievement levels ’Cater to differences in learners’ performance/achievement levels ’Barely cater to differences in learners’ performance/achievement levels ’Does not cater to differences in learners’ performance/achievement levels Comments: Use of Technological Aids ’Strong evidence of effective use of technological aids in materials production and instruction ’Evidence of effective use of technological aids in materials production and instruction ’Little evidence of effective use of technological aids in materials production and instruction ’No evidence of effective use of technological aids in materials production and instruction Comments: Additional Comments: Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 59 Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project TEACHER TRAINING Questionnaire: For use about teacher training issues, in Ministry or in a teacher training program, or with others knowledgeable about teacher training issues: · What is the typical undergraduate training received by Jamaican teachers? · Number of years · Style of training (participative/traditional, etc.) · Content in reading and mathematics courses · What is the percentage of teachers without certification? · In general · In NHP schools · How much schooling have such teachers typically had? · What in-service training has been provided in recent years to teachers in general, apart from that provided for NHP teachers? · Who provided it? · In what areas? · What style of training? · Is in-class follow-up and support typically provided? · What is generally perceived to be the greatest need of current Jamaican teachers in regard to desired changes in their classroom practice? How do these desired changes differ from those sought by the NHP project? · Has in-service training been provided to teachers in general on interpretation and use of the available tests? · What is the role of the principal in Jamaica in providing supervision and pedagogical support to classroom teachers? · What special training and/or certification is necessary to become a principal in Jamaica? · Who, besides the principal, is responsible for providing pedagogical supervision and support for classroom teachers? (District supervisors or inspectors?) · What types of school health and/or feeding programs exist in primary schools? What has been the experience-to-date with these programs? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 60 Aguirre International · With specific reference to training provided by New Horizons project, procure outline and syllabi for specific training activities provided: · For principals · For resource teachers · For classroom teachers · For School Board and community members · For parents · Has training for classroom teachers included use of the new materials? · How different is training provided to NHP teachers from that provided for other teachers? · In content? · In classroom approaches suggested? · In use of materials? · In use of diagnostic measure to diversify instruction? · Has training provided to principals included ways for them to provide pedagogical leadership, supervision and support for teachers in the changes sought by NHP? · What is the role suggested for the Resource Teacher in the provision of in-service training or pedagogical support to teachers? · Have teachers received sufficient in-class support and follow-up over a period of sufficient years to help them implement the desired changes? · What have been the strengths/weaknesses of training provided to date? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 61 Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project FOCUS GROUP Interview Sessions Date: Time: Venue: Schools Represented and Names of Representatives: School Principal MRT LRT Board Member Parent · How long have you been associated with your school? a. 21- 30 years b. 11 - 20 years c. 6-10 years d. 1-5 years · Please give us your impressions of the New Horizons Project. · What changes in your school would you credit to the New Horizons Project? · Why? – How has NHP affected children in your school? · Share with us your experiences of making a school development plan. What were the drawbacks in making this plan? What were the benefits? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 62 Aguirre International · Describe the changes in the school that you believe have resulted from making a school development plan. · Tell us about the training that you or other members of your school community have received from NHP (discuss usefulness and suggestions for improvement). · Describe the changes you have seen in classrooms as a result of the training provided to teachers. · Give your impressions on the role of the resource teacher in your school. · What are your impressions of the materials (books and A/V materials) provided by the NHP? · What suggestions would you make regarding these materials and their use? · How is the computer provided by New Horizons used in your school? · How would you like it to be used? · Tell us about the ways in which parents of your school contribute to their child’s learning. · Does your school have a breakfast program? · Please tell us how this program operates in your school. · What do you believe have been the most successful aspects of this project? · What have been the least successful? · What recommendations would you make for improving the project in its final phase? · For expansion of the project to other schools? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 63 Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for JUAREZ AND ASSOCIATES · Could you please summarize the Juarez contract, e.g., goals, level of effort, and principal implementation and evaluation mechanisms? · How would you rate the NHP project’s overall progress to date, in relation to its objectives? · Which components of the project have been particularly successful and which have encountered the most difficulties? · Has NHP generally operated in a cost-effective manner? Are there specific interventions that have been especially cost-effective? Are there others that raise concerns that the cost may not justify the results? · The results in terms of improved student performance shown in the formative evaluation are surprising. What do you think underlies those results? I understand that you are completing the 2002 formative evaluation. Will there be significant changes in the trends? Might it be possible for us to see an advance copy of that report? · In retrospect, what changes in project design might have produced superior results? · How would you rate the Ministry of Education, Youth, and Culture’s support to the project in policy and practical terms? · How would you respond to the same question, substituting USAID for MOEYC? · How well have the planned evaluation mechanisms worked? If there have been problems, were they due to the evaluation design itself or weaknesses in implementation capacity or both? · In your opinion, are there changes that need to be made in NHP that could be implemented relatively easily and at low cost between now and the expiration of the project? · What, in your view, are the principal lessons learned to date from NHP? · Do you have any suggestions for the evaluation team and USAID as to what should follow the current project? · Are there particular people or organizations that it is important for us to see? · Anything else you would care to ask or any questions for us? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 64 Aguirre International Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for Ministry of Education, Youth, and Culture (MOEYC) · What are the Ministry’s strategic goals and priorities for the public education sector over the next several years? Where does strengthening basic education, especially primary education, fit in the overall strategy? What is NHP’s role in implementing the strategy? · Could you describe the Inter-American Development Bank-assisted primary education project? What are its size, scope, and principal goals? What interventions are involved, and how does the emphasis given to those interventions compare with NHP? How effective has it been to date, compared with NHP? How is it being evaluated? · What is your overall assessment of the results of the NHP project to date? Has it met your expectations? · NHP’s emphasis has been on strengthening numeracy, literacy, related teacher training and teaching, school management, attendance, and parental and community involvement in the schools. In which of these areas has NHP had the greatest success? The least? In your view, what are the main factors underlying either successful or less-successful performance? Do the evaluation and measurement tools being used in the project permit one to adequately answer these questions? · In retrospect, do you feel that changes in the way NHP was designed and implemented would have materially improved results? What are they? · Currently, do you feel that changes are needed in either the substance or structure of NHP? What changes? · Has NHP been cost-effective, that is, are the results it has achieved commensurate with the costs? Are there project components that have been particularly cost￾effective or particularly cost-ineffective? · Looking forward, do you think NHP should be expanded to cover more, if not all, of the schools with needs similar to those included in NHP? If so, what do you see as the greatest constraints? Public support? Money? Teachers? Ministry and school staff? Administrative capacity? Other? · Is it realistic to expect Jamaica, at both national and local levels, to assume a greater share of the costs of NHP, especially if it is expanded? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 65 · Has USAID assistance to NHP been well-targeted and utilized? In retrospect, are there ways that it might have been structured and implemented that would have made it more effective? · Has the USAID contractor for NHP, Juarez and Associates, performed up to your expectations? Are there areas in which you think their support for the project could have been more effective? Are there changes you would like to see made at the present time? · Is the NHP Project Implementation Unit (PIU) meeting its objectives? Is it well￾integrated into the Ministry structure and is working within Ministry guidelines? · Are Ministry staff, teachers, and local leaders involved in the project receiving the training and experience anticipated in the project design? If all the financial and material resources for expansion and replication were available, does Jamaica have the human resources to get the job done without outside help? · Is there adequate coordination between NHP and other, related Ministry projects, including other USAID or other donor-assisted activities? · In general, is the Ministry satisfied with current arrangements for coordination between it and the donor community, as well as coordination among donors? How might coordination be improved? · Are there any other comments or suggestions that you would care to make regarding the NHP, the Aguirre evaluation team’s work, or on any other topic? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 66 Aguirre International Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for DONORS: · What education activities are you currently supporting in Jamaica? How do they fit within your overall development program in the country (i.e., what is the underlying strategy)? Are any changes contemplated? · In general, are you satisfied with the results? Have you run into any significant problems and if so, what are they? · How do you evaluate your projects? Are there built-in evaluation systems? If so, are there evaluation reports, analyses, and data that could be shared with us? · What is the size of your annual education budget for Jamaica and what is the current total of your commitments to education? What education staff or contractors do you currently have in the country? · Who are the principal counterparts for your education work? If the Ministry of Education, Youth, and Culture, which MOEYC units do you work with most closely? · How would you characterize your relationship with MOEYC? What do you see as its strengths and weaknesses? Do any of your education activities involve assisting the Ministry to develop its capacity, as well as direct support to projects? · Are you currently cooperating with USAID or other donors on any of your education projects? If so, who are they and which are the projects? · How would you describe the current state of cooperation, coordination, and information sharing among the donor community in Jamaica? Does it need strengthening? If so, what are some of the things that might be done? · Are you familiar with the New Horizons for Primary Schools (NHP) project? Do you see what NHP is doing as closely related, somewhat related, or not related to your education work? Do you have any views on how the project is doing, or any suggestions for the evaluation team as to how it might be strengthened? · What do you see as the greatest unfilled needs of Jamaican public education? If your organization should receive a significant increase in its education budget, where and how would you invest it? · Do you have any other comments or suggestions? Any questions regarding the NHP evaluation? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 4 · 67 Aguirre International Evaluation of USAID Jamaica New Horizons for Primary Schools Project Questions for AID/USAID · Where and how well does NHP fit into the Bureau and the Caribbean Regional Program’s education portfolio? Is it viewed as a regional, as well as Jamaican model? Are there any significant differences in the Bureau’s and the Mission’s perspectives on this program? · How do you feel, generally, about the project’s progress to date, in relation to its goals? Do you currently have concerns that you would like to bring to the evaluation team’s attention? · In your opinion, how well has the project’s design worked out? In retrospect, are there things that you think might have been done differently? Are you satisfied with the performance of MOEYC? · The improvements in student performance as shown in the 2001 formative evaluation were disappointing. What do you think explains those results? What has been the reaction of USAID, the contractor and the Jamaican government to those results? · From your perspective, how well have the planned evaluation mechanisms worked? If there have been problems, are they due to the evaluation design, weaknesses in implementation, or a combination of factors? · Are you comfortable with project cost performance to date? Are there components that you feel have been particularly cost-effective or cost-ineffective? · Has the project met its goals of supporting the objectives of other Mission SOs and other IRs in SO 4 and vice versa? · In your view, what are the principal lessons that have been learned so far from the NHP experience? · Are there changes in the current project that would be relatively easy and low cost to implement that you feel should be made before its planned expiration? · In your opinion, what should follow NHP? Should it be extended and, if so, with what modifications? Are there other ways to address Jamaican basic education priorities, building on the NHP experience? · What have been the sources of funding for NHP and other Jamaica education program components? What are the prospects for future funding of Mission education initiatives? Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 4 · 68 Aguirre International · Are there other donor programs that focus on the quality of primary education (sponsored by the government, other donors or USAID) that would be important for the evaluation team to know about? · Do you have recommendations for us about organizations or people to whom we should speak? Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 5 · 69 ANNEX 5 HEALTH AND NUTRITION SUMMARY Research has shown a relation between the nutritional status of school-age children and educational performance. Of most importance, perhaps, is the link demonstrated between cognition and fasting hunger and its relevance for student learning ability during morning classes when a nutritious breakfast has not been eaten (Pollitt 1998). Various studies demonstrate a relation between anemia in school-age children and decreased learning. Infection with intestinal worms is a common cause of child anemia and can be treated inexpensively (Guyatt 2001). Based on these potential linkages, the NHP proposal initially included an intervention which called for the provision of breakfast for anemic children, with a bi-annual de-worming activity in 25 of the 72 NHP schools. In preliminary meetings between the NHP nutritionist, Ministry of Health officials, and representatives from the University of the West Indies, it was stated that worm infestation was not a public health problem or priority (NHP 1999). In response, NHP re-designed and reduced the scope of the health and nutrition intervention. Eight schools were selected, based on defined criteria which included low attendance (below 75%), poor academic performance (NAP scores below 35), and local support for initiating the program. Participating students are selected by school staff, based on knowledge of the student's situation. All eight schools which initiated the breakfast program in September of 1999 have continued and are presently focused on fundraising to sustain the program after project funding ends. The school breakfast program has become a key vehicle for community involvement. Parents have received training in organizational strengthening, community development, management, finance and fundraising. Starting in 2001-2002, the health intervention was expanded to include other components – parent education in health and nutrition, teacher training in health and nutrition, and community mobilization of health resources. Six additional schools have been selected to initiate the intervention for another 470 needy students in Phase 2, September 2002. PTA sub-committees have received training and have actively initiated fundraising efforts. Results of school attendance by students participating in the breakfast program show an increase in the average attendance from 68% to 73%. Comparing participating students to non-participating students in Grade 3, Grade 4 and Grade 6 tests showed similar results for both groups, even though participants were selected as being disadvantaged. The program is currently being started in six additional schools. It is recommended that NHP continue the school health and nutrition intervention through the life of the project, so that the impact and potential for sustainability of the program can be fully assessed. If results are Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 5 · 70 Aguirre International favorable, a model will have been created for replication in future primary education projects. Background Research has shown a relation between the nutritional status of school-age children and educational performance. Of most importance, perhaps, is the link demonstrated between cognition and fasting hunger and its relevance for student learning ability during morning classes when a nutritious breakfast has not been eaten (Pollitt 1998). A study of school breakfast programs for rural Jamaican schoolchildren showed increases in verbal fluency for those children classified as undernourished (Chandler 1995) and improvements in classroom behaviors, such as attention to task, if the setting was well￾organized and structured (Chang 1996). A study conducted among children in Grades 2 to 5 in rural Jamaica found a school breakfast to provide benefits in children's nutritional status, attendance and achievement; however, it was noted that "the massive problem of low achievement levels requires more than nutritional support alone" (Powell 1998). Various studies have also demonstrated a relation between anemia in school-age children and decreased learning. Infection with intestinal worms, especially hookworm (Ancylostoma duodenale) and/or Schistosoma spp., is a common cause of child anemia and can be treated inexpensively (Guyatt 2001). Based on the potential linkages between nutrition and educational performance, the NHP proposal initially included an intervention with the purpose to “treat the conditions of malnutrition, parasitic infection and iron deficiency anemia that negatively affect the learning achievement and school attendance among some pupils in primary schools” (NHP 1999). It called for the provision of breakfast for anemic children, with a biannual de-worming activity in 25 of 72 NHP schools. Several factors led to a re-design of this intervention. The original proposal recognized that parasitic infection with worms and iron-deficiency anemia was not widespread in Jamaica. In preliminary meetings between the NHP nutritionist, Ministry of Health officials, and representatives from the University of the West Indies, it was stated that worm infestation was not a public health problem in Jamaica and therefore not considered a priority (NHP 1999). In terms of health activities for school-age children, the Ministry of Health visits all schools at the start of the school year to update student immunizations. Parents obtain health care for school-age children through the public or private health system. Occasional activities are coordinated on a one-by-one basis between schools and organizations for screening of vision, hearing or dental exams. At present, the Ministry of Health and MOEYC are taking a close look at possible interventions for child health in the pre-school centers attended by a high percentage of the early childhood population. In this way, child growth retardation can be addressed at an earlier and potentially more reversible stage. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 5 · 71 It should also be noted that MOEYC supports school feeding through the school lunch program, which remains fully viable in all primary schools four years after support from the World Food Program ended on March 31, 1998. However, one study of schoolchildren's lunch habits in Jamaica found that energy intakes were below optimal level (Walker 1998). Although health education has always been a part of schooling, revision of the primary school curriculum has resulted in a much stronger emphasis on health and nutrition education within the core subjects and/or integrated studies. NHP re-designed the health and nutrition intervention plan, in response to input from the various ministries involved, and decided to reduce the scope of the nutrition intervention, partly in response to reduced funding. A needs assessment was conducted, which included a set of criteria to select schools with a profile of low attendance (below 75%) and poor academic performance (NAP scores below 35), along with a demand-driven request for assistance in establishing a sustainable school breakfast program. This resulted in the selection of eight schools in seven Parishes in all six regions of Jamaica; three of these were classified as urban, four as rural, and one as remote rural. Implementation status All eight schools which initiated the breakfast program in September of 1999 have continued and are presently focused on fundraising to sustain the program in future years. Participating students have been subjectively selected by teachers and guidance counselors, based on their level of alertness or concentration during early classes, their attendance, mention of morning hunger and/or knowledge of their home socio-economic situation. The percentage of participating students, a total of 468 in eight schools, varies from almost 100% of the student population in small rural schools to less than 15% in large urban schools. About half of participants receive the breakfast for free, while half pay a sliding-scale fee from J$10-15/meal. Additional students may participate if they pay the full cost for breakfast, J$20/meal/student (US$0.43). Parents rotate as volunteer cooks to prepare the breakfast, which is a balanced hot meal, such as beef in a vegetable sauce with green bananas. Parents helped design the five-day menu. The school breakfast program has become a key vehicle for involving parents in the schools. Health and nutrition sub-committees have been formed through the PTAs. Parents have received training in community development, organizational strengthening, planning and management, finance, fundraising and income generation. Manuals on many of these topics will be provided through the Jamaica Social Investment Fund (JSIF). At the beginning of the school year 2001-2002, the health intervention was expanded to include other components – parent education in health and nutrition, teacher training in health and nutrition, and community mobilization of health resources. A two-day workshop based on materials adapted from JSIF was held with parent groups from four of the Phase 1 schools and four of the Phase 2 schools. A manual, Food and Nutrition Concepts for the Primary Curriculum, has been prepared by the NHP nutritionist for use by teachers. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 5 · 72 Aguirre International The emphasis on attaining sustainability of the breakfast program through community and parental support is shown in the design of the program, in which financial support is reduced to 50% in Year 2, 25% in Year 3, and 0% in Year 4. However, as NHP funding reduction delayed the training of parent groups in organizational capacity and financial management, the schools selected in Phase 1 will receive 100% of funding for the 2002-2003 school term. Beyond June 2003, funding for Phase 1 and Phase 2 schools would be steadily reduced. Quantitative and qualitative evaluation Any monitoring and evaluation plan to assess the impact of a school breakfast program on student achievement faces obstacles as varied as depending upon breakfast records maintained by volunteers with low literacy skills to the difficulty inherent in measuring improvements in cognition and alertness. The NHP nutritionist has collected data on student attendance, alertness (as rated by teachers), assessment scores and height and weight-for-age. There have been difficulties in obtaining complete monitoring information and/or information by individual student. However, results of school attendance by students participating in the breakfast program show an increase in the average attendance from 68% to 73%. Comparing participating students to non-participating students in Grade 3, Grade 4 and Grade 6, tests showed similar results for both groups, even though participants were selected as being disadvantaged. Assessment of student Body Mass Index has shown improvement for the small percentage of students that are significantly malnourished, decreasing from 7.3% in 2000-2001 to 2.7% in 2002. Per monitoring results, teachers also rate students benefiting from the breakfast program as demonstrating improved concentration, alertness and an increase in class participation. As all participating schools have also received the full package of other NHP interventions, it is difficult to determine what percentage of improvement can be attributed to the breakfast program alone. In evaluation interviews with school staff and parents, all interviewed perceive the breakfast program as necessary. They cite poverty and/or disintegration of the family structure as leading to a portion of their students not receiving a nutritious breakfast in the home. Teachers describe some students as lethargic in the morning previous to the breakfast program and alert since its inception. Principals also see the program as an incentive for parents to enroll students in the school. Increased student attendance was also mentioned, with a more punctual arrival in the morning specifically noted. Teachers interviewed do not consider the breakfast program to detract in any way from classroom learning. Schools have either stipulated that the breakfast program only operate prior to the beginning of the school day or, in larger schools, teachers have developed a breakfast shift schedule for students. All participants interviewed are very clear that the program is intended to become self￾sustaining. Parent groups, with the support of at least one teacher or guidance counselor, are implementing plans to generate funds on a regular basis. Several urban schools describe the local fundraising environment as highly competitive and have found it difficult to raise Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 5 · 73 significant amounts of funds. It should be noted, however, that Cavaliers All-Age School, an urban school, raised J$28,000 (approximately US$610) in one event. Rural schools do not mention fundraising as competitive. Bull Bay All-Age School recently raised approximately J$17,000 (US$370) during “Jamaica Day” and has plans for a future “Health Day” in which community organizations will provide free screening services (vision, hearing, dental) to adults who pay an entrance fee in support of the breakfast program. At present, PTA parent sub-committees in both urban and rural schools are trying to establish a Past Students Group as a potential resource for support of the breakfast program. In addition to sustaining the purchase of food commodities, some schools have determined that their breakfast program cannot rely upon volunteer parents. These schools will test the sustainability of providing a small financial stipend to a cook and requesting that parents rotate as volunteers to assist the cook. Recommendations for NHP extension Extend the period of NHP support for the breakfast program for Phase 1 and Phase 2 schools and complete the planned reduction in funding support to assess sustainability. As the nutrition component of the project was delayed and reduced in scope, it is suggested that NHP be able to carry-out to its conclusion the plan for assessing the self-sustainability of a school breakfast program. NHP should not support any additional schools in initiating a breakfast program until a clear model for sustainability is developed. Rather, the focus should be on assisting schools and parent groups in their acquisition of fundraising and program management skills. To date, the activities involved in organizing and supporting the school breakfast program have provided a concrete mechanism for stimulating parental involvement in the targeted primary schools, and their usefulness as a vehicle for increasing parenting skills should be further explored. Review the focus of the monitoring and evaluation elements of the school breakfast program. As is recognized by NHP staff, schools have not been fully compliant in monitoring the breakfast program. A look at streamlining the monitoring reports to the minimum amount of information necessary would be useful at this point. Collection of information obtainable through the JSAS should also be considered as a means to eliminate any potential redundancy of reporting. For example, student enrollment in the breakfast program may be sufficient for tracking their attendance and academic achievement through the JSAS over time. Separately, reporting on the cost and total number of breakfasts provided might be sufficient for monitoring the program financially. NHP should continue to survey teachers to evaluate changes in student alertness. A minimum number of surveys, a restricted time period (three weeks has been noted by teachers as the minimum period in which a change is demonstrated), and/or limiting follow￾up to a sub-group of students could reduce the additional effort requested of teachers. It is doubtful that monitoring weight and height of students will provide useful information beyond that for the small percentage of students seen to be low weight-for-height (<7% in Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 5 · 74 Aguirre International 2000-2001); however, it can be a useful exercise to promote student health in and of itself. It would be useful for NHP to try to find a less time-intensive variable that would strongly correlate with the Body Mass Index in identifying students with the highest need. The health and nutrition monitoring activities should contribute to the development of non￾subjective criteria for selecting student participants in the breakfast program. A simple wealth ranking scale has been considered as a socio-economic indicator of poverty. Other criteria for selection of students, such as a single parent family, could also be developed by school staff and community representatives. At present, selection is subjective, and all subjective judgment is subject to bias. Recommendations for a follow-on program Replicate a model proven to be sustainable of the school breakfast component of the health and nutrition intervention. If a sustainable model is developed that has been proven over at least a two year period, the school breakfast component should then be multiplied further with project support. Food assistance programs often create a high degree of dependency and demand among populations served. It is recommended that replication proceed cautiously. Involve professionals from the University of the West Indies and/or the Caribbean Food and Nutrition Institute in a controlled evaluation study of the breakfast program. The literature still lacks sufficient controlled studies that substantiate the linkages between school feeding programs and improved academic performance by students. Professional researchers from the University of the West Indies have been involved in several seminal studies, and their expertise could provide further contributions in this area. Expand efforts to assist teachers in incorporating health and nutrition education within the revised curriculum. School is an important source of information on health and nutrition that can influence student choices throughout life. As for any academic topic, specific teaching strategies can be employed to increase student absorption and retention of key messages. Any project in support of improved educational strategies should further explore ways in which nutrition and health education messages can be incorporated into efforts to increase student literacy and numeracy, in support of the revised curriculum. Bibliography Chandler 1995. School breakfast improves verbal fluency in undernourished Jamaican children, by A.M. Chandler, et al.; Journal of Nutrition, 1995. Chang 1996. The effects of breakfast on classroom behaviors in rural Jamaican schoolchildren, by S.M. Chang, S.P. Walker, J. Himes and S. Grantham-McGregor; Food and Nutrition Bulletin, 17: 248-257. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 5 · 75 Del Rosso 1999. School Feeding Programs: Improving effectiveness and increasing the benefit to education. A Guide for Program Managers, by Joy Miller Del Rosso for the Partnership for Child Development; June 1999. Grantham-McGregor 1998. Evaluation of school feeding programs: some Jamaican examples, by S.M. Grantham-McGregor, S. Chang and S.P. Walker; American Journal of Clinical Nutrition, 67(4): 785S-789S. Guyatt 2001. Evaluation of Efficacy of School-based Anthelmintic Treatments against Anaemia in Children in the United Republic of Tanzania, by H.L. Guyatt, S. Brooker, C.M. Kihamia, A. Hall and D.A.P. Bundy; Bulletin of the World Health Organization 2001, 79(8): 695-703. NHP 1996. New Horizons for Primary Schools Project Paper, Juarez and Associates, September 1997. NHP 1999. New Horizons for Primary Schools, Health and Nutrition Intervention #5: School Selection and Implementation Plan, prepared by Patricia Thompson for Juarez and Associates, June 1999. Pollitt 1998. Breakfast and cognition: an integrative summary, by E. Pollitt and R. Matthews; American Journal of Clinical Nutrition 1998, 67(S):804S-813S. Powell 1998. Nutrition and education: a randomized trial of the effects of breakfast in rural primary school children, by C.A. Powell, et al.; American Journal of Clinical Nutrition 1998, 68(4): 873-879. Walker 1998. Schoolchildren's diets and participation in school feeding programmes in Jamaica, by S.P. Walker, C.A. Powell, et al.; Public Health Nutrition 1998, 1(1): 43-49. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 6 · 76 Aguirre International ANNEX 6 EDUCATION MANAGEMENT INFORMATION SYSTEM (EMIS) Executive Summary Building upon previous assistance projects, the New Horizons for Primary Schools Project (NHP) project proposed interventions to assist schools to develop a capacity for using monitoring information systems to improve the quality of education. NHP also proposed to assist MOEYC in consolidating various databases, linking these to other sources of information (such as the census), and establishing a network between regional education offices and MOEYC. All 72 NHP schools are using a license-free and royalty-free software package, the Jamaica School Administrative System (JSAS), developed through technical assistance performed by Interamerican Development Advisory Services. Multiple staff at each school have been trained in its use and are comfortable doing all types of data entry in the system, including a school profile, teacher profile, student attendance, and student assessment scores. Awareness and use of the variety of reports the system can generate is less complete. NHP plans to extend the use of JSAS to at least 200 non-NHP schools within the next year by providing a self-installable CD and user manual (in development). A recently initiated large IDB-funded project, the Primary Education Support Project (PESP), also includes a focus on monitoring information systems and network communications. The project plans to begin with the reformulation and consolidation of databases in use at the ministry level and extend towards connection with the Regional Education Offices. PESP project personnel see the NHP project as supportive of PESP goals and objectives, and discussions have been held to ensure compatibility between systems at the point of interface between the schools and the ministry – the Regional Education Offices. It is recommended that the project be able to extend the life of the EMIS component to provide for sustainable use of the full potential of the JSAS to improve the quality of education. Background The NHP interventions which focus on educational technology through improved information systems are expected to contribute to the achievement of the Intermediate Results of “Improved Quality of Teaching,” as well as “Improved Management of Schools.” In previous USAID projects, MOEYC was assisted in creating a management information system and database in its Planning Unit. Through other projects, such as the IDB PEIP Project, MOEYC has developed other databases, for example, a database for the National Assessment Program (NAP) examinations. Based upon this recent history, the NHP proposed two interventions: Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 6 · 77 Intervention #9: To integrate and consolidate the various database systems of MOEYC into a unified system, along with further connections to other inter-related national databases, such as census data and/or information from the Jamaican Social Investment Fund (JSIF); and Intervention #10: To link schools with a regional and national EMIS network. A precondition for implementation has been that MOEYC ensure that each school has a stable electricity supply as well as at least one telephone line. Two aspects contributed to a re-design of this portion of the project: 1) funding delays resulted in delayed implementation of these interventions; and 2) a large project funded by IDB has been initiated and includes a focus on management information systems, along with network and communications. Implementation Status NHP established a school database system to assist in improving both school management and the quality of teaching in all 72 project schools. Part of this technical assistance activity is performed by Interamerican Development Advisory Services (IDEAS). Linkage between schools, regional offices, and MOEYC is pending the establishment of and access to a network system. An MIS software package, the Jamaica School Administrative System (JSAS), has been developed and provided to each of the participating schools, along with a computer for administrative use of this system. The principal and two to three teachers or other staff (such as a guidance counselor) at each school have been trained in the use of JSAS through both cluster workshops and through frequent on-site visits by the NHP staff on the EMIS team (originally six, now reduced to four). The JSAS is web/internet-ready and was developed in Microsoft Access and Visual Basic for Applications using Microsoft Office Developer. Thus the system can be distributed to schools license-free and royalty-free and requires no additional software other than the Microsoft Windows 98 or later operating system. Recently, version 4.3 was installed in schools, providing further improvements for user capability and for internal compatibility, without changing any of the essential core formats and functions as seen by users. One further modification, or version 4.4, is planned before the end of 2002. A report by the IDEAS EMIS specialist states that version 4.3 accomplishes the goal that the system become “twice as easy and four times as useful.” NHP reports also note that systematizing the data was a considerable challenge “given the differing styles and degree of organization of records at each school.” The JSAS can store information on school enrollment and other school census information (such as the number of grades and teachers, year of school construction, special courses offered, etc.), information on teacher profiles, student profiles, student attendance records, student assessment scores, and classroom grades. It provides for student withdrawal and re￾entry into a school. The system has several internal methods to cross-check the quality of Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 6 · 78 Aguirre International data entry and highlights the areas in which data are pending. It suggests and provides for complete back-up and recovery and provides for automatic re-enrollment to reduce work at the end of each term. (It also – oddly, in this age of political correctness – color codes information by gender, with blue for boys and pink for girls.) The JSAS provides for the generation of a variety of reports with multiple uses, both MOEYC-mandated and user￾selected; for example, class summaries of assessment scores can be used by teachers to adjust their lesson plans based on class competencies and weaknesses. NHP has also provided three computers to each of the six Regional Education Offices, and JSAS training for Education Officers is planned for this year. NHP has recently accumulated a fully functional database, with complete data entry for two years from one school, which can be used for demonstration training purposes. The JSAS, although Internet-ready, is not presently linked between schools and regional offices nor the MOEYC national offices. Most schools have a stable source of electricity but do not yet have a phone line that can be dedicated to Internet use. NHP reports cite that schools have “a high degree of dependency” for hardware and software technical support from the NHP EMIS team. Given the previously mentioned factors affecting project implementation, the re-designed plan for these interventions has as its objective “extending the use of the JSAS school-based administrative software system to as many primary and all-age schools as possible in a sustainable and cost-efficient manner.” It calls for extending the use of JSAS to at least another 200 non-NHP schools by the end of December 2002. These schools will initially be provided with a self-installable CD and user manual that are presently being finalized. Other forms of cluster training will then be coordinated once follow-up needs are identified. Qualitative evaluation results All schools visited have several staff who have been trained and input data into the JSAS. Usually this includes the principal and at least several teachers, along with the guidance counselor (if the school has this staff position). All staff interviewed describe the system as user-friendly and see the latest version as even better, including improvements which they themselves have suggested. They also state that there were no drastic or disorienting changes between the three versions, but rather continuous improvements. The computer capabilities of the users interviewed varied widely prior to training by NHP, although all had some experience with computers. Training workshops in JSAS use were followed up by frequent visits to the school by NHP personnel. Therefore, all users feel EMIS training was tailored to their needs. All principals interviewed have the student information files up-to-date, and a few have also continued on to other school data screens that are part of the school census required by MOEYC. Most staff are thoroughly comfortable with the system's screens for entering attendance data; however, only one grade in one school was up-to-date in attendance data entry. In this case, the guidance counselor had entered the data for the principal, who also is a teacher for a multi-grade class. Most schools had about half of the attendance data for this Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 6 · 79 school year entered. Many stated they plan to update attendance from their written logs after the school term ends. Only one school visited had entered any student assessment scores for this year (Grade One Readiness Inventory) although all stated they planned to enter Grade Three Language Arts and Mathematics diagnostic assessment results once the teaching period ended for this term. (It should be noted that only recently has the JSAS been thoroughly tested in the section focused on assessment scores; it is possible school staff have been waiting for this recent version, rather than that they are hesitant to use the full capacity of the system.) All the staff interviewed are aware of the variety of attendance reports which JSAS can generate (by student, by class, by date, by month, etc.), including mandatory reports to MOEYC. Staff displayed less familiarity with the variety of other reports, including class and school reports of assessment scores, which the system can generate. One very computer literate principal, who stated that during training sessions he assists others, enthusiastically described the potential uses of the JSAS reports – from increasing community awareness of the school's achievements, to motivating parental involvement in their child's education, to assisting teachers to evaluate their class's progress. Of the eight project schools visited, few have access to the Internet though several have plans to connect within the next year. All but one of the computers received for administrative use were fully functional during the evaluation visits. A few of the schools stated that they also permit small groups of students to use the administrative computer for educational purposes. Most schools had other computers which had been previously donated from other sources; however, none of the donated computers seen have the hard drive capacity of those received from NHP. Recommendations for NHP Extension Extend the life of the EMIS component. It is highly recommended that technical assistance in EMIS be extended beyond the present life of the project. The initial proposal was to "provide technical assistance for about two years following the initial training, to support the effective use of the system" and to "provide maintenance for both computer hardware and software with gradual transfer of responsibilities to the schools two years prior to the end of the project". Both of these activities have been delayed due to the overall delay of project funding. These activities are clearly critical to sustainability of the system. Provide additional training workshops. It is recommended that NHP conduct more training workshops with hands-on use of the complete sample database for principals, teachers and guidance counselors. NHP plans to reduce individual site visits to promote the sustainability of the system at each school; however, schools still need further training in the use of the reporting element of JSAS for improving the quality of education. Workshops should also include non-EMIS NHP staff, such as the literacy/math specialists. These staff can contribute their perspective as educators within the MOEYC system. They can assist in increasing participant awareness of and skills for the potential uses of the reports Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 6 · 80 Aguirre International which JSAS can generate, beyond meeting the requirements of MOEYC and towards stimulating community awareness, teacher accountability and parent involvement. In all interviews with project participants – including school staff, parents, students and NHP personnel – it was clear that NHP has created an environment in which change is fostered in a positive manner. The techniques used to promote this environment should, of course, also be a part of workshops in which the schools' use of JSAS reports is modeled. Pursue Internet connectivity for NHP schools. It would be worthwhile pursuing the initial plans to assist all NHP schools in becoming connected to the Internet in the short term during the final year of project implementation. This will support sustainability of JSAS if it links directly with regional MOEYC offices by the end of the life of the project. This will also assist in identifying potential long-term technical assistance needs to maintain network systems. Assist NHP schools to use the computer for educational activities. It would also be worthwhile for NHP to at least assist schools to identify the most cost-effective software available to promote literacy and numeracy (see the section on Educational Technology for more on this project intervention) as they are doing this on their own at present with, most likely, variations in effectiveness. Recommendations for Follow-On Project Linkage with the Primary Education Support Project. A project funded through the IDB, the Primary Education Support Project (PESP), contains plans to strengthen and consolidate the MOEYC database systems for improvement of management practices. The initial focus will be on financial systems and systems that come within the traditional school census, such as human resource information, school maintenance, etc., and also on networking technology and communications. Interviews with PESP and NHP project personnel revealed that they consider the PESP EMIS activities and NHP EMIS activities to be mutually supportive, with PESP working from the national ministry towards the regional offices, while NHP focuses from the individual schools towards the regional offices. PESP and NHP EMIS personnel have held discussions, which have also touched upon the details necessary to ensure future compatibility of systems. However, it is doubtful that this linkage between PESP and NHP systems will occur within the present life of the project. To ensure the long-term sustainability of what appears to be a potentially very useful system, the follow-on project should continue to have a strong EMIS component. Close coordination with MOEYC and PESP will be critical. Expand the JSAS to provide additional data for monitoring and analysis of student progress. Any future project should further build upon the JSAS, so that it supports additional educational strategies promoted by MOEYC and counterpart projects. An example is the possible incorporation within the MIS system of a means to assist teachers in Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 6 · 81 identifying and reporting on use of instructional technologies. At present the JSAS includes a required MOEYC report on administration of the school lunch program. If the school breakfast program proves to be sustainable during the remaining life of the project, any future project which might expand and replicate the school breakfast model should also add a feature to the JSAS that could be used to monitor and evaluate this program. Bibliography NHP 1997. New Horizons for Primary Schools Project Paper, Juarez and Associates, Inc.; September 1997. IDEAS 2002. Trip Report May 27 to June 4, 2002, by Bruce Newman, EMIS Specialist for IDEAS; June 2002. NHP 2001. Re-design Plan for Interventions 9 & 10 of New Horizons for Primary Schools, Juarez and Associates, Inc.; December 2001. Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 7 · 82 Aguirre International ANNEX 7 PARENT AND COMMUNITY INVOLVEMENT Background The New Horizons for Primary Schools project proposed two interventions to strengthen community and parent involvement in schools: governance and leadership training for school board members, PTA representatives, and members of Past Student Associations and other community groups; and parenting education and training in support of children's acquisition of literacy and numeracy skills. Within the partnership between the NHP, the MOEYC Project Implementation Unit (PIU) and the National Council on Education (NCE), NCE would take the lead role in promoting these interventions. School Development Teams would be formed and receive training to enhance the effectiveness of the role they play in fostering community participation in school management. Parents would receive education on their rights and responsibilities and be encouraged to participate more actively in the schools, through the PTA and other school activities, such as a school breakfast program. A number of parents from each school would receive training in organizing Study Circles with other parents as a way to further cascade the training received in the promotion of literacy and numeracy skills in primary school children. Implementation Status School governance and leadership training: A local consultant assisted the National Council on Education in bringing together community participants as stakeholders in the development of an Operations Manual on Governance and Leadership of School Boards. This guideline went through multiple reviews and improvements by stakeholders and formed the basis for the creation of School Development Teams. A training module developed includes governance and leadership training, operational guidelines for school principals and school development planning. All school principals have participated in an annual three-day workshop each summer. All 72 NHP schools have formed School Development Teams and participated in two cycles of designing a School Development Plan. The second cycle immediately followed upon the first, when the plans developed highlighted the areas in which School Development Teams required additional training. The National Council on Education also utilized a training module, with an accompanying manual, for “The Revitalization of School Parent Teacher Associations.” At present, several PTA leaders are preparing to visit the U.S. and participate in the National PTA conference. As is true for several of the project interventions, implementation of the parent training intervention was delayed by funding problems. Nevertheless, in 2001, between 268 and 340 Parent Training Coordinators (from 70 of the 72 NHP schools) participated in training sessions held in regional clusters. Evaluation and Recommendations: New Horizons for Primary Schools Project Aguirre International Annex 7 · 83 Qualitative Evaluation Results Principals, School Boards and School Development Plans: The School Development Plan (SDP) is the focal point of NHP's approach to school governance. It requires an effective school principal who involves stakeholders in a thorough needs assessment for the school and in careful planning to reach common goals. NHP has provided training to school principals in how to develop the SDP, and all schools visited had an SDP, but these were of varying quality. Some plans were simply a list of objectives and activities. The best plans, however, contained diagnostic needs assessments and prioritized objectives. Some non-NHP schools visited also had SDPs, but these tended to be focused more on infrastructure objectives while the SDPs of NHP assisted schools emphasized goals for student learning. The School Development Plans are created by a School Development Team (SDT). In the schools visited, SDT members included the principal, one or more Resource Teachers, one or more PTA members, the School Board Chairman and sometimes other teachers and/or community members. Students are rarely included. Principals mentioned that at first the SDT members did not understand the importance of the process and found it time￾consuming. However, most of those interviewed indicated that the SDP is now central to their plans for the school and provides a "focus" or "vision" for their efforts. Many cited specific achievements of the SDP; for example, one school has improved attendance on Fridays by introducing a "Friday afternoon club." Many principals and teachers interviewed did note that in the future they think it will be important to actively include more stakeholders in the design of the SDP. PTA representation in School Development Plans: PTA representatives interviewed describe a revitalization of the PTA organizations in their schools and a new vision of their role as promoting and supporting stronger relations between the schools and community through improved lines of communication and communication skills, conflict negotiation, structured processes for parental involvement, etc. Most PTA representatives report regular monthly attendance by 20-40 parents at PTA meetings, regardless of the size of the school, with at least three to six parents considered as "very active" in supporting school activities. Some PTAs mention bringing in guest lecturers on important aspects of parenting skills, an idea promoted through the training for Study Circles. Parent participation in schools: The participation by parents suggested in School Development Plan tends to still be limited to school maintenance or beautification aspects, although parents interviewed describe this as a role that they select. Principals interviewed are satisfied with the level of parental support received for Community Work Days, voluntary coverage of teachers’ classes (when sick or attending training activities), assistance with decoration of classrooms, etc. Parent Training Coordinators interviewed stated that the concept of Study Circles had not yet taken hold. One reason, they felt, was that training occurred near the end of each school term and parents were not easily motivated to initiate something new at that time. They describe a willingness to promote the Study Circles but do not have ideas or a plan of how to overcome Evaluation and Recommendations: New Horizons for Primary Schools Project Annex 7 · 84 Aguirre International the obstacles they have encountered thus far among the parent population. They expressed a need for additional training, follow-up visits on-site and support for visits to other NHP schools within their region. Perhaps in part due to the delay in implementation of the full range of activities foreseen to promote parental involvement, in schools with a breakfast program, this program has become the key vehicle for promoting more active involvement of parents through fundraising and management of these funds (see the section on the Health and Nutrition intervention for more information on parent participation). Recommendations for NHP Extension Extend the life of the Governance and Leadership Training intervention. School Development Teams have acquired a clearer vision of their potential role in schools but need further immediate support in order to achieve a sustainable model for community participation. NHP support should be extended in order to complete the cycle of trainings planned and support another cycle of the School Development planning process. Recommendations for Follow-On Project Reformulate the intervention for Parenting Education and Training. Design an intervention to address parenting skills through parent groups (Study Circles?), separate from other project activities such as the revitalization of PTAs, school breakfast programs, etc. Consider incentives to stimulate parent participation and provide close and frequent on site follow-up for the first year of support.