New Horizons for Primary Schools/Jamaica
FORMATIVE EVALUATION 2003
October 2003
Prepared for:
United States Agency for International Development
Jamaica
Project Undertaken by:
Juárez and Associates, Inc.
Prepared by
Kjell Enge and Heather Simpson
Juárez and Associates, Inc.
Contract No.: NHA-532-C-00-03-00060-00
ii
Table of Contents
B. Evaluation Methodology................................................................................................... i
C. Principal Findings .......................................................................................................... i
D. Implications ................................................................................................................... ii
A. Background .......................................................................................................................1
B. Methodology......................................................................................................................2
1. Indicators .......................................................................................................................2
2. Design............................................................................................................................2
C. Assumptions..................................................................................................................4
A. Student Performance.......................................................................................................5
1. Mathematics..................................................................................................................5
a) Third Grade ...............................................................................................................5
b) Sixth Grade ...............................................................................................................7
2. Language Arts...............................................................................................................9
a) Third Grade ...............................................................................................................9
b) Sixth Grade .............................................................................................................11
B. Teaching Quality.............................................................................................................13
C. Teaching Skills............................................................................................................15
D. School Visits by NHP Specialists.............................................................................18
E. System Support..............................................................................................................18
1. Professional Development........................................................................................18
2. School management..................................................................................................19
F. Community Involvement................................................................................................20
A. Conclusions .....................................................................................................................22
B. Implications .....................................................................................................................24
I. Executive Summary.............................................................................................................. i
I. INTRODUCTION..................................................................................................................1
II. FINDINGS..............................................................................................................................5
III. CONCLUSIONS AND IMPLICATIONS ......................................................................22
Appendix A: USAID Reporting Tables...................................................................................25
iii
List of Tables
Table 1: Change in Near Mastery on Third Grade Diagnostic Mathematics Test in NHP 
and non-NHP Schools by Gender and Year.................................................................... 6
Table 2: Change in Mastery on Third Grade Diagnostic Mathematic Test in NHP and 
non-NHP Schools by Gender and Year............................................................................ 6
Table 3: Change in Mastery and Near Mastery on Third Grade Diagnostic Mathematics 
Test in NHP and non-NHP Schools by Gender and Year............................................. 7
Table 4: Change in Near Mastery on GSAT Mathematics in NHP and non-NHP Schools 
by Gender and Year............................................................................................................. 7
Table 5: Change in Mastery on GSAT Mathematics in NHP and non-NHP Schools by 
Gender and Year.................................................................................................................. 8
Table 6: Change in Near Mastery and Mastery on GSAT Mathematics in NHP and non￾NHP Schools by Gender and Year.................................................................................... 8
Table 7: Change in Mean Scores on GSAT Mathematics in NHP and Comparison 
Schools by Gender and Year............................................................................................. 9
Table 8: Change in Near Mastery on Third Grade Diagnostic Language Arts Test in 
NHP and non-NHP Schools by Gender and Year ........................................................ 10
Table 9: Change in Mastery on Third Grade Diagnostic Language Arts Test in NHP and 
non-NHP Schools by Gender and Year.......................................................................... 10
Table 10: Change in Mastery and Near Mastery on Third Grade Diagnostic Language 
Arts Test in NHP and non-NHP Schools by Gender and Year................................... 10
Table 11: Change in Near Mastery on GSAT Language Arts in NHP and non-NHP 
Schools by Gender and Year........................................................................................... 11
Table 12: Change in Mastery on GSAT Language Arts in NHP and non-NHP Schools by 
Gender and Year................................................................................................................ 11
Table 13: Change in Near Mastery and Mastery on GSAT Language Arts in NHP and 
non-NHP Schools by Gender and Year.......................................................................... 12
Table 14: Change in Mean Scores on GSAT Language Arts in NHP and Comparison 
Schools by Gender and Year........................................................................................... 12
Table 15: Mean Classroom Environment Scores by School Size..................................... 13
Table 16: Interactions Initiated by Teachers and Students ................................................ 14
Table 17: Interactions by Classroom Context........................................................................ 15
Table 18: Quality of Interactions ............................................................................................. 16
Table 19: Availability and Use of Texts and Other Learning Materials............................. 17
Table 20: Teacher Support of NHP ........................................................................................ 17
Table 21: Teachers’ Recollection of NHP Specialists’ Activities ........................................ 18
Table 22: NHP Professional Development............................................................................. 19
Table 23: NHP School Management....................................................................................... 20
Table 24: NHP Community Involvement ................................................................................ 21
I. Executive Summary
A. Introduction
This document summarizes the results of the 2003 formative evaluation of the New Horizons for 
Primary Schools (NHP) Project in Jamaica. NHP is a five-year1
 effort to improve the 
Mathematics and Language arts of Jamaican primary school students, who because of poverty or 
other factors have had little success in school. The project is a partnership between the Jamaican 
Ministry of Education, Youth and Culture, USAID, and the NHP technical assistance contractor, 
Juárez and Associates.
The formative evaluation is conducted yearly near the end of the school year. It is designed to 
inform the implementation of NHP interventions and thereby permit NHP staff to target 
interventions in critical areas of the program. The formative evaluation process also serves to 
measure project results from an established baseline, which will contribute to the measurement 
of final project results. In 2001, 2002 and 2003, the formative evaluation had the additional 
purpose of building the capacity of Jamaican Education professionals in systematic qualitative 
data collection and the integration and interpretation of qualitative and quantitative data.
B. Evaluation Methodology
A team of Jamaican education professionals carried out the data collection for the evaluation. 
They employed a multi-method design, consisting of inventories, checklists, classroom 
observation forms, and focused interviews, to measure the conditions in place for effective 
learning in NHP classrooms. A stratified sample of 25 schools, or 35% of the 72 NHP schools 
served as the data source for the evaluation. Observational data were complemented by the 
results of the third grade diagnostic tests and the sixth grade GSAT results for 2003. Evaluators 
were trained in workshops dealing with qualitative data collection and data reduction, analysis 
and interpretation. The evaluation took place in May of 2003.
C. Principal Findings
NHP has been most successful in improving the near mastery levels of Mathematics. NHP 
students have improved over the baseline in1998 in both third and sixth grade and the 
improvement has been greater than that for children in the system as a whole. NHP students also 
have higher mean scores in Mathematics in 2003 than a matched comparison group of schools. 
Language Arts mastery appears to be a problem for the Jamaican primary education system as a 
whole. There is a general decrease in Language Arts performance in 2003 at both third and sixth 
grade levels. This follows a decline in the percentage of students reaching at least near mastery 
in 2001, 2002 and 2003. The increase in NHP girls reaching mastery over the baseline is equal to 
the non-NHP schools, but for boys it was considerably lower. 
The success of NHP in improving student performance is questionable. Although NHP students 
have improved in their mastery of Language Arts and Mathematics over the baseline in 1998 to 
2003, this improvement is only slightly higher for girls than that of similar schools without the 
NHP program over the same time period, and for boys it has been about equal or slightly lower.
 
1
 The original five-year contact was from 1998 through 2003, and in 2003, a two-year extension was signed with 
USAID/Jamaica
ii
NHP has been successful in changing classroom environments so that they are organized to 
facilitate learning. Classroom environments improved each year in NHP schools. Children’s 
work was displayed to a greater extent, teachers were positive when interacting with students, 
and in many classrooms, there was an improvement in the organization of space.
Although some progress has been made by the 2002 school year, the drop in 2003 indicates that 
the participatory, child-center classroom approaches, emphasized by the NHP program, have 
generally not been fully implemented in NHP classrooms.
NHP has been highly successful in providing ancillary learning materials to schools. However, 
such materials continue to be under-utilized in the classrooms.
The concentrated effort by NHP to provide hands-on professional development and other 
technical assistance at the school level has yet to show a significant impact on teacher behavior.
D. Implications 
The decline of language arts performance over three consecutive years is somewhat alarming. 
NHP might explore these trends at their training activities with teachers and principals. If funds 
are available, NHP might conduct a special study in collaboration with the Ministry evaluation 
unit in project and non-project schools to determine the cause of the problem.
The similarity of test performance between NHP students and students in matched comparison 
schools bring into question the amount of resources needed to make significant change among 
schools serving those students who have had the least success in school, because of poverty and 
other factors. NHP performance in relation to the comparison group should be monitored closely 
over the remaining life of the project.
The increased use of the participatory, child-centered methodologies, espoused by NHP and the 
new primary curriculum suggests that achieving behavior change in schools and classrooms is a 
long-term endeavor. It may be that significant changes will only be found as the project nears 
completion at the end of the two-year extension in 2004. However, the high percentage of 
traditional pedagogical practices bring in to question whether such change will be sufficient to 
improve student performance, beyond that related to general system improvement.
The increased availability of materials should be taken advantage of as part of the NHP technical 
assistance. If not already contemplated, workshops and technical assistance visits should focus 
on training teachers to effectively use materials.
Although the administrative infrastructure for improvement in learning appears to be in place 
and is an important achievement of the NHP project, it is not yet focused on supporting NHP 
objectives. The relatively low percentage of schools implementing activities related to language 
arts and mathematics improvement may require special training for teachers and administrators 
iii
to make diagnosis of student performance and planning of strategies that will enhance student 
abilities in Mathematics and Language Arts and explicit part of the administrative process.
1
I. INTRODUCTION
This document describes the results of the fifth year of formative evaluation of the New 
Horizons for Primary Schools (NHP) Project. The evaluation is carried out near the end of the 
Jamaican school year (May-June) to provide a barometer of the progress of the project on a 
series of school and classroom indicators. However, as these indicators are related to the results 
of the third grade diagnostic and sixth grade GSAT tests, the report is not available until those 
test results are reported in at the end of August and the beginning of September. Originally, the 
formative evaluation had two purposes. First, the formative evaluation results inform the 
implementation of NHP interventions and permit NHP staff to target interventions in critical 
areas of the program. The results complement those of ongoing assessments of the 
implementation process undertaken informally through school visits, feedback on professional 
development efforts and periodic communication with school administrators and teachers. 
Second, the formative evaluation process serves to measure project results from an established 
baseline. As it provides systematic monitoring of performance over time, formative evaluation 
contributes to the measurement of final project results. Baseline indicators and projections of 
change over time derived from the 1999 formative evaluation are found in Appendix A of this 
report. 
In 2001, 2002 and 2003, the formative evaluation had an additional purpose. In order to respond 
to the capacity building interests of the Ministry of Education, workshops on evaluation 
methodology were held for technicians in the Ministry of Education, members of local teachers’ 
colleges, and New Horizon Project personnel. The workshops dealt with observation and 
interview techniques to measure progress toward NHP objectives. An additional workshop on 
data analysis and results of the evaluation was also conducted in 2001. Dr. Ray Chesterfield and 
Dr. Kjell Enge, who are experienced education evaluators, conducted the 2001 workshops. In 
2002, Dr. Yasmeen Yusuf-Khalil, together with Heather Simpson and José Ferrel of Juárez and 
Associates conducted the data collection workshop, and in 2003, the workshop was managed and 
conducted by Heather Simpson and José Ferrel. Following the data collection workshop 
training, a team of the workshop participants collected data from a sample of NHP primary 
schools.
A. Background 
The primary objective of New Horizons for Primary Schools (NHP) is to enhance the 
performance of Jamaican primary school students in numeracy and literacy. The focus of the 
technical assistance component of the project is on those children who, because of poverty and a 
lack of other enabling conditions, have had little academic success in school. Increased 
academic success is to be accomplished through the development of model interventions that, 
when tested, can be used to improve the performance of low-achieving children throughout 
Jamaica. Thus, the products of the contractor’s work are changes in schools and classrooms that 
result in individual students having greater academic success in primary school. Such results 
include measurement of the indicators for the USAID strategic objective. 
Systems, such as computerized administrative and student tracking systems, are also being 
implemented over the life of NHP. These systems are to assist schools in monitoring their own 
2
performance. The results of such individual school monitoring can be aggregated to examine 
project performance. Similarly, NHP is integrating MOEC databases to provide additional data 
sources for monitoring performance. Until such systems are fully operational, however, 
monitoring is being carried out as part of the formative evaluation effort designed to provide 
feedback to program technicians implementing the interventions. As formative evaluation 
requires in-depth data collection, a representative sample of NHP schools is selected each year 
for evaluation purposes.
Many of the indicators for monitoring performance are complex concepts that require the 
combination of qualitative and quantitative methods to measure accurately. The following pages 
discuss the procedures used to collect data on NHP indicators. Subsequent chapters present the 
findings of the formative evaluation, in terms of change from the baseline data, and provide 
conclusions and implications drawn from these findings.
B. Methodology 
1. Indicators
The indicators are taken largely from the U. S. literature on school/classroom effectiveness and 
on the growing body of international literature on classroom interaction and educational quality. 
Three levels of indicators were used. The first relates to student performance in terms of 
mastering the curriculum. The second consists of indicators of teacher performance that are 
generally associated with greater quality in terms of students’ academic performance. The third 
are the indicators of system support or enabling factors such as efficient school management, 
professional development opportunities for teachers, and parent participation in the education of 
their children, that must be in place to improve the performance of individual children.
2. Design
A multi-method design, consisting of inventories, checklists, classroom observation forms, and 
focused interviews, was employed to measure the conditions in place for effective learning. This 
design allows for the measurement of the impact of the interventions implemented to improve 
learning, especially among students who have had limited success in school. Evaluation efforts 
focused on both females and males. This is important not only to ensure that initiatives are 
equitable but also to identify initiatives and strategies that are successful regardless of gender. 
Study Sample: A stratified sample of 35% of project schools was drawn from the universe of 
72 schools. Schools were stratified by size (small, medium, or large) and type (primary or all 
age) then randomly selected within strata. As the focus of the project is a “ground-up” approach 
that begins with needs identified by participating schools, those schools that had been most 
involved in NHP activities during the year were over-sampled. The final sample consists of 25 
schools and 47 classrooms for intensive data collection and analysis.
The focus of the formative evaluation was on third grade. The purpose of the formative 
evaluation was to obtain in-depth, systematic data, in a limited amount of time. Thus, it 
concentrated on one grade as an indicator of general progress. Third grade was chosen, because 
there are test scores available that allow greater diagnostic ability and permit the monitoring of 
3
change in the cohort of third graders serving as the baseline over the life of the project. This is 
important because both the1998 and 1999 NAP scores suggest that NHP children fall behind 
principally between third and sixth grade.
In the first two years of the evaluation, first, second, fourth, fifth and sixth grade classrooms 
were also observed. The data from these classrooms showed the same general patterns as those 
for the sample as a whole. This suggests that for monitoring purposes, third grade results can be 
used as a general indicator of progress.
Comparison Sample: A midterm evaluation of NHP conducted in June/July 2002, suggested 
that a comparison sample of schools similar to the NHP schools should be drawn. The 
evaluators argued that this would provide a fairer measure of NHP progress than comparing NHP 
to all non-NHP schools. In order to comply with this request, the formative evaluation team 
created a retrospective comparison group. Each of the 72 NHP schools were matched by size 
and by 1998 GSAT performance to a similar school in the same geographical area. The GSAT 
test results for each year for this group of schools were then compared to NHP schools.
Instruments: Instruments included classroom maps, materials inventories, classroom 
observation forms, classroom environment assessments, and interviews guides for use with 
teachers, students and school principals. Maps were employed to identify children and to 
examine the context in which they interact with teachers. Materials inventories measured both 
the presence and use of all materials at different times during math and language arts lessons. 
Observational sweeps were made at three points in time during each academic context. At each 
sweep, the number of books and ancillary materials available and in use, were counted. 
Classroom interaction was measured through a teacher-student interaction protocol. This 
instrument focused on teachers' interactions with individual students and the nature of those 
interactions in different academic classroom activities. In order to ensure consistency and 
control for contemporaneous events that might influence behavior patterns, the form was used 
for ten minutes at six different times during the instructional day in third grade classrooms. 
Three observations took place during mathematics lessons and three during language arts. Thus, 
a behavioral sample of 30 minutes for each of the target content areas was created. Researchers 
used the classroom environment instrument to rate the appropriateness of the classrooms for 
child-centered learning. 
Teachers’ perceptions of the interventions, as well as their mastery of and commitment to the 
new approaches implemented under NHP, were tapped by a teacher interview schedule. 
Similarly, changes in the school management planning and systems were measured through an 
interview with the principal. Students were queried about activities in the home and involvement 
of parents in the children’s reading.
Fieldwork Procedures: A schedule of school visits was developed with the field workers, and 
NHP staff contacted the principals and informed them of the visits. Two of the backstop 
personnel for the institutional contractor assisted a local researcher coordinator in scheduling and 
supervising the fieldwork. Fieldworkers synchronized observations through training exercises 
during the workshop. This training included exercises with the instruments using videotapes of 
4
classroom interaction in schools to ensure consistency in observations and interviewing. Parallel 
observations were conducted with the instruments until an inter-observer agreement coefficient 
of above .76 was reached for all observational instruments. 
The researchers worked in small teams of up to six people and spent up to one full day at each 
school collecting data. Procedural guides and operational definitions were attached to specific 
instruments as references to ensure consistency in field procedures during the investigation. 
Following each day of fieldwork, the coordinator gathered the instruments from the teams and 
the backstop personnel monitored the quality of the data collection and entered the information 
into SPSS spreadsheets. Twenty-five schools were visited and complete sets of data were 
collected from 48 classrooms.
 
Data Analysis: The principal unit of analysis was the classroom. As the interventions are 
focused largely on improving teaching, it is changes in classroom-level environments and 
behaviors that affect student learning. Data analysis consisted of calculating the absolute and 
relative frequencies of each behavioral indicator and making comparisons across the three 
evaluation years. Differences by types of schools were also examined. Special indices were 
created to measure complex issues such as teaching quality. Where appropriate, statistics such as 
chi-square and correlations were used to examine relationships among the sample.
C. Assumptions 
The ongoing formative evaluation is based on several assumptions. First, the school and the 
class are the key units of analysis in planning and intervening to improve the quality of learning. 
Second, the school is a social system and the interaction of all of the elements within a school 
has an influence on student learning beyond that provided individually by inputs to the school. 
This is not to suggest that the uniqueness of each school makes aggregate measurement 
impossible, but rather that accurate measurement of the impact of schooling is a complex 
undertaking requiring the integration of a variety of data collection approaches.
5
II. FINDINGS
A. Student Performance 
Jamaica is promoting pupil-centered “everyone can learn” concept of teaching rather than a 
norm-based “cream of the crop” approach. Thus, the focus is shifting to all children’s mastery of 
the curricular content. This means that the array of individual scores will shift from the normal 
distribution or “bell shaped curve” associated with a norm-based assessment and mean scores, 
toward a “J-curve” with a few students falling at the low end and the middle and most scores 
reflecting a high degree of learning. However, with the current inverse J-curve, the first step is 
to move students to “near mastery” levels. The formative evaluation originally examined both 
third and sixth grade mastery. Thus, the NAP and Student Assessment Unit criteria of less than 
50% of the items in each domain correct as “no mastery” level, was used in the evaluation. 
Although NAP does not designate mastery levels for the sixth grade GSAT, the criteria used at 
the third grade level was employed in determining student progress (less than 50% correct = “no 
mastery,” 50% to 75% correct = “near mastery” and above 75% = “mastery”. 
It has proved somewhat difficult to obtain complete data sets of either NHP or non-NHP third 
grade tests, owing to their diagnostic purpose, which leads schools not to report results. Thus, 
the USAID strategic objective team uses only sixth grade in their reporting. The formative 
evaluation will continue to include third grade tests when they become available. Both third 
grade and results and sixth grade results for 2003 are included in this report. All test results are 
reported in relation to 1998, the baseline year.
1. Mathematics
a) Third Grade
Table 1 shows the change in the percentage of children reaching near mastery of the third grade 
mathematics curriculum, as measured on the diagnostic test for that subject. Changes in student 
performance in NHP schools are compared to all primary schools not participating in the NHP 
program. Both yearly change and total change from the baseline are provided. As can be seen, 
there is a moderate overall change (+9.5%) for NHP from 1998 to 2003. Over these six years, 
NHP children have made relatively greater gains in test performance in reaching near mastery 
than Jamaican third grade school children as a whole, and this is true for both boys and girls. 
However, in the 2000 school year, there was a decline in third grade near mastery performance 
for all groups of children and NHP children had greater declines than their counterparts. In 
2003, the NHP girls experienced a slight decline in comparison to a 3.4% increase for the non￾NHP population, whereas NHP boys were 5.4% lower than non-NHP boys.
6
Table 1: Change in Near Mastery on Third Grade Diagnostic Mathematics Test in NHP 
and non-NHP Schools by Gender and Year
Third Grade Female Third Grade Male
Year NHP Change 
by Year
Non￾NHP
Change 
by Year
NHP Change 
by Year
Non￾NHP
Change 
by Year
1998 37.9 43.0 28.8 33.8
1999 45.1 +7.2 45.0 +2.0 37.0 +8.2 38.5 +4.7
2000 38.0 -7.1 43.0 -2.0 29.0 -8.0 35.0 -3.5
2001 41.8 +3.8 41.0 -2.0 34.8 +5.8 36.5 +1.5
2002 48.1 +6.3 45.2 +4.2 37.1 +2.2 40.5 +4.0
2003 47.4 -0.7 48.6 +3.4 31.2 -5.9 40.0 -0.5
Change from 
Baseline +9.5 +5.6 +2.4 +6.2
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
Lower near mastery levels may be the result of greater numbers of the third grade population 
reaching mastery. This is shown both by the percentage of children in the mastery category in 
subsequent years and by the total percentage of children in the mastery and near mastery 
categories. Ideally, all children will be in the mastery category. Table 2 shows a continued 
decline for both NHP and non-NHP schools, but the largest declines are in the non-NHP schools 
for both boys and girls. Overall, girls had nearly twice the decline as compared to boys.
Table 2: Change in Mastery on Third Grade Diagnostic Mathematic Test in NHP and 
non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change 
by Year
Non￾NHP
Change 
by Year
NHP Change 
by Year
Non￾NHP
Change 
by Year
1998 9.3 12.7 4.1 7.9
1999 19.4 +10.1 28.0 +15.3 11.8 +7.7 19.5 +11.6
2000 18.0 -1.4 24.0 -4.0 9.0 -2.8 15.0 -4.5
2001 21.9 +3.9 35.3 +11.3 11.4 +2.4 25.3 +10.3
2002 17.2 -4.7 28.7 -6.6 8.9 -2.5 19.7 -5.6
2003 10.3 -6.9 19.3 -9.4 6.0 -2.9 15.0 -4.7
Change from 
Baseline +1.0 +6.6 +1.9 +7.1
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
Table 3 shows that there has been substantial improvement in third grade children’s performance 
in mathematics from the 1998 baseline to 2002. But one year later in 2003, both genders in NHP 
and non-NHP schools experienced declines, and the declines in the NHP schools were 2-3% 
greater than in their non-NHP counterparts. NHP boys had the highest decrease and the lowest 
was for the non-NHP boys. At the same time, the NHP girls had lower decreases than their non￾NHP girls. 
7
Table 3: Change in Mastery and Near Mastery on Third Grade Diagnostic Mathematics 
Test in NHP and non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change 
by Year
Non￾NHP
Change 
by Year
NHP Change 
by Year
Non￾NHP
Change 
by Year
1998 47.2 55.7 32.9 41.5
1999 64.5 +17.3 73.0 +17.3 48.8 +15.9 58.0 +16.5
2000 56.0 -8.5 67.0 -6.0 38.0 -10.8 50.0 -8.0
2001 63.7 +7.7 76.3 +9.3 46.2 +8.2 61.8 +11.8
2002 65.3 +1.6 73.9 -2.4 46.0 -0.2 60.2 -1.6
2003 57.7 -7.6 68.0 -5.9 37.2 -8.8 54.5 -5.7
Change from
Baseline +10.5 +12.3 +4.3 +13.0
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
b) Sixth Grade
Table 4 shows the change in the percentage of children reaching near mastery of the sixth grade 
mathematics curriculum, as measured on the GSAT test for that subject. Changes in student 
performance in NHP schools are compared to all primary schools not participating in the NHP 
program. As can be seen, there is significant change from 1998 to 2002. NHP children have 
made relatively greater gains in reaching near mastery than Jamaican primary school children as 
a whole, and this is true for both boys and girls. Both girls and boys in NHP schools had larger 
gains than their counterparts in the 2001-2002 school year. But all children experienced a 
decrease in near mastery from 2002 to 2003, and the decreases for both NHP boys and girls were 
about twice that of their non-NHP counterparts.
Table 4: Change in Near Mastery on GSAT Mathematics in NHP and non-NHP Schools 
by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by 
Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change by 
Year
1998 13.9 26.5 5.5 13.8
1999 22.2 +8.3 31.9 +5.4 8.8 +3.3 17.3 +3.5
2000 30.0 +7.8 31.8 -0.1 19.2 +10.4 22.9 +5.6
2001 32.2 +2.2 36.8 +5.0 20.6 +1.4 25.9 +3.0
2002 36.0 +3.8 38.4 +1.6 23.5 +2.9 27.6 +1.9
2003 27.1 -8.9 34.2 -4.2 17.2 -6.3 24.2 -3.4
Change from 
Baseline +13.2 +7.7 +11.7 +10.4
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
NHP has been less successful in moving children to mastery than to near mastery. As shown in 
Table 5, the overall increase in mastery levels have been higher for non-NHP boys and girls, but 
the NHP girls have increase more than the boys. Although there has been an overall positive 
increase among NHP children of both genders from 1998 to 2000, there have been annual 
8
decreases for both NHP and non-NHP students from 2000 to 2003, and the largest decline was 
for girls from 2002 to 2003. Both genders in the Jamaican primary school population as a whole 
declined in the percentage of students reaching mastery, but the percentage of children at the 
mastery level in non-NHP schools is almost triple that of NHP boys and girls.
Table 5: Change in Mastery on GSAT Mathematics in NHP and non-NHP Schools by 
Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 0.3 2.9 0.2 2.3
1999 1.7 +1.4 6.9 +4 0.8 +0.6 4.0 +1.7
2000 10.9 +9.2 22.8 +15.9 5.7 +4.9 15.5 +11.5
2001 9.8 -1.1 21.5 -1.3 5.2 -0.5 14.8 -0.7
2002 9.2 -0.6 19.2 -2.3 5.3 +0.1 13.5 -1.3
2003 6.0 -3.2 15.1 -4.1 4.0 -1.3 11.6 -1.9
Change from 
Baseline +5.7 +12.2 +3.8 +9.3
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
The change in children at near mastery and mastery has been more almost 20% for girls and 15% 
for boys (Table 6). This increase is about the same for NHP girls as for girls in the system as a 
whole. However, the percentage of girls with no mastery is still 16% greater in NHP than in 
system as a whole, owing to the low initial performance of children in the program. The general 
population of boys has shown a four percent greater increase in the combined near￾mastery/mastery as the boys in NHP.
Table 6: Change in Near Mastery and Mastery on GSAT Mathematics in NHP and non￾NHP Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 14.2 29.4 5.7 16.1
1999 23.9 +9.7 38.8 +9.4 9.6 +3.9 21.3 +5.2
2000 40.9 +17.0 54.6 +15.8 24.9 +15.3 38.4 +17.1
2001 42.0 +1.1 58.3 +3.7 25.8 +0.9 40.7 +2.3
2002 45.2 +3.2 57.6 -0.7 28.8 +3.0 41.2 +0.5
2003 33.1 -12.1 49.3 -8.3 21.2 -7.6 35.9 -5.3
Change from 
Baseline +18.9 +19.9 +15.5 +19.8
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002 database
Table 7 presents the mean scores in mathematics for NHP and a matched comparison group. As 
can be seen, NHP progress in Mathematics has been almost identical to that of similar schools 
9
without the NHP program. Both groups have increased by about 15 to 20% in terms of mean 
scores. 
Table 7: Change in Mean Scores on GSAT Mathematics in NHP and Comparison 
Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Comparison Change 
by
 Year
NHP Change 
by
 Year
Comparison Change 
by
 Year
1998 26.6 28.5 21.2 21.7
1999 31.3 +4.7 32.3 +3.8 25.9 +4.7 26.3 +4.6
2000 35.3 +4.0 36.0 +3.7 28.3 +2.7 28.0 +1.7
2001 37.0 +1.7 38.0 +2.0 30.2 +1.9 31.1 +3.1
2002 38.2 +1.2 37.9 -0.1 32.1 +1.9 31.4 +0.3
2003 43.4 +5.2 44.0 +6.1 37.9 +5.8 38.0 +6.6
Change 
from 
Baseline
+16.8 +15.5 +16.7 +16.3
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
2. Language Arts
a) Third Grade
Tables 8, 9 and 10 show that improvement in students’ mastery of third grade Language Arts 
curriculum has been difficult for NHP to achieve. The percentage of both NHP and non-NHP 
children reaching near mastery has declined since 1998. As mentioned, such a decline may be 
the result of a greater percentage of students reaching mastery. This appears to be the case 
among NHP and non-NHP children, but as shown in Table 8, the 2002 to 2003 decline for NHP 
boys was the lowest in comparison to the others. Furthermore, in the mastery levels the NHP 
boys show the smallest increase since 1998. All students have an increase in mastery levels from 
the baseline that are greater than the decline in near mastery. At the same time, there is a net 
increase in the combined near mastery and mastery levels non-NHP students and NHP boys, but 
for the NHP girls there has been a small decrease since the 1998 baseline. 
10
Table 8: Change in Near Mastery on Third Grade Diagnostic Language Arts Test in 
NHP and non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change 
by Year
Non￾NHP
Change 
by Year
NHP Change 
by Year
Non￾NHP
Change 
by Year
1998 46.9 40.7 37.8 40.0
1999 42.0 -4.9 34.6 -6.1 37.9 +0.1 34.8 -5.2
2000 42.0 0 39.0 +4.4 34.0 -3.9 37.0 +2.2
2001 36.7 -5.3 32.4 -6.6 33.3 -0.7 32.9 -4.1
2002 42.1 +5.4 35.7 +3.3 36.4 +3.1 36.3 +3.4
2003 40.2 -1.9 36.9 +1.2 36.3 -0.1 36.8 +0.5
Change from 
Baseline -6.7 -3.8 -1.5 -3.2
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001 and 2002 database
Table 9: Change in Mastery on Third Grade Diagnostic Language Arts Test in NHP and 
non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change 
by Year
Non￾NHP
Change 
by Year
NHP Change 
by Year
Non￾NHP
Change 
by Year
1998 26.2 37.7 13.5 21.9
1999 31.1 +4.9 46.1 +8.4 16.5 +3.0 29.0 +7.1
2000 28.0 -3.1 38.0 -8.1 13.0 -3.5 23.0 -6.0
2001 32.8 +4.8 48.5 +10.5 16.9 +3.9 33.1 +10.1
2002 33.9 +1.1 45.7 -2.8 16.2 -0.7 29.4 -3.7
2003 32.1 -1.8 44.4 -1.3 15.6 -0.6 28.7 -0.7
Change from 
Baseline +5.9 +8.0 +2.1 +6.8
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
Table 10: Change in Mastery and Near Mastery on Third Grade Diagnostic Language 
Arts Test in NHP and non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change 
by Year
Non￾NHP
Change 
by Year
NHP Change 
by Year
Non￾NHP
Change 
by Year
1998 73.1 78.4 51.3 61.9
1999 73.1 0 80.7 +2.3 54.4 +3.1 63.8 +1.9
2000 70.0 -3.1 77.0 -3.7 47.0 -7.4 60.0 -3.8
2001 69.5 -0.5 80.9 +3.9 50.2 +3.2 66.0 +6.0
2002 75.9 +6.4 81.4 +0.5 52.6 +2.4 65.7 -0.3
2003 72.3 -3.6 81.3 -0.1 51.9 +0.7 65.5 -0.2
Change from 
Baseline -0.8 +2.9 +0.6 +3.6
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
11
b) Sixth Grade
Change in language arts performance at the near mastery level follows a similar trend to that for 
mathematics among NHP students. There are greater gains over time for NHP students than for 
their counterparts. However, in 2003 there is a continuation of the general decline in the 
percentage of NHP children at near mastery which started 2002. Boys in the general population 
follow a pattern similar to NHP children. Non-NHP girls, however, show an overall drop from 
the baseline year.
Table 11: Change in Near Mastery on GSAT Language Arts in NHP and non-NHP 
Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 34.2 39.6 15.2 23.5
1999 37.3 +3.1 42.3 +2.7 17.3 +2.1 25.3 +1.8
2000 36.8 -0.5 33.7 -8.6 22.4 +5.1 25.0 -0.3
2001 38.7 +1.9 37.8 +4.1 25.6 +3.2 27.8 +2.8
2002 37.7 -1.0 38.4 +0.6 22.0 -3.6 27.1 -0.7
2003 37.5 -0.2 39.4 +1.0 20.9 -1.1 27.1 0.0
Change from 
Baseline +3.3 -0.2 +5.7 +3.6
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
Change in the percentage of students reaching mastery is similar for both NHP and the general 
population of Jamaican primary students. There is a decline among all groups in 2002, and the 
declined continued in 2003, except for the non-NHP boys that had a small increase of about 1%. 
The decline among NHP girls is slightly greater than that of their counterparts. Overall change 
from the baseline is similar for all groups. However, lower percentages of NHP children are at 
mastery because of lower initial levels in 1998.
Table 12: Change in Mastery on GSAT Language Arts in NHP and non-NHP Schools by 
Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 8.3 20.4 2.1 8.2
1999 9.3 +1 18.2 -2.2 1.6 -0.5 7.4 -0.8
2000 26.6 +17.3 39.3 +21.1 12.6 +11.0 24.6 +17.2
2001 18.6 -8.0 33.7 -5.6 8.4 -4.2 20.2 -4.4
2002 12.9 -5.7 24.7 -9.0 7.4 -1.0 13.5 -6.7
2003 10.4 -2.5 22.5 -2.2 5.5 -1.9 14.6 +1.1
Change from 
Baseline +2.1 +2.1 +3.4 +6.4
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
12
As shown in Table 13, NHP children of both genders have had larger relative decreases in 
reaching near mastery or mastery from 2002 to 2003 than the remaining sixth grade population. 
However, there has not been the same success in closing the original gap between NHP students 
and the general population in language arts as was found in mathematics. The difference in the 
combined change is 5.4%, but the boys have nearly double the gain reaching 9.1%. The gap 
between NHP and non-NHP boys and girls is about 15%. The negative change found in children 
reaching mastery in 2001, 2002 and 2003 is reflected in the relative drops in the combined 
percentages across all groups.
Table 13: Change in Near Mastery and Mastery on GSAT Language Arts in NHP and 
non-NHP Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 42.5 60.0 17.3 31.7
1999 46.6 +4.1 60.5 +0.5 19.9 +2.6 32.7 +1.0
2000 63.4 +16.8 73.0 +12.5 35.0 +15.1 49.6 +16.9
2001 57.3 -6.1 71.5 -1.5 34.0 -1.0 48.0 -1.6
2002 50.6 -6.7 63.9 -7.6 29.5 -4.5 43.1 -4.9
2003 47.9 -2.7 61.9 -2.0 26.4 -3.1 41.7 -1.4
Change from 
Baseline +5.4 +1.9 +9.1 +10.0
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
The results for language arts mean scores with the comparison group of children shown in Table 
14 are similar to those of mastery for the population as a whole. NHP students have made 
greater gains over comparison group. However the difference in gains is about three percentage 
points for the girls and less than one percent for the boys. In addition, both groups have had 
decline in 2001 and 2002, but increased substantially in 2003. However, all the increase were 
remarkably similar in magnitude.
Table 14: Change in Mean Scores on GSAT Language Arts in NHP and Comparison 
Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Comparison Change 
by
 Year
NHP Change 
by
 Year
Comparison Change 
by
 Year
1998 37.3 40.5 27.7 28.7
1999 38.5 +1.2 40.1 -0.4 28.4 +0.7 29.5 +0.8
2000 44.7 +7.4 44.9 +4.8 33.7 +5.3 33.4 +3.9
2001 42.9 -1.8 44.7 -0.2 33.6 -0.1 34.1 +0.7
2002 40.7 -2.2 40.9 -3.8 33.4 -0.2 32.6 -1.5
2003 49.7 +9.0 50.2 +9.3 40.5 +7.1 40.9 +8.3
Change 
from 
Baseline
+12.4 +9.7 +12.8 +12.2
Source: NAP 1998 and 1999 database, Student assessment Unit 2000, 2001, 2002, 2003 database
13
B. Teaching Quality
Teaching quality was measured through an index made up of three generally accepted standards 
for determining teacher performance: content knowledge of students; environment for student 
learning; and teaching for student learning. The first of these dimensions has been discussed in 
the previous section. Third grade performance, measured as the percentage of NHP children 
reaching near mastery and mastery over all NHP children taking the third grade diagnostic tests 
was used as the measure of content knowledge. Both mathematics and language arts 
performance are used in this index. There has been improvement in the overall index each year 
of project implementation. However, the change has been small. It has gone from .43 in 1999 to 
.52 in 2002 and then back down to .47 in 2003. As will be shown below, this is largely the result 
of the minimal change in teaching for student learning. 
Learning environment standards relate to the social and emotional components of learning as 
prerequisites to and context for academic achievement. Thus, the focus is on the physical setting 
created by the teacher and the resources available. A six-item scale, dealing with the fostering of 
a positive self-concept, the creation of a nurturing environment that supports gender equity, and 
the organization of space and materials to allow a variety of learning opportunities, was used to 
measure the quality of the environment. Researchers used the assessment instrument after a 
complete series of observations in a classroom. Specific criteria were provided with each item to 
ground the ratings. Ratings were made on a three-point scale of “not met,” “partially met,” and 
“fully met”. Thus, scores ranged between a minimum of six and a maximum of 18. Scores 
were expressed as a ratio of the actual score over the total possible score.
Table 15 compares the classroom environment scores for 1999, 2000, 2001, 2002 and 2003 as 
been and improvement each year. This improvement is related to the implementation of the new 
curriculum in NHP schools and the interventions of NHP. Both emphasize changing the 
classroom environment to create a participatory situation for students. Classroom environment 
scores that were lower for the large schools in 1999 to 2001 have become quite similar by 2003.
Table 15: Mean Classroom Environment Scores by School Size
Mean/School Size 1999 2000 2001 2002 2003
Small .5929 .6389 .7350 .7589 .7996
Medium .5900 .6588 .7359 .7597 .7993
Large .4867 .5490 .7080 .7845 .7948
Total .5464 .6115 .7218 .7711 .7977
14
In 2001, 2002 and 2003, classrooms generally met criteria of lack of physical punishment and 
interacting with individual children often. Equal lighting, ventilation, and furniture for boys and 
girls were also generally met, and there was an increase in displaying children’s work. Other 
criteria such as creating a variety of learning opportunities within the classroom, encouraging 
children to express themselves with peers and adults, using materials that showed males and 
females in traditional and non-traditional roles, showed improvement in 2003. This reflects 
teachers increasing ability to use the limited space available in many of the classrooms, 
especially those in larger schools, in creative ways.
Teaching for student learning is concerned with the act of teaching and its overall goal of helping 
students understand the content that they are imparting and the ability to present the content in a 
manner that is consistent with the knowledge, interests and abilities of the students. For the 
purposes of monitoring, the focus has been on interactions in the classroom between teachers and 
students. Student-initiated interactions were taken as an indicator; as such interactions show 
teachers’ willingness to recognize student input. Student-initiated were found to be a very low 
percentage of all interactions in teacher-centered classrooms. As mentioned, a corpus of 60 
minutes of observations of academic lessons was collected in each classroom. These 
observations were divided equally between mathematics lessons and language arts lessons.
Table 16 presents the percentage of observed interactions initiated by teachers and students in the 
normally occurring contexts of the classroom in 1999, 2000, 2001, 2002 and 2003. The table 
shows the percentage of interactions initiated by each actor in the contexts observed taking place 
in the classroom. The bottom row provides the overall percentage of interactions initiated by 
teachers, boys, and girls. Teacher-initiated interactions predominate in all four years. They make 
up at least 87.7% of all interactions. Student-initiated interactions increased somewhat from 
1999 to 2000, but decreased in 2001. They increased slightly in 2002 but went on to decrease in 
2003, nearly reaching the 1999 baseline levels. The continued high percentage of teacher￾initiated interactions suggest that there has been little progress in changing the pedagogy 
employed by NHP teachers, as teaching strategies remain centered on the teacher initiating 
learning opportunities for children. Little difference is noted by the gender of the students, as 
both boys and girls initiate interactions with similar frequency.
Table 16: Interactions Initiated by Teachers and Students
Interaction Initiator Year Teacher Boy Girl
1999 92.5% 3.8% 3.6%
2000 88.7% 5.2% 6.1%
2001 90.1% 3.6% 4.9%
2002 87.7% 5.1% 6.9%
2003 91.2% 3.8% 5.0%
15
Table 17 shows the types of contexts in which the interactions occurred. As can be seen there 
has been a change in use of small group contexts in NHP schools during the 2002 school year. 
The use of this learning context, which is indicative of student participation and a 
decentralization of learning, has increased by 10%, but then went on to decrease by about 3% in 
2003. At the same time there was a 10% increase in large group activities and a continued 
decrease in the relative amount of seatwork. The traditional context of a large group in which the 
teacher works with the entire class remains the principal instructional method and is the context 
in which three-quarters of interactions occur. As would be expected, the participation in these 
contexts is very similar for girls and boys.
Table 17: Interactions by Classroom Context
Classroom Context 1999 2000 2001 2002 2003
Teacher-led small group 9.2% 2.4% 7.3% 17.5% 13.3%
Student-led small group 2.1% .3% .5% 0.4% 1.5%
Large group 49.2% 75.5% 65.2% 66.0% 75.7%
Seatwork 34.4% 19.4% 23.9% 15.2% 6.4%
No instruction 5.1% 2.5% 3.2% 0.7% 3.1%
C. Teaching Skills 
Several indicators of teaching skill are important to the NHP project. Obviously, the ability to 
effectively create an environment that instills self-confidence in students and allows them 
multiple learning opportunities, discussed previously under teaching quality, is related to 
pedagogical ability. The focus here is on specific behaviors engaged in by teachers that 
encourage children to participate in the learning process. Included are: the quality of teacher￾student interactions and the use of materials by students; teachers’ mastery of and commitment to 
the interventions introduced by NHP; and teachers’ strategies for encouraging student 
participation through regular attendance. 
Quality of teacher-student interactions: Teachers’ ability to impart information and encourage 
inquiry rests largely with the types of verbal and non-verbal interactions that they use to engage 
students. To be effective, such interactions create situations that allow students to apply their 
knowledge and not merely memorize facts. Teachers must also monitor learning to make certain 
that students assimilate information accurately and can use what they have learned. Permitting 
students to expand ideas together with providing feedback and explanation as needed are 
generally considered manifestations of these skills.
The structured observations of mathematics and language arts, described previously, were used 
to collect data on the quality of student-teacher interactions. The percentage of all interactions 
that involved explanation and feedback was used as the measure of teaching skill. As shown in 
Table 18, the percentage of interactions that included explanation or expansion of ideas has 
increase by about 10% over the life of the NHP project (from 13.4% to 22.5%). Feedback in the 
form of praise shows a positive increase in the first three years, a slight decline in 2002, and a 
16
5% increase in 2003. Feedback through punishment was similar for the four years and occurs in 
a small percentage of interactions. 
Table 18: Quality of Interactions
Context/Interaction 1999 2000 2001 2002 2003
Questions 37.3% 64.1% 48.3% 59.0% 58.1%
Expands 13.4% 7.3% 17.1% 23.0% 22.5%
Orders 40.6% 30.3% 38.5% 47.7% 51.7%
Dictates/Lectures 20.3% 18.1% 9.2% 3.2% 4.6%
Reinforces 2.9% 3.3% 8.2% 7.7% 12.9%
Punishes 1.5% 1.3% 3.2% 1.6% 1.6%
Questions and commands are the principal types of speech behaviors engaged in by teachers. 
They have increased over 2001 percentages. Dictation and lecturing decreased from 2001 to 
2002, but then increased slightly in 2003. Although explanation and feedback remain a small part 
of the quality of teacher’s speech acts, this again reflects attempts to engage students in the 
learning process. 
Use of materials: A principal focus of the project is on improving the availability and use of 
instructional materials. Both texts and supplementary instructional provide children with a 
channel for interacting with academic content on an ongoing basis. Often, however, it is 
assumed that children have books available and that teachers are trained in using instructional 
materials effectively. Teachers may lack practical experience in using texts and when working 
in a development situation may face overcrowded classrooms, children without books and little 
in alternative instructional resources. Thus, they resort to extensive lecture and use of the 
chalkboard. The purpose of this indicator is to confirm the provision of sufficient 
supplementary materials to classrooms of project schools to enrich the teaching and learning of 
literacy and numeracy. However, availability of materials alone is not an adequate measure, as 
students must use materials in order to enhance academic achievement. 
Use of materials was measured by three visual sweeps of the classroom during both mathematics 
and language arts lessons. During the sweeps, the number of available books and supplementary 
instructional materials and manipulatives were counted separately then the number actually in 
use was noted. The average number of materials available per child, as well as the average 
number of materials in use was calculated.
As shown in Table 19, both mathematics texts and supplementary materials such as 
manipulatives, and reading materials increased in the classrooms. This was in part due to the 
supplementary materials provided by NHP, which were present in a number of sample 
classrooms. However, in several schools these materials were found stored in the teacher’s office 
or in libraries rather than present in classrooms. The availability of reading materials increased 
to the extent that almost two texts per child, on the average, were observed to be readily 
available in the sample classrooms. 
17
Table 19: Availability and Use of Texts and Other Learning Materials
Subject Availability Use
1999 2000 2001 2002 2003 1999 2000 2001 2002 2003
Math .20 .40 .54 .69 .55 .25 .13 .18 .18 .13
Reading .40 .90 .91 1.73 .96 .27 .20 .13 .25 .15
The use of materials has remained low. Less that one in five children, on the average were 
observed to use mathematics texts or manipulatives during lessons. Similarly, only one in four 
students was observed to use reading/language arts materials during lessons in this area. This is 
especially alarming given the number of resources that NHP and the Ministry have made 
available in this area.
Mastery of NHP interventions: There is consensus in the international literature on educational 
innovation that mastery of new instructional approaches by teachers is a critical factor in 
adoption and sustainability. As NHP interventions were not yet in place when the formative 
evaluation was initiated in 1999, mastery was measured by asking teachers about the general 
objectives of the program. A second factor closely associated with mastery of the innovation is 
commitment to the new approach. This aspect of teaching skill was measured through a series of 
hypothetical questions in the teacher interview on circumstances that might deter a teacher from 
using an approach.
When asked about their knowledge of the NHP program, only about one-fifth of the teachers said 
it targets less successful students, and almost one-half mentioned strategies for using 
instructional materials. Since the main focus of NHP is on reading and math skills, the Table 20 
shows that in 1999, a little over one-third answered reading and math, about one-half the next 
year, over two-thirds in 2001, by 2002, the percentage dropped by about 10, but went up to about 
70% by 2003. A plausible explanation of the 2002 reduction in knowledge is that many of the 
teachers were reassigned and new teachers entered the NHP schools and by 2003 more had 
become familiar with the NHP interventions.
Table 20: Teacher Support of NHP
Year/Teacher Response 1999 2000 2001 2002 2003
Knowledge of NHP 36% 52% 72% 62.5% 70.2%
Use of Incentives 57% 70% 72% 79.2% 76.6%
Strategies for encouraging attendance: The purpose of this indicator is to measure the extent 
to which project activities impact absenteeism rates among students. Attendance was examined 
by student gender, as male attendance is traditionally lower than female attendance throughout 
the country. As official school attendance may run the risk of inflation or deflation, a correction 
factor of observed attendance recorded by the evaluation team was built into the measure. The 
key to the success of incentive programs will be their integration with the teaching-learning 
process; thus, classroom teachers are the appropriate source of information about incentives. 
Teachers were asked to list all of those incentives that they were using in their classrooms.
18
Up to 2002, there was an increase in the percentage of teachers using incentive strategies and a 
small decrease in 2003. In 1999, half of the teachers interviewed stated that they used incentives 
to increase attendance. In 2000, 70% of the sample described strategies used to encourage 
students to come to school. In 2001, 72% of the sample identified the incentives. In 2002, 79% 
of the sample identified incentives and in 2003 this had dropped to 77%, a very small decrease.
D. School Visits by NHP Specialists
In 2003, 38 of the 47 (81%) teachers interviewed reported that NHP specialists had visited their 
classrooms, and the table below shows the specific activities carried out by the project 
specialists. The data show that NHP observation of teaching was reduced considerably by 2002 
and 2003 and didactic training and the demonstration of new teaching methods have increased 
since 2001. 
Table 21: Teachers’ Recollection of NHP Specialists’ Activities
Activity Number Percent
2001 2002 2003 2001 2002 2003
Observe Teaching 29 28 28 91 58 60
Didactic Training 8 14 26 24 29 55
Demonstrate New Materials 2 9 5 6 19 11
Show how to use technology n/a n/a 1 n/a n/a 2
Demonstrate New Teaching 
Methods 8 18 16 24 38 34
E. System Support
In order to improve the success of children, teachers must be supported by an infrastructure at 
the school and national level. This includes support for professional development that will 
contribute to successful teaching and learning, effective management of the local learning 
institution to ensure that teachers can focus on teaching, and participation of community 
members in the education of their children.
1. Professional Development
Training to upgrade skills and knowledge is one of the main ways that a school system provides 
support for teachers. Such training can come about through in-service courses and workshops or 
through interaction with colleagues who have specialized knowledge in a particular subject area 
such as mathematics or language arts. This indicator establishes the number of teachers that 
have engaged in professional development activities as a consequence of their participation in 
New Horizons. The indicator takes into account training in Jamaica and abroad. Schools with 
resource teachers are also used as an indicator. All professional development activities are 
coordinated with the Professional Development Unit of the MOEC.
Table 22 shows the four-year trends of teachers’ participation in NHP training workshops and 
the percentage of schools that have resource teachers that provide in-service training and support 
19
for the implementation of NHP interventions. In May of 1999, none of the sample teachers had 
participated in NHP workshops, but by the 2001 evaluation, all of the sample teachers had 
participated. In 2002, the percent of the sample teachers who had participated had dropped to 
88%, and as mentioned above, this was due to teacher reassignments, and some of the new 
teachers had not yet participated in NHP training workshops. However, in 2003 the percentage 
who attended NHP workshops was once again up to 100%.
Table 22: NHP Professional Development
2. School management
Tracking of school resources and students is an important function of school management. Such 
tracking should be undertaken within a framework of specific objectives and activities. Thus, the 
utilization of school development plans in regard to NHP activities together with the utilization 
of the computer and accompanying administrative software, which can speed principals’ 
decision-making and ease reporting burdens, are the indicators of effective school management. 
Effectiveness of school boards is an additional indicator of school management. Measures for 
this aspect of management will be developed by the NCE.
As part of the NHP program, principals were asked to design development plans taking into 
consideration school needs, teacher training, curriculum design and parent/community 
involvement, especially as related to improving student literacy and numeracy. Among sample 
principals, 30% had completed this task at the time of 1999 formative evaluation data collection. 
Since most of those interviewed mentioned progress in completing the plans, it was expected that 
the number would increase rapidly. As can be seen from Table 26, all principals were 
implementing their development plans by May of 2000. 
Given that all of the sample schools had school development, a new indicator that was sensitive 
to implementation of the plans was developed. The new indicator is an index that measures 
whether or not schools are implementing activities in their SDPs related to literacy and numeracy 
by assigning the value of 1 to schools that are doing both, 0.5 to schools that are doing either 
literacy or numeracy, and 0 to schools that are doing neither; the sum of these values was then 
divided by the number of schools in the sample. The value of the index was .52 in 2001, .67 in 
2002 and dropped to .44 in 2003, reaching less than half the planned target for 2003. 
Professional Development/Year 1999 2000 2001 2002 2003
Teachers participate in Workshops 0% 85% 100% 88% 100%
Schools with Resource Teachers 15% 94% 100% 98% 100%
20
Table 23: NHP School Management
The percentage of schools with computers increased each year, and all NHP schools had 
computers in 2001 2002 and 2003. In 2001, ninety-four percent of the principals said that they 
had received a computer from NHP; in 2002, 96% of the principals said they had received a 
computer from NHP, and 2003 it was 100%. With the training provided by NHP, the use of 
computers for administration increased from 61% in 2001 to 88% in 2002 and dropped to 60% 
the following year. 
F. Community Involvement
The body of research on parent participation shows positive effects brought about by parental 
emphasis on literacy and other achievement in the home. As the focus of the project is on 
improved student learning, parental participation in learning is measured. In addition, parental 
participation in management is important to assure that schooling is relevant to community 
interests. Thus, the presence of parent-teacher associations and the frequency of their meetings 
are also indicators monitored through the formative evaluation. Other indicators, such as the 
number of schools with parent participation programs and training for parent and community 
leaders, will be monitored in partnership with the NCE.
Samples of NHP students were asked about parental involvement in their studies. In 1999, these 
interviews were conducted as part of the NHP school survey, whereas in 2000 and 2001, data 
were collected as part of the formative evaluation. Table 27 shows that there has been a slight 
increase each year from 1999 to 2001 in the number of students who stated that either their father 
or their mother assisted them in their reading, a 10% decrease in 2002, and another small 
decrease in 2003. When all family members are considered, in 2001, 94% of the children who 
said that they read at home did so with a family member. In 2002, the percentage had dropped to 
76% of the children in the sample, and in 2003 the percentage had dropped to 74.
As with the previous two years, all the schools in the 2002 sample reported having Parent￾Teacher organizations, but there was a 10% drop in the percent of PTAs that meet on a regular 
basis, and in 2003 the same percentage of school PTAs continued to meet regularly as in the 
previous year.
Professional Development/Year 1999 2000 2001 2002 2003
School Development Plan 30% 100% 100% 100% 100%
School Development Plan 
Implementation NA NA .52 .67 .44
Computer present 25% 68% 100% 100% 100%
Computer used for administration 0 20% 61% 88% 60%
21
Table 24: NHP Community Involvement
Year 1999 2000 2001 2002 2003
Parent Participation in Learning 36% 42% 54% 44.4% 42.9%
PTA present 89% 100% 100% 100% 100%
PTA meets regularly 33% 94% 94% 84% 84%
22
III. CONCLUSIONS AND IMPLICATIONS
The purpose of the study was to assess the progress made by the New Horizons in implementing 
activities that will lead to increased numeracy and literacy for students who have had limited 
success in school. The comparisons made from the baseline year of 1998, or in the case of the 
qualitative data 1999, with the results of the formative evaluation in subsequent years (2000, 
2001, 2002, and 2003) allow certain conclusions and implications to be drawn that can help to 
guide further implementation of the program.
A. Conclusions 
NHP has been most successful in improving the near mastery levels of Mathematics. 
NHP students have improved over the baseline in1998 in both third and sixth grade and 
the improvement has been greater than that for children in the system as a whole. 
However, NHP students have mean scores in Mathematics in 2003 similar to a matched 
comparison group of schools.
The percentage of NHP students reaching near mastery and mastery on the Mathematics 
GSAT increased 13.2% for girls and 7.7% for boys from the 1998 baseline to 2002. This 
compares to increases of 11.7% for girls and 10.4% for boys in non-NHP schools. In 
third grade, the percentages of students at near mastery and mastery were 1.7% lower for 
NHP girls and 8.7% lower for NHP boys. However, the changes from the 1998 baseline 
in GSAT mean scores were at least 5.5% higher for NHP girls and 1.3% for boys in 
relation to the matched comparison group.
Language Arts mastery appears to be a problem for the Jamaican primary education 
system as a whole. There is a general decrease in Language Arts performance in 2003 at 
both third and sixth grade levels. The percentage of children at mastery in third grade 
and sixth grade declined in Language Arts from 2002 to 2003. Near mastery levels were 
also generally lower on the GSAT in 2003. NHP students of both sexes in 2003 had a 
greater decline than their counterparts in the system as a whole. 
The success of NHP in improving student performance is questionable. NHP students 
have approximately the same means in Language Arts and Mathematics in 2003 as that of
similar schools without the NHP program. Furthermore, the overall increases in the mean 
scores have also been remarkably similar from 1998 to 2003. 
NHP has been successful in changing classroom environments so that they are organized 
to facilitate learning. Classroom environments improved each year in NHP schools. 
Children’s work was displayed to a greater extent, teachers were positive when 
interacting with students, and in many classrooms, there was an improvement in the 
organization of space.
23
Although some progress has been made in the 2002 school year, the participatory, child￾center classroom approaches, emphasized by the NHP program, have generally not been 
implemented in NHP classrooms.
In 2003, use of small group learning contexts decreased by 3% as compared to the 10% 
increase from 2001 to 2002. Furthermore, instructional delivery in NHP schools became 
even more traditional as teachers initiated 91.2% of the observed classroom interactions, 
representing a 4% increase over 2002. In other words, the majority of instruction 
continues to take place in teacher-directed large group context.
NHP has been highly successful in providing ancillary learning materials to school, but 
there has been a decrease from 2002 to 2003. This may mean that the bulk of the 
programmed books and materials have been delivered to the schools but were not 
available and visible at the time of the school visits by the NHP researchers in May. 
However, such materials are under-utilized in the classroom, and the trend has taken a 
downward turn from 2002 to 2003.
In 1999, the number of materials observed in the classroom was sufficient for only about 
20% of the students. In 2002, there are 1.7 language arts texts or materials for every 
student. Mathematics materials for almost 70% of the students were readily observable, 
but by 2003 three, the availability of both had dropped to 96% and 55% for reading and 
math, respectively. During lessons, such materials were actually used by about 15% of 
the students. 
The concentrated effort by NHP to provide hands-on professional development and other 
technical assistance at the school level has yet to show a significant impact on teacher 
behavior. Thirty four percent of the teachers interviewed stated that they had received 
training in new teaching methods from NHP specialists. Nevertheless, the number of 
interactions initiated by the teacher and the percent of interactions that took place in large 
groups both went up, showing a trend contrary to the student-centered active teaching 
methodologies that are an integral part of the NHP approach to pedagogy.
24
B. Implications 
The decline of language arts performance over three consecutive years is somewhat 
alarming. NHP might explore these tends at their training activities with teachers and 
principals. If funds are available, NHP might conduct a special study in both project and 
non-project schools with a decline and without a decline to determine the cause of the 
problem. Such a study might be conducted in collaboration with the evaluation unit of 
the Ministry.
The similarity of test performance between NHP students and students in matched 
comparison schools suggest that targeted efforts in schools serving those students who 
have had the least success in school, because of poverty and other factors, may require 
greater investment to make significant change. The slight difference in increased mastery 
and average mean scores generally favor NHP students, but they are of such small 
magnitude to question the cost-effectiveness of the project. NHP performance in relation 
to the comparison group should be monitored closely over the remaining life of the 
project.
The increased use of the participatory, child-centered methodologies, espoused by NHP 
and the new primary curriculum suggests that achieving behavior change in schools and 
classrooms is a long-term endeavor. It may be that significant changes will only be found 
as the project nears completion. However, the high percentage of traditional pedagogical 
practices bring in to question whether such change will be sufficient to improve student 
performance, beyond that related to general system improvement.
The overall increased availability of materials should be taken advantage of as part of the 
NHP technical assistance. If not already contemplated, workshops and technical 
assistance visits should focus on training teachers to effectively use materials.
Although the administrative infrastructure for improvement in learning appears to be in 
place and is an important achievement of the NHP project, it is not yet focused on 
supporting NHP objectives. The relatively low percentage of schools implementing 
activities related to language arts and mathematics improvement may require special 
training for teachers and administrators to make diagnosis of student performance and 
planning of strategies that will enhance student abilities in Mathematics and Language 
Arts and explicit part of the administrative process.
25
Appendix A: USAID Reporting Tables
26
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on language arts performance, allows comparison with national average. This is 
important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 boys’ GSAT Language Arts scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 80% of the grade 6 boys were in 
the “no mastery” group in Language Arts.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: 
Year Planned Actual
1998 17.0
1999 19.0 20.0
2000 22.0 35.0
2001 25.0 34.0
2002 30.0 29.5
2003 35.0 26.4
F. Other
Comments: The slight decline is consistent with a decline for the system as a whole. This decline is likely related to an 
increased number of students, who were formerly held back, taking the GSAT and scoring in the “no mastery” level.
27
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on language arts performance, allows comparison with national average. 
Important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 girls’ GSAT Language Arts scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 57% of the grade 6 girls were in 
the “no mastery” group in Language Arts.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: Year 2000 actual differs from previously reported percentages because of corrections made 
in the database
Year Planned Actual
1998 43.0
1999 45.0 47.0
2000 48.0 63.0
2001 52.0 57.0
2002 56.0 50.6
2003 60.0 47.9
F. Other
Comments: The slight decline is consistent with a decline for the system as a whole. This decline is likely related to an 
increased number of students, who were formerly held, taking the GSAT and scoring in the “no mastery” level.
28
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on mathematics performance, allows comparison with national average. This is 
important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 boys’ GSAT Mathematics scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 90% of the grade 6 boys were in 
the “no mastery” group in Mathematics.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively, are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: 
Year Planned Actual
1998 6.0
1999 7.0 10.0
2000 10.0 25.0
2001 13.0 26.0
2002 20.0 28.8
2003 30.0 21.2
F. Other
Comments: The slight increase is consistent with that for the system as a whole. This is likely related to an increased 
number of students, who were formerly held back, taking the GSAT and scoring in the “no mastery” level.
29
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on mathematics performance, allows comparison with national average. This is 
important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 girls’ GSAT Mathematics scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 85% of the grade 6 girls were in 
the “no mastery” group in Mathematics.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively, are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: 
Year Planned Actual
1998 14.0
1999 16.0 24.0
2000 18.0 41.0
2001 20.0 42.0
2002 25.0 45.2
2003 30.0 33.1
F. Other
Comments: The slight increase is consistent with that for the system as a whole. This is likely related to an increased 
number of students, who were formerly held back, taking the GSAT and scoring in the “no mastery” level.
30
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: 4.1 532-004 Improved Teaching Quality
A. Description
Precise Definition: Composite of: 1) content knowledge of students; 2) classroom learning environment; and 3) 
teaching for learning, aggregated across sample classrooms and expressed as values between 0 (minimum) and 1 
maximum
Unit of Measure: Index of third grade mastery levels – mathematics and language arts, score on classroom 
environment scale and percentage of child-initiated interactions, aggregated across sample classrooms.
Disaggregated By: Unnecessary
Management Utility: To track improvement in the quality of teaching over the life of the project.
B. Plan for Data Collection
Indicator: Index of Teacher Quality
Source: Student test data from Student Assessment Unit, observational data from formative evaluation of a stratified, 
random sample of NHP schools
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Aggregate each measure and average into an overall index of sample schools.
Presentation of Data: Index value between 0 – minimum and 1 – maximum in Tables of planned and actual 
performance
Review of Data: Review is performed by the SO team, the institutional contractor and other stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: Data collected by trained observers
Known Data Limitations: Diagnostic purposes of third grade tests results leading to lack of full reporting by schools. 
Actions Addressing Limits: The COP for the institutional contractor will ensure that adequate data are available prior to 
the R4.
E. Performance Data Table
Method of Calculations: Scores of three dimensions are averaged as an overall index
Key to Table: No key
Baseline & Target Notes: 
Year Planned Actual
1999 .43
2000 .50 .44
2001 .58 .48
2002 .65 .52
2003 .71 .47
 
F. Other
Comments: Planned levels have not been met owing to teachers’ continued use of traditional teacher-centered 
pedagogy.
31
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: 4.3 Improved Management of Schools
A. Description
Precise Definition: Number of schools implementing School Development Plan activities in literacy and numeracy; 
plus schools implementing activities in either literacy or numeracy; plus schools not implementing activities in these 
areas divided by the total number of schools in the sample
Unit of Measure: Weighted index where (L&N=1;L or N =.5; and other activities = 0)
Disaggregated By: Unnecessary
Management Utility: To measure the integration of project interventions with school activities.
B. Plan for Data Collection
Indicator: NHP schools implement SDP activities in numeracy and literacy
Source: Principals in a stratified, random sample of NHP schools
Data Collection: Yearly
Est. Cost:
Responsible Organization: Institutional contractor’s formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Weight responses, sum response categories, divide by number of sample schools using Excel or SPSS 
software.
Presentation of Data: Index value between 0 – minimum and 1 – maximum.
Review of Data: Review is performed by the SO team, the formative evaluation team and other stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: Data collected by trained interviewers
Known Data Limitations: None
Actions Addressing Limits: None
E. Performance Data Table
Method of Calculations: Weighted index of SDP implementation
Key to Table: None
Baseline & Target Notes: New indicator with 2001 as baseline year
Year Planned Actual
2001 .52
2002 .70 .67
2003 .90 .44
 
F. Other
Comments: This indicator was revised after 5-year targets were reached in 2 years with previous indicator.