New Horizons for Primary Schools/Jamaica
FORMATIVE EVALUATION
November 2001
Prepared for:
United States Agency for International Development
Jamaica
Project Undertaken by:
Juárez and Associates, Inc.
Prepared by
Ray Chesterfield, Kjell Enge, 
and Heather Simpson 
Contract No. NHA-532-C-00-98-12345-00 Juárez and Associates, Inc.
b
Table of Contents
Executive Summary…………………………………………………………………………i
I. INTRODUCTION………………………………………………………………………. 1
A. Background…………………………………………………………………….. 1
B. Methodology……………………………………………………………………
2
1. Indicators………………………………………………………………. 2
2. Design………………………………………………………………….. 2
C. Assumptions……………………………………………………………………. 4
II. Findings………………………………………………………………………………...
5
A. Student Performance…………………………………………………………… 5
1. Mathematics……………………………………………………………. 5
a. Third Grade…………………………………………………... 5
b. Sixth Grade……………………………………………………7
2. Language Arts………………………………………………………….. 9
a. Third Grade…………………………………………………... 9
b. Sixth Grade……………………………………………………10
B. Teaching Quality………………………………………………………………..12
C. Teaching Skills………………………………………………………………….14
D. School Visits by NHP Specialists……………………………………………… 17
E. System Support………………………………………………………………... 20
1. Professional Development………………………………………………20
2. School Management……………………………………………………. 20
3. Community Involvement………………………………………………. 22
III. Conclusions and Implications…………………………………………………………. 24
A. Conclusions…………………………………………………………………….. 24
B. Implications…………………………………………………………………….. 27
Appendix A: Indicators and Measures……………………………………………………..
28
Appendix B: USAID Reporting Tables…………………………………………………… 32
c
List of Tables
Table 1: Change in Near Mastery on Third Grade Diagnostic Mathematics Test in NHP and 
non-NHP Schools by Gender and Year…………………………………………………… 6
Table 2: Change in Mastery on Third Grade Diagnostic Mathematic Test in NHP and non￾NHP Schools by Gender and Year………………………………………………………… 6
Table 3: Change in Mastery and Near Mastery on Third Grade Diagnostic Mathematics Test in 
NHP and non-NHP Schools by Gender and Year………………………………………… 7
Table 4: Change in Near Mastery on GSAT Mathematics in NHP and non-NHP Schools by 
Gender and Year…………………………………………………………………………… 7
Table 5: Change in Mastery on GSAT Mathematics in NHP and non-NHP Schools by Gender 
and Year…………………………………………………………………………………… 8
Table 6: Change in Near Mastery and Mastery on GSAT Mathematics in NHP and non-NHP 
Schools by Gender and Year……………………………………………………………… 8
Table 7: Change in Near Mastery on Third Grade Diagnostic Language Arts Test in NHP and 
non-NHP Schools by Gender and Year…………………………………………………… 9
Table 8: Change in Mastery on Third Grade Diagnostic Language Arts Test in NHP and non￾NHP Schools by Gender and Year………………………………………………………… 9
Table 9: Change in Mastery and Near Mastery on Third Grade Diagnostic Language Arts Test 
in NHP and non-NHP Schools by Gender and Year……………………………………… 10
Table 10: Change in Near Mastery on GSAT Language Arts in NHP and non-NHP Schools by 
Gender and Year…………………………………………………………………………… 10
Table 11: Change in Mastery on GSAT Language Arts in NHP and non-NHP Schools by 
Gender and Year…………………………………………………………………………… 11
Table 12: Change in Near Mastery and Mastery on GSAT Language Arts in NHP and non￾NHP Schools by Gender and Year………………………………………………………… 11
Table 13: GSAT Means and Standard Deviations 1998-2001…………………………………... 12
Table 14: Mean Classroom Environment Scores by School Size………………………………. 13
Table 15: Interactions Initiated by Teachers and Students……………………………………... 14
Table 16: Interactions by Classroom Context…………………………………………………… 14
Table 17: Quality of Interactions……………………………………………………………….. 15
Table 18: Availability and Use of Texts and Other Learning Materials………………………... 16
Table 19: Teacher Support of 
NHP…………………………………………………………….. 16
Table 20: NHP Schools- Specialists Activities………………………………………………….. 17
Table 21: Teachers’ Recollection of NHP Specialists’ Activities 2001………………………… 18
Table 22: Correlations of Changes in the Levels of Mastery—2000-2001 with NHP Specialist 
Activities in Schools……………………………………………………………………….. 19
Table 23: NHP Professional Development……………………………………………………… 20
Table 24: NHP School Management…………………………………………………………….. 21 
Table 25: Mathematics Mastery Levels on GSAT by School Principal Designation…………… 22
Table 26: Language Arts Mastery Levels on GSAT by School Principal Designation…………. 22
Table 27: NHP Community Involvement……………………………………………………….. 23
USAID Reporting Tables…………………………………………………………….………… 32
NHP Grade 6 Boys’ GSAT Language Arts Scores…………………………….…………. 33
d
NHP Grade 6 Girls’ GSAT Language Arts Scores……………………..……….………… 34
NHP Grade 6 Boys’ GSAT Mathematics Scores…………………………….……………. 35
NHP Grade 6 Girls’ GSAT Mathematics Scores………………………..………………… 36
Index of Teacher Quality…………………………………………………..………………. 37
School Development Plan Activities in NHP Schools…………………………………….. 38
i
EXECUTIVE SUMMARY
Introduction
This document summarizes the results of the 2001 formative evaluation of the New 
Horizons for Primary Schools (NHP) Project in Jamaica. NHP is a five-year effort to improve 
the Mathematics and Language arts of Jamaican primary school students, who because of 
poverty or other factors have had little success in school. The project is a partnership between 
the Jamaican Ministry of Education, Youth and Culture, USAID, and the NHP technical 
assistance contractor, Juárez and Associates.
The formative evaluation is conducted yearly near the end of the school year. It is 
designed to inform the implementation of NHP interventions and thereby permit NHP staff to 
target interventions in critical areas of the program. The formative evaluation process also 
serves to measure project results from an established baseline, which will contribute to the 
measurement of final project results. In 2001, the formative evaluation had the additional 
purpose of building the capacity of Jamaican Education professionals in systematic qualitative 
data collection and the integration and interpretation of qualitative and quantitative data.
Evaluation Methodology
A team of 11 Jamaican education professionals carried out the data collection for the 
evaluation. They employed a multi-method design, consisting of inventories, checklists, 
classroom observation forms, and focused interviews, to measure the conditions in place for 
effective learning in NHP classrooms. A stratified sample of 19 schools, or 26% of the 72 NHP 
schools served as the data source for the evaluation. Observational data were complemented by 
the results of the third grade diagnostic tests and the sixth grade GSAT results for 2001. 
Evaluators were trained in two workshops dealing with qualitative data collection and data 
reduction, analysis and interpretation, respectively. The evaluation took place in May of 2001.
Principal Findings
Language Arts mastery levels among NHP students in the upper primary grades have improved 
as a result of the project. The percentage of NHP students reaching at least near mastery on the 
Language Arts GSAT increased 14.8% for girls and 16.7% for boys from the 1998 baseline to 
2001. This is greater than the increases of 11.5% for girls and 16.3% for boys in non-NHP 
schools. 
NHP has not performed as well as the primary school system in general in improving mastery 
levels of sixth grade students in Mathematics. The percentage of NHP students reaching at least 
near mastery has risen by 24.8% for girls and 20.2% for boys in Mathematics. However, the 
changes were not as great as those of 28.8% for girls and 24.6% for boys in non-NHP schools.
NHP has had the least success in improving the performance of children in the early primary 
grades. In Language Arts, NHP third graders have had a decrease in the combined near mastery 
ii
and mastery levels of -3.6% for girls and –1.1% for boys, since the 1998 baseline. In 
Mathematics the increases for both boys (+13.5%) and girls (+16.5) are less than that for girls 
(20.6%) and boys (20.3%)in non-NHP schools.
Despite positive gains, the percentage of NHP students who master the curriculum is low. This 
low success rate was exacerbated by poor performance of the entire primary system in 2001. 
Only about one-third of NHP girls and one-fourth of boys exhibit even near mastery on the 
GSAT. Less than half of third grade students have mastered the Language Arts curriculum and 
only about 35% achieve mastery in Mathematics. In both NHP and non-NHP schools, the 
percentage of children reaching mastery in Language Arts declined by at least 4.2% from 
the2000 school year. In Mathematics, the decline was 0.5%, or more.
NHP has been successful in changing classroom environments so that they are organized to 
facilitate learning and in providing ancillary learning materials. However, pedagogical 
approaches that emphasize participation of the child in a variety of learning opportunities have 
not been implemented. Instructional delivery in NHP schools remains highly traditional. 
Teachers initiate more than 90% of the interactions with children, allowing little opportunity for 
self-expression or expansion of ideas. Although there are sufficient reading materials for 90% of 
the students, and mathematics materials for over half of the students, such materials were 
actually used by less that 20% of the students. 
The concentrated effort by NHP to provide hands-on professional development and other 
technical assistance at the school level has yet to show an impact on student performance or 
teacher behavior. No significant correlations were found between the number, type or level of 
effort of activities carried out by NHP professionals in individual schools and the change in 
mastery levels for schools.
At the school level, NHP has been successful in creating an environment to support improved 
learning of Mathematics and Language Arts. Every indicator of system support has had a 
positive change from the 1999 baseline year and most were 100% or nearly 100% implemented.
Implications 
 NHP specialists should review their activities to determine interventions that are contributing to 
improved Language Arts performance in the upper grades and examine the applicability of such 
strategies to earlier grades, and perhaps to Mathematics. One possibility is that greater use is 
being made of ancillary reading materials provided by the project. 
NHP teachers’ lack of use of the participatory, child-centered methodologies, espoused 
by NHP and the new primary curriculum argues for a change in the professional development 
strategies employed by the NHP project. More direct demonstration and modeling with teachers 
when specialists visit the schools may be necessary to change classroom behavior. Given the 
small number of NHP professional staff, their expertise may have to be supplemented by 
additional subject matter or pedagogical specialists.
iii
The administrative infrastructure for improvement in learning appears to be in place and 
is an important achievement of the NHP project. Given the limited improvement in student 
performance, however, may require special training for teachers and administrators to make 
diagnosis of student performance and planning of strategies that will enhance student abilities in 
Mathematics and Language Arts and explicit part pf the administrative process.
The Jamaican primary education system, of which NHP is a part, is not yet producing the 
types of graduates that the society desires. Although there have been improvements, the 
percentage of sixth grade students with mastery of the curriculum is less than 35% in Language 
Arts and fewer than 20% the Mathematics. The monitoring carried out under the formative 
evaluation has coincided not only with the implementation of NHP but also with the launching of 
a new primary school curriculum. It may be that when the curriculum is fully implemented in all 
schools, that improvement in student performance will be accelerated. 
1
I. INTRODUCTION
This document describes the results of the third year of formative evaluation of the New 
Horizons for Primary Schools (NHP) Project. The evaluation is carried out near the end of the 
Jamaican school year (May-June) to provide a barometer of the progress of the project on a 
series of indicators. (A list of indicators and measures used in this evaluation are found in 
Appendix A of this report). Originally, the formative evaluation had two purposes. First, the 
formative evaluation results inform the implementation of NHP interventions and permit NHP 
staff to target interventions in critical areas of the program. The results complement those of 
ongoing assessments of the implementation process undertaken informally through school visits, 
feedback on professional development efforts and periodic communication with school 
administrators and teachers. Second, the formative evaluation process serves to measure project 
results from an established baseline. As it provides systematic monitoring of performance over 
time, formative evaluation contributes to the measurement of final project results. (Baseline 
indicators and projections of change over time derived from the 1999 formative evaluation are 
found in Appendix B of this report). 
In 2001, the formative evaluation had an additional purpose. In order to respond to the 
capacity building interests of the Ministry of Education, a workshop on evaluation methodology 
was held for technicians in the Ministry of Education, members of local teachers’ colleges, and 
New Horizon Project personnel. The workshop dealt with observation and interview techniques 
to measure progress toward NHP objectives. Dr. Ray Chesterfield and Dr. Kjell Enge, who are 
experienced education evaluators, conducted the workshop. These individuals also carried out 
the data collection in the first two years of the formative evaluation effort. Following the 
training, a team of 11 of the workshop participants collected data from a sample of NHP primary 
schools. A follow-up workshop on data analysis and interpretation was held subsequent to the 
data collection, entry, and cleaning.
A. Background 
The primary objective of New Horizons for Primary Schools (NHP) is to enhance the 
performance of Jamaican primary school students in numeracy and literacy. The focus of the 
technical assistance component of the project is on those children who, because of poverty and a 
lack of other enabling conditions, have had little academic success in school. Increased 
academic success is to be accomplished through the development of model interventions that, 
when tested, can be used to improve the performance of low-achieving children throughout 
Jamaica. Thus, the products of the contractor’s work are changes in schools and classrooms that 
result in individual students having greater academic success in primary school. Such results 
include measurement of the indicators for the USAID strategic objective. 
Systems, such as computerized administrative and student tracking systems, are also 
being implemented over the life of NHP. These systems are to assist schools in monitoring their 
own performance. The results of such individual school monitoring can be aggregated to 
examine project performance. Similarly, NHP is integrating MOEC databases to provide 
additional data sources for monitoring performance. Until such systems are fully operational, 
2
however, monitoring is being carried out as part of the formative evaluation effort designed to 
provide feedback to program technicians implementing the interventions. As formative 
evaluation requires in-depth data collection, a representative sample of NHP schools is selected 
each year for evaluation purposes.
Many of the indicators for monitoring performance are complex concepts that require the 
combination of qualitative and quantitative methods to measure accurately. The following pages 
discuss the procedures used to collect data on NHP indicators. Subsequent chapters present the 
findings of the formative evaluation, in terms of change from the baseline data, and provide 
conclusions and implications drawn from these findings.
B. Methodology 
1. Indicators
The indicators are taken largely from the U. S. literature on school/classroom 
effectiveness and on the growing body of international literature on classroom interaction and 
educational quality (see attached bibliography for examples). Three levels of indicators were 
used. The first relates to student performance in terms of mastering the curriculum. The second 
consists of indicators of teacher performance that are generally associated with greater quality in 
terms of students’ academic performance. Third, are there indicators of system support or 
enabling factors such as efficient school management, professional development opportunities 
for teachers, and parent participation in the education of their children, that must be in place to 
improve the performance of individual children?
2. Design
A multi-method design, consisting of inventories, checklists, classroom observation 
forms, and focused interviews, was employed to measure the conditions in place for effective 
learning. This design allows for the measurement of the impact of the interventions 
implemented to improve learning, especially among students who have had limited success in 
school. Evaluation efforts focused on both females and males. This is important not only to 
ensure that initiatives are equitable but also to identify initiatives and strategies that are 
successful regardless of gender. 
Sample. A stratified sample of 26% of project schools was drawn from the universe of 
72 schools. Schools were stratified by size (small, medium, or large) and type (primary or all 
age) then randomly selected within strata. As the focus of the project is a “ground-up” approach 
that begins with needs identified by participating schools, those schools that had been most 
involved in NHP activities during the year were over-sampled. The final sample consists of 19 
schools and 32 classrooms for intensive data collection and analysis.
The focus of the formative evaluation was on third grade. The purpose of the formative 
evaluation was to obtain in-depth, systematic data, in a limited amount of time. Thus, it 
concentrated on one grade as an indicator of general progress. Third grade was chosen, because 
3
there are test scores available that allow greater diagnostic ability and permit the monitoring of 
change in the cohort of third graders serving as the baseline over the life of the project. This is 
important because both the1998 and 1999 NAP scores suggest that NHP children fall behind 
principally between third and sixth grade.
In the first two years of the evaluation, first, second, fourth, fifth and sixth grade 
classrooms were also observed. The data from these classrooms showed the same general 
patterns as those for the sample as a whole. This suggests that for monitoring purposes, third 
grade results can be used as a general indicator of progress.
Instruments. Instruments included classroom maps, materials inventories, classroom 
observation forms, classroom environment assessments, and interviews guides for use with 
teachers, students and school principals. Maps were employed to identify children and to 
examine the context in which they interact with teachers. Materials inventories measured both 
the presence and use of all materials at different times during math and language arts lessons. 
Observational sweeps were made at three points in time during each academic context. At each 
sweep, the number of books and ancillary materials available and in use, were counted. 
Classroom interaction was measured through a teacher-student interaction protocol. This 
instrument focused on teachers' interactions with individual students and the nature of those 
interactions in different academic classroom activities. In order to ensure consistency and 
control for contemporaneous events that might influence behavior patterns, the form was used 
for ten minutes at four different times during the instructional day in third grade classrooms. 
Two observations took place during mathematics lessons and two during language arts. Thus, a 
behavioral sample of 20 minutes for each of the target content areas was created. Researchers 
used the classroom environment instrument to rate the appropriateness of the classrooms for 
child-centered learning. 
Teachers’ perceptions of the interventions, as well as their mastery of and commitment to 
the new approaches implemented under NHP, were tapped by a teacher interview schedule. 
Similarly, changes in the school management planning and systems were measured through an 
interview with the principal. Students were queried about activities in the home and involvement 
of parents in the children’s reading.
Fieldwork Procedures. A schedule of school visits was developed with the field 
workers, and NHP staff contacted the principals and informed them of the visits. Two of the 
backstop personnel for the institutional contractor assisted a local researcher coordinator in 
scheduling and supervising the fieldwork. Fieldworkers synchronized observations through 
training exercises during the workshop. This training included exercises with the instruments 
using videotapes of classroom interaction in schools to ensure consistency in observations and 
interviewing. Parallel observations were conducted with the instruments until an inter-observer 
agreement coefficient of above .70 was reached for all observational instruments. 
The researchers worked in small teams of up to four people and spent up to one full day 
at each school collecting data. Procedural guides and operational definitions were attached to 
specific instruments as references to ensure consistency in field procedures during the 
4
investigation. Following each day of fieldwork, the coordinator gathered the instruments from 
the teams and the backstop personnel monitored the quality of the data collection and entered the 
information into SPSS spreadsheets. Eighteen schools were visited and complete sets of data 
were collected from 30 classrooms.
 
Data Analysis. The principal unit of analysis was the classroom. As the interventions 
are focused largely on improving teaching, it is changes in classroom-level environments and 
behaviors that affect student learning. Data analysis consisted of calculating the absolute and 
relative frequencies of each behavioral indicator and making comparisons across the three 
evaluation years. Differences by types of schools were also examined. Special indices were 
created to measure complex issues such as teaching quality. Where appropriate, statistics such as 
chi-square and correlations were used to examine relationships among the sample.
C. Assumptions 
The ongoing formative evaluation is based on several assumptions. First, the school and 
the class are the key units of analysis in planning and intervening to improve the quality of 
learning. Second, the school is a social system and the interaction of all of the elements within a 
school has an influence on student learning beyond that provided individually by inputs to the 
school. This is not to suggest that the uniqueness of each school makes aggregate measurement 
impossible, but rather that accurate measurement of the impact of schooling is a complex 
undertaking requiring the integration of a variety of data collection approaches.
5
II. FINDINGS
A. Student Performance 
Jamaica is promoting pupil-centered “everyone can learn” concept of teaching rather than 
a norm-based “cream of the crop” approach. Thus, the focus is shifting to all children’s mastery 
of the curricular content. This means that the array of individual scores will shift from the 
normal distribution or “bell shaped curve” associated with a norm-based assessment and mean 
scores, toward a “J-curve” with a few students falling at the low end and the middle and most 
scores reflecting a high degree of learning. However, with the current inverse J-curve, the first 
step is to move students to “near mastery” levels. The formative evaluation originally examined 
both third and sixth grade mastery. Thus, the NAP and Student Assessment Unit criteria of less 
than 50% of the items in each domain correct as “no mastery” level, was used in the evaluation. 
Although NAP does not designate mastery levels for the sixth grade GSAT, the criteria used at 
the third grade level was employed in determining student progress (less than 50% correct = “no 
mastery,” 50% to 75% correct = “near mastery” and above 75% = “mastery”. 
It has proved somewhat difficult to obtain complete data sets of either NHP or non-NHP 
third grade tests, owing to their diagnostic purpose, which leads schools not to report results. 
Thus, the USAID strategic objective team will use only sixth grade in their reporting. The 
formative evaluation will continue to include third grade tests when they become available, 
through update reports. Both third grade and results and sixth grade results for 2001 are included 
in this report. All test results are reported in relation to 1998, the baseline year.
1. Mathematics
a. Third Grade
Table 1 shows the change in the percentage of children reaching near mastery of the third 
grade mathematics curriculum, as measured on the diagnostic test for that subject. Changes in 
student performance in NHP schools are compared to all primary schools not participating in the 
NHP program. Both yearly change and total change from the baseline are provided. As can be 
seen, there is a small overall change for NHP from 1998 to 2001. NHP children have made 
relatively greater gains in test performance in reaching near mastery than Jamaican third grade 
school children as a whole, and this is true for both boys and girls. However, in the 2000 school 
year, there was a decline in third grade near mastery performance for all groups of children and 
NHP children had greater declines than their counterparts. Girls in the 2001 NHP third grade 
population made up the gap on the non-NHP population, whereas NHP boys are 1.7% lower than 
non-NHP boys.
6
Table 1: Change in Near Mastery on Third Grade Diagnostic Mathematics Test in NHP 
and non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change by 
Year
Non￾NHP
Change by 
Year
NHP Change by 
Year
Non￾NHP
Change by 
Year
1998 37.9 43.0 28.8 33.8
1999 45.1 +7.2 45.0 +2.0 37.0 +8.2 38.5 +4.7
2000 38.0 -7.1 43.0 -2.0 29.0 -8.0 35.0 -3.5
2001 41.8 +3.8 41.0 -2.0 34.8 +5.8 36.5 +1.5
Change from 
Baseline
+3.9 -2.0 +6.0 +2.7
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
Lower near mastery levels may be the result of greater numbers of the third grade 
population reaching mastery. This is shown both by the percentage of children in the mastery 
category in subsequent years and by the total percentage of children in the mastery and near 
mastery categories. Ideally, all children will be in the mastery category. Table 2 shows that a 
greater percentage of NHP third graders have mastered the curriculum than in the 1998 baseline 
year. However, they have made less improvement than non-NHP children of both genders and 
the gap has increased. Again there is a decline from 1999 to 2000 among all children.
Table 2: Change in Mastery on Third Grade Diagnostic Mathematic Test in NHP and 
non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change by 
Year
Non￾NHP
Change by 
Year
NHP Change 
by 
Year
Non￾NHP
Change by 
Year
1998 9.3 12.7 4.1 7.9
1999 19.4 +10.1 28.0 +15.3 11.8 +7.7 19.5 +11.6
2000 18.0 -1.4 24.0 -4.0 9.0 -2.8 15.0 -4.5
2001 21.9 +3.9 35.3 +11.3 11.4 +2.4 25.3 +10.3
Change 
from 
Baseline
+12.6 +22.6 +7.3 +17.4
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
Table 3 shows that there has been substantial improvement in third grade children’s 
performance in mathematics from the 1998 baseline. However, non-NHP children have had 
greater success than NHP children. This is largely a result of NHP children failing to overcome 
the decline from 1999 to 2000. Non-NHP third graders, on the other hand, are at higher levels 
than similar third graders in 1999.
7
Table 3: Change in Mastery and Near Mastery on Third Grade Diagnostic Mathematics 
Test in NHP and non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change by
Year
Non￾NHP
Change by 
Year
NHP Change 
by Year
Non￾NHP
Change 
by Year
1998 47.2 55.7 32.9 41.5
1999 64.5 +17.3 73.0 +17.3 48.8 +15.9 58.0 +16.5
2000 56.0 -8.5 67.0 -6.0 38.0 -10.8 50.0 -8.0
2001 63.7 +7.7 76.3 +9.3 46.2 +8.2 61.8 +11.8
Change 
from
Baseline
+16.5 +20.6 +13.3 +20.3
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
b. Sixth Grade
Table 4 shows the change in the percentage of children reaching near mastery of the sixth 
grade mathematics curriculum, as measured on the GSAT test for that subject. Changes in 
student performance in NHP schools are compared to all primary schools not participating in the 
NHP program. As can be seen, there is significant change from 1998 to 2001. NHP children 
have made relatively greater gains in reaching near mastery than Jamaican primary school 
children as a whole, and this is true for both boys and girls. However, in the 2000-2001 school 
year, NHP children made smaller gains than their counterparts. Neither gender has completely 
made up the gap on the non-NHP population, as about 5% less NHP children of each gender has 
reached near mastery. The general decline in mastery levels found for the third grade from 1999 
to 2000 did not occur at the near mastery level in sixth grade.
Table 4: Change in Near Mastery on GSAT Mathematics in NHP and non-NHP Schools 
by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by 
Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change by 
Year
1998 13.9 26.5 5.5 13.8
1999 22.2 +8.3 31.9 +5.4 8.8 +3.3 17.3 +3.5
2000 30.0 +7.8 31.8 -0.1 19.2 +10.4 22.9 +5.6
2001 32.2 +2.2 36.8 +5.0 20.6 +1.4 25.9 +3.0
Change from 
Baseline
+18.3 +10.3 +15.1 +12.1
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
NHP has been less successful in moving children to mastery than to near mastery. 
Although there has been a positive increase among NHP children of both genders in each year of 
8
monitoring test performance, gains in the number of children reaching mastery have been lower 
than those of other Jamaican sixth graders. The increase in the percentage of children reaching 
mastery in non-NHP schools is double that in NHP schools. There is a slight decline in for all 
groups from 2000 to 2001.
Table 5: Change in Mastery on GSAT Mathematics in NHP and non-NHP Schools by 
Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 0.3 2.9 0.2 2.3
1999 1.7 +1.4 6.9 +4 0.8 +0.6 4.0 +1.7
2000 10.9 +9.2 22.8 +15.9 5.7 +4.9 15.5 +11.5.
2001 9.8 -1.1 21.5 -1.3 5.2 -0.5 14.8 -0.7
Change from 
Baseline
+9.5 +18.6 +5.0 +12.5
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
The change in children at near mastery and mastery has been more than a fourth of the 
sixth grade primary school population, if it is assumed that all sixth graders take the GSAT. The 
general population has shown greater relative change in the percentage of children in the 
combined near-mastery/mastery group because of the greater number of children moving to 
mastery. The relative difference in NHP girls and the general sixth grade population has 
decreased from 15.2% in 1998 to 1% in 2001 and among boys from 10.4% to 4.4%. It should be 
noted, however, that only a quarter of NHP boys are above the near mastery level, despite these 
gains.
Table 6: Change in Near Mastery and Mastery on GSAT Mathematics in NHP and non￾NHP Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 14.2 29.4 5.7 16.1
1999 23.9 +9.7 38.8 +9.4 9.6 +3.9 21.3 +5.2
2000 40.9 +17.0 54.6 +15.8 24.9 +15.3 38.4 +17.1
2001 42.0 +1.1 58.3 +3.7 25.8 +0.9 40.7 +2.3
Change from 
Baseline
+27.8 +28.9 +20.1 +24.6
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
9
2. Language Arts
a. Third Grade
Tables 7, 8 and 9 show that improvement in students’ mastery of third grade Language 
Arts curriculum has been difficult for NHP to achieve. The percentage of both NHP and non￾NHP children reaching near mastery has declined from 1998. As mentioned, such a decline may 
be the result of a greater percentage of students reaching mastery. This appears to be the case 
among non-NHP third graders. As shown in Table 8, the increase in mastery levels is greater 
overall than the decline in near mastery for non-NHP students. This is also reflected in the 
combined near mastery and mastery levels, where non-NHP students show a net increase from 
the 1998 baseline. NHP students, on the other hand, have an increase in mastery levels from the 
baseline, but this does not surpass the decline in near mastery. Similarly, there is a net decrease 
in the combined near mastery and mastery levels for NHP students.
Table 7: Change in Near Mastery on Third Grade Diagnostic Language Arts Test in 
NHP and non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by 
Year
NHP Change 
by 
Year
Non￾NHP
Change 
by 
Year
1998 46.9 40.7 37.8 40.0
1999 42.0 -4.9 34.6 -6.1 37.9 +0.1 34.8 -5.2
2000 42.0 0 39.0 +4.4 34.0 -3.9 37.0 +2.2
2001 36.7 -5.3 32.4 -6.6 33.3 -0.7 32.9 -4.1
Change from 
Baseline
-10.2 -8.3 -4.5 -7.1
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
Table 8: Change in Mastery on Third Grade Diagnostic Language Arts Test in NHP and 
non-NHP Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by 
Year
Non￾NHP
Change 
by 
Year
NHP Change 
by 
Year
Non￾NHP
Change 
by
 Year
1998 26.2 37.7 13.5 21.9
1999 31.1 +4.9 46.1 +8.4 16.5 +3.0 29.0 +7.1
2000 28.0 -3.1 38.0 -8.1 13.0 -3.5 23.0 -6.0
2001 32.8 +4.8 48.5 +10.5 16.9 +3.9 33.1 +10.1
Change from 
Baseline
+6.6 +10.8 +3.4 +11.2
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
10
Table 9: Change in Mastery and Near Mastery on Third Grade Diagnostic Language 
Arts Test in NHP and non-NHP Schools by Gender and Year
Year Third Grade Female Third Grade Male
NHP Change 
by 
Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 73.1 78.4 51.3 61.9
1999 73.1 0 80.7 +2.3 54.4 +3.1 63.8 +1.9
2000 70.0 -3.1 77.0 -3.7 47.0 -7.4 60.0 -3.8
2001 69.5 -0.5 80.9 +3.9 50.2 +3.2 66.0 +6.0
Change from 
Baseline
-3.6 +2.5 -1.1 +4.1
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
b. Sixth Grade
Change in language arts performance at the near mastery level follows a similar trend to 
that for mathematics among NHP students. There are consistent gains each year and the relative 
gain in children reaching near mastery is greater for NHP children than for their counterparts. 
Boys in the general population follow a pattern similar to NHP children. Non-NHP girls, 
however, show an overall drop from the baseline year.
Table 10: Change in Near Mastery on GSAT Language Arts in NHP and non-NHP 
Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 34.2 39.6 15.2 23.5
1999 37.3 +3.1 42.3 +2.7 17.3 +2.1 25.3 +1.8
2000 36.8 -0.5 33.7 -8.6 22.4 +5.1 25.0 -0.3
2001 38.7 +1.9 37.8 +4.1 25.6 +3.2 27.8 +2.8
Change from 
Baseline
+4.5 -1.8 +10.4 +4.3
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
Change in the percentage of students reaching mastery also has a similar overall trend to 
that found for mathematics. Greater relative numbers of non-NHP students have reached the 
mastery level than non-NHP students. The overall increases are, however, somewhat lower than 
for mathematics. This appears to be the result of something occurring in the system, during the 
2000-2001 school year, as the drop in children reaching mastery was found across all groups. 
11
This may have been the result of adjustments in the tests or testing procedures, given the 
consistency in the drop.
Table 11: Change in Mastery on GSAT Language Arts in NHP and non-NHP Schools by 
Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 8.3 20.4 2.1 8.2
1999 9.3 +1 18.2 -2.2 1.6 -0.5 7.4 -0.8
2000 26.6 +17.3 39.3 +21.1 12.6 +11.0 24.6 +17.2
2001 18.6 -8.0 33.7 -5.6 8.4 -4.2 20.2 -4.4
Change from 
Baseline
+10.3 +13.3 +6.3 +12.0
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
As shown in Table 12, NHP children of both genders have had greater overall relative 
gains in reaching near mastery or mastery than the remaining sixth grade population. However, 
there has not been the same success in closing the original gap between NHP students and the 
general population in language arts as was found in mathematics. The difference in the relative 
combined change goes from 17.5% among girls in 1998 to 14.2% in 2001. Among boys, the gap 
goes from 14.4% to 14%. The negative change found in children reaching mastery in 2000-2001 
is reflected in the relative drops in the combined percentages across all groups.
Table 12: Change in Near Mastery and Mastery on GSAT Language Arts in NHP and 
non-NHP Schools by Gender and Year
Year Sixth Grade Female Sixth Grade Male
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
NHP Change 
by
 Year
Non￾NHP
Change 
by
 Year
1998 42.5 60.0 17.3 31.7
1999 46.6 +4.1 60.5 +0.5 19.9 +2.6 32.7 +1.0
2000 63.4 +16.8 73.0 +12.5 35.0 +15.1 49.6 +16.9
2001 57.3 -6.1 71.5 -1.5 34.0 -1.0 48.0 -1.6
Change from 
Baseline
+14.8 +11.5 +16.7 +16.3
Source: NAP 1998 and 1999 database, Student assessment Unit 2000 and 2001 database
In order to examine possible reasons for the decline in mastery, means scores and 
standard deviations were examined for the four GSAT application years. As can be seen from 
Table 13, after three years of mean gains, scores decline abruptly in 2001. As the student 
assessment unit stated that tests were very similar, an explanation for lower scores would be that 
more children who scored at the lower end took the test in 2001. This is possible since students 
12
are only allowed to take the GSAT once and as parents this fact more children are likely to take 
the test. As 1000-2000 more children took the test in 2001, and standard deviations are 
somewhat larger, a greater dispersion of scores may be somewhat responsible for the lower 
overall mean scores, and decreases in mastery levels.
Table 13: GSAT Means and Standard Deviations 1998-2001
1998 1999 2000 2001 Year/Stats
Maths L. Arts Maths L. Arts Maths L. Arts Maths L. Arts
NHP F M F M F M F M F M F M F M F M
Mean 33.2 26.4 46.6 34.7 39.1 32.4 48.1 35.5 45.6 37.8 57.8 44.9 46.1 37.6 53.5 41.8
N 1780 1804 1737 1819 1810 1609 1809 1614 1790 1727 1788 1728 1928 2059 1928 2059
SD 14.7 13.0 19.1 15.7 15.3 12.5 19.2 16.0 20.6 19.2 21.1 20.7 21.4 20.1 22.2 21.5
Non-NHP F M F M F M F M F M F M F M F M
Mean 40.7 33.8 55.6 42.4 45.9 38.1 55.2 42.2 53.4 45.4 64.5 52.4 55.1 46.2 62.5 50.1
N 20618 17737 20685 17863 19034 17137 19029 17131 21499 20029 21506 20029 22371 21530 22371 21530
SD 17.0 16.4 20.7 19.8 17.9 16.8 20.2 19.7 22.9 23.3 21.7 23.3 22.9 23.9 23.0 24.9
B. Teaching Quality
Teaching quality was measured through an index made up of three generally accepted 
standards for determining teacher performance: content knowledge of students; environment for 
student learning; and teaching for student learning. The first of these dimensions has been 
discussed in the previous section. Third grade performance, measured as the percentage of NHP 
children reaching near mastery and mastery over all NHP children taking the third grade 
diagnostic tests was used as the measure of content knowledge. Both mathematics and language 
arts performance are used in the index. 
Learning environment standards relate to the social and emotional components of 
learning as prerequisites to and context for academic achievement. Thus, the focus is on the 
physical setting created by the teacher and the resources available. A six-item scale, dealing 
with the fostering of a positive self-concept, the creation of a nurturing environment that 
supports gender equity, and the organization of space and materials to allow a variety of learning 
opportunities, was used to measure the quality of the environment (See Appendix B: Instruments 
for copies of all research tools). Researchers used the assessment instrument after a complete 
series of observations in a classroom. Specific criteria were provided with each item to ground 
the ratings. Ratings were made on a three-point scale of “not met,” “partially met,” and “fully 
met”. Thus, scores ranged between a minimum of six and a maximum of 18. Scores were 
expressed as a ratio of the actual score over the total possible score.
Table 14 compares the classroom environment scores for 1999, 2000 and 2001. There 
has been and improvement each year. This improvement is likely related to the implementation 
of the new curriculum in NHP schools, as the new curriculum emphasizes changing the 
classroom environment. At least a 14% improvement in scores was found for each type of 
school. This improvement is reflected in the 17% increase for the NHP sample as a whole. 
Classrooms generally met criteria of lack of physical punishment and interacting with individual 
children often. Equal lighting, ventilation, and furniture for boys and girls were also generally 
13
met, and there was an increase in displaying children’s work. Other criteria such as creating a 
variety of learning opportunities within the classroom, encouraging children to express 
themselves with peers and adults, using materials that showed males and females in traditional 
and non-traditional roles, were usually not met. In many of the classrooms, especially those in 
larger schools, the lack of space contributed to a less than optimal classroom environment. 
Children in these classrooms usually were wedged tightly into desks and the only space for 
displaying materials were blackboards that served as partitions between classrooms.
Table 14: Mean Classroom Environment Scores by School Size
 
Teaching for student learning is concerned with the act of teaching and its overall goal of 
helping students understand the content that they are imparting and the ability to present the 
content in a manner that is consistent with the knowledge, interests and abilities of the students. 
For the purposes of monitoring, the focus has been on interactions in the classroom between 
teachers and students. Student-initiated interactions were taken as an indicator; as such 
interactions show teachers’ willingness to recognize student input. Student-initiated were found 
to be a very low percentage of all interactions in teacher-centered classrooms. As mentioned, a 
corpus of 40 minutes of observations of academic lessons was collected in each classroom. 
These observations were divided equally between mathematics lessons and language arts lessons.
Table 15 presents the percentage of observed interactions initiated by teachers and 
students in the normally occurring contexts of the classroom in 1999, 2000, and 2001. The table 
shows the percentage of interactions initiated by each actor in the contexts observed taking place 
in the classroom. The bottom row provides the overall percentage of interactions initiated by 
teachers, boys, and girls. Teacher-initiated interactions predominate in both years. They make 
up at least 88.7% of all interactions. Student-initiated interactions increased somewhat from 
1999 to 2000, but decreased in 2001. This suggests that there has been little progress in 
changing the pedagogy employed by NHP teachers, as teaching strategies remain centered on the 
teacher initiating learning opportunities for children. Little difference is noted by the gender of 
the students, as both boys and girls initiate interactions with similar frequency.
In 2000, members of USAID and the PIU expressed the hypothesis that there was little 
change in student-initiated interactions because students were intimidated by observers of a 
different ethnicity than themselves. In order to test that hypothesis, local researchers, of the 
same ethnicity as the children, were trained to collect the observational data. Despite this 
change, the trends are almost identical for all three years, confirming that the data show teacher￾centered pedagogical approaches throughout the NHP schools, rather than researcher effects. 
Mean/School Size 1999 2000 2001
Small .5929 .6389 .7350
Medium .5900 .6588 .7359
Large .4867 .5490 .7080
Total .5464 .6115 .7218
14
Table 15: Interactions Initiated by Teachers and Students
Interaction Initiator
Teacher Boy Girl
1999 92.5% 3.8% 3.6%
2000 88.7% 5.2% 6.1%
2001 90.1% 3.6% 4.9%
Table 16 shows the types of contexts in which the interactions occurred. As can be seen 
there has been very little change in the contexts in which the majority of interactions between 
students and teachers takes place. The traditional context of a large group in which the teacher 
works with the entire class is the principal instructional principal change in is in the types of 
contexts in which the majority of interactions occur. This is followed by seatwork in which 
children work individually at their desks and the teacher circulates among them or carries out 
other work at his or her desk. 
The two contexts that are indicative of student participation and a decentralization of 
learning are the small group contexts. As shown in Table 9, these contexts make up a very small 
total of lessons. In fact, student led learning activities are almost non-existent. As would be 
expected, the participation in these contexts is very similar for girls and boys.
Table 16: Interactions by Classroom Context
Classroom Context 1999 2000 2001
Teacher-led small group 9.2% 2.4% 7.3%
Student-led small group 2.1% .3% .5%
Large group 49.2% 75.5% 65.2%
Seatwork 34.4% 19.4% 23.9%
No instruction 5.1% 2.5% 3.2%
C. Teaching Skills 
Several indicators of teaching skill are important to the NHP project. Obviously, the 
ability to effectively create an environment that instills self-confidence in students and allows 
them multiple learning opportunities, discussed previously under teaching quality, is related to 
pedagogical ability. The focus here is on specific behaviors engaged in by teachers that 
encourage children to participate in the learning process. Included are: the quality of teacher￾student interactions and the use of materials by students; teachers’ mastery of and commitment to 
the interventions introduced by NHP; and teachers’ strategies for encouraging student 
participation through regular attendance. 
Quality of teacher-student interactions. Teachers’ ability to impart information and 
encourage inquiry rests largely with the types of verbal and non-verbal interactions that they use 
to engage students. To be effective, such interactions create situations that allow students to 
15
apply their knowledge and not merely memorize facts. Teachers must also monitor learning to 
make certain that students assimilate information accurately and can use what they have learned. 
Permitting students to expand ideas together with providing feedback and explanation as needed 
are generally considered manifestations of these skills.
The structured observations of mathematics and language arts, described previously, were 
used to collect data on the quality of student-teacher interactions. The percentage of all 
interactions that involved explanation and feedback was used as the measure of teaching skill. 
As shown in Table 17, teachers provided relatively little explanation or expansion of ideas. This 
type of behavior was found in only 13.4% of all interactions in 1999, 7.3% of such interactions 
in 2000 and 17.1% of the interactions observed in 2001. Feedback in the form of either praise 
shows a positive increase in each year. Feedback through punishment was similar for the three 
years, but shows a slight increase in 2001. 
Table 17: Quality of Interactions
Context/Interaction 1999 2000 2001
Questions 37.3% 64.1% 48.3%
Expands 13.4% 7.3% 17.1%
Orders 40.6% 30.3% 38.5%
Dictates/Lectures 20.3% 18.1% 9.2%
Reinforces 2.9% 3.3% 8.2%
Punishes 1.5% 1.3% 3.2%
Use of materials. A principal focus of the project is on improving the availability and use 
of instructional materials. Both texts and supplementary instructional materials provide children 
with a channel for interacting with academic content on an ongoing basis. Often, however, it is 
assumed that children have books available and that teachers are trained in using instructional 
materials effectively. Teachers may lack practical experience in using texts and when working 
in a development situation may face overcrowded classrooms, children without books and little 
in alternative instructional resources. Thus, they resort to extensive lecture and use of the 
chalkboard. The purpose of this indicator is to confirm the provision of sufficient 
supplementary materials to classrooms of project schools to enrich the teaching and learning of 
literacy and numeracy. However, availability of materials alone is not an adequate measure, as 
students must use materials in order to enhance academic achievement. 
Use of materials was measured by three visual sweeps of the classroom during both 
mathematics and language arts lessons. During the sweeps, the number of available books and 
supplementary instructional materials and manipulatives were counted separately then the 
number actually in use was noted. The average number of materials available per child, as well 
as the average number of materials in use was calculated.
As shown in Table 18, both mathematics texts and supplementary materials such as 
manipulatives, and reading materials increased in the classrooms. This was in part due to the 
supplementary materials provided by NHP, which were present in a number of sample 
16
classrooms. However, in several schools these materials were found stored in the teacher’s office 
rather than present in classrooms. The availability of reading materials increased to the extent 
that almost a text per child, on the average, were observed to be readily available in the sample 
classrooms. 
Table 18: Availability and Use of Texts and Other Learning Materials
Subject Availability Use
1999 2000 2001 1999 2000 2001
Math .20 .40 .54 .25 .13 .18
Reading .40 .90 .91 .27 .20 .13
The use of materials had, however, declined somewhat over the three years of study. 
Only about two children in ten were observed to use math materials, whereas one child in ten 
used reading materials during the 2001 observations. The decrease in use of materials may be a 
result of teachers’ lack of familiarity with the new materials. 
Mastery of the intervention. There is consensus in the international literature on 
educational innovation that mastery of new instructional approaches by teachers is a critical 
factor in adoption and sustainability. As NHP interventions were not yet in place when the 
formative evaluation was initiated in 1999, mastery was measured by asking teachers about the 
general objectives of the program. A second factor closely associated with mastery of the 
innovation is commitment to the new approach. This aspect of teaching skill was measured 
through a series of hypothetical questions in the teacher interview on circumstances that might 
deter a teacher from using an approach.
Table 19: Teacher Support of NHP
Year/Teacher Response 1999 2000 2001
Knowledge of NHP 36% 52% 72%
Use of Incentives 57% 70% 72%
Teacher mastery has improved each year of the project. In 2000, teachers were able to 
identify 28% of the major objectives of NHP as compared to an average of 12% in 1999. In 
2001, all of the elements were identified. However, the increased understanding of the program 
appears to be tied to the dual implementation of the NHP innovations and the new national 
primary curriculum in NHP schools. Most of the teachers identified elements that the programs 
have in common, whereas only about 20% of the sample teachers were able to identify elements 
unique to the NHP program such as a focus on less successful children, and strategies of mixed 
skill and age groupings aimed at increasing the participation of these children. Commitment to 
combined emphasis of the two programs was strong, with 85% of the teachers in 2000 and 100% 
of the teacher in 2001, stating that they would continue to use what they had learned even if the 
program were discontinued.
17
Strategies for encouraging attendance. The purpose of this indicator is to measure the 
extent to which project activities impact absenteeism rates among students. Attendance was 
examined by student gender, as male attendance is traditionally lower than female attendance 
throughout the country. As official school attendance may run the risk of inflation or deflation, a 
correction factor of observed attendance recorded by the evaluation team was built into the 
measure. The key to the success of incentive programs will be their integration with the 
teaching-learning process; thus, classroom teachers are the appropriate source of information 
about incentives. Teachers were asked to list all of those incentives that they were using in their 
classrooms.
There was an increase in the percentage of teachers using incentive strategies. In 1999, 
half of the teachers interviewed stated that they used incentives to increase attendance. In 2000, 
70% of the sample described strategies used to encourage students to come to school. In 2001, 
72% of the sample identified the incentives. 
D. School Visits by NHP Specialists
In order to examine the impact of working directly at the school level, NHP records on 
specialists’ visits to schools and the activities carried out as part of the NHP site-based technical 
assistance strategy were incorporated into the analysis. Table 20 shows the activities carried out 
by the specialists in providing more than 1000 hours of technical assistance in schools. As can 
be seen, the majority of the specialists’ time was spent in observing classes. 
Table 20: NHP Schools- Specialists Activities
Activity Total
Visits to school 227
Number of visitors from NHP 350
Number of Classrooms Observed 777
Number of Demonstration Lessons Given 231
Number of teachers with portfolios 710
Number of man hours spent in schools 1069
Specialists’ monitoring of their activities in schools was consistent with the recollection 
that teachers had of the specialists’ work. As shown in Table 21, over 90% of the teachers 
identified observing teaching as an activity carried out by the NHP specialists. As this was a 
multi-response question, some teachers also identified training, and demonstration activities.
18
Table 21: Teachers’ Recollection of NHP Specialists’ Activities 2001
Activity Number Percent
Observe Teaching 29 90.6
Didactic Training 8 24
Demonstrate New Materials 2 6.3
Demonstrate New Teaching Methods 8 24
Other Activities 5 15.6
In order to examine the impact of the visits, the individual activities at the school level 
were correlated with student performance on the GSAT. As can be seen in Table 22, there was 
very little relationship between the activities and student performance. In fact, most correlations 
are near zero. The lack of any significant relationship is similar for the performance of both boys 
and girls.
This may be a result of relatively few hours spent in each school. On the average, about 
15 person hours were spent in each school. Thus, approximately two days of on-site technical 
assistance during a school year may not be sufficient to have an impact on student performance. 
The nature of the activities also may have contributed to the lack of impact. The focus on 
observing classes, may not have provided teachers and resource personnel with the type of 
hands-on support needed to improve instructional strategies, as appears clear from the lack of 
change found in teaching quality.
19
Table 22: Correlations of Changes in the Levels of Mastery—2000-2001 with NHP Specialist Activities in Schools
Gender 
Subject
Mastery 
Level
Pearson 
Correlation E F G H I AMH REG JA JB JC L M N
No Correlation 0.009 -0.040 0.019 0.028 0.014 -0.043 -0.130 -0.032 -0.044 0.045 0.191 0.118 0.046
Mastery Sig. (2-tailed) 0.938 0.741 0.874 0.814 0.906 0.721 0.279 0.788 0.714 0.708 0.110 0.328 0.702
Near Correlation 0.049 0.117 0.129 0.081 0.058 0.125 0.183 0.129 0.108 0.011 -0.173 -0.025 0.014
mastery Sig. (2-tailed) 0.687 0.329 0.285 0.503 0.629 0.298 0.127 0.282 0.371 0.928 0.149 0.837 0.907
Girls Maths
Mastery Correlation -0.032 -0.047 -0.125 -0.086 -0.047 -0.040 -0.051 -0.046 -0.008 -0.014 0.034 0.034 -0.066
Sig. (2-tailed) 0.788 0.699 0.301 0.475 0.699 0.740 0.671 0.702 0.950 0.911 0.776 0.776 0.584
No Correlation 0.030 -0.059 -0.047 0.029 0.022 -0.088 -0.228 -0.155 -0.065 0.066 -0.035 -0.051 -0.212
Mastery Sig. (2-tailed) 0.805 0.628 0.695 0.813 0.853 0.466 0.056 0.196 0.592 0.584 0.770 0.674 0.075
Near Correlation -0.029 0.036 0.131 -0.013 -0.028 0.057 0.243 0.230 0.079 -0.033 0.096 0.124 0.183
mastery Sig. (2-tailed) 0.813 0.768 0.276 0.913 0.815 0.634 0.041 0.053 0.512 0.784 0.425 0.304 0.128
 Correlation 0.057 0.102 -0.013 0.062 0.073 0.122 0.004 -0.010 0.067 0.028 -0.051 0.052 0.041
Girls Language 
Arts
Mastery
Sig. (2-tailed) 0.640 0.397 0.913 0.607 0.544 0.311 0.973 0.937 0.580 0.818 0.671 0.667 0.732
No Correlation -0.006 0.020 -0.035 -0.071 -0.012 -0.005 -0.036 0.034 -0.093 0.047 -0.060 -0.013 -0.050
Mastery Sig. (2-tailed) 0.959 0.870 0.775 0.559 0.920 0.967 0.763 0.781 0.439 0.696 0.620 0.914 0.678
Near Correlation 0.048 0.018 0.116 0.098 0.033 -0.084 0.069 0.045 0.091 -0.030 -0.018 0.091 0.084
mastery Sig. (2-tailed) 0.692 0.884 0.337 0.418 0.785 0.488 0.565 0.707 0.449 0.802 0.884 0.452 0.486
 Correlation -0.086 -0.079 -0.165 -0.049 -0.042 0.185 -0.065 -0.167 0.013 -0.040 0.167 -0.160 -0.065
Boys Maths
Mastery
Sig. (2-tailed) 0.478 0.510 0.169 0.683 0.730 0.123 0.591 0.163 0.912 0.744 0.165 0.183 0.588
No Correlation 0.092 0.125 -0.033 0.020 0.061 -0.071 0.023 0.048 -0.037 0.111 -0.211 -0.165 -0.112
Mastery Sig. (2-tailed) 0.446 0.299 0.783 0.868 0.611 0.559 0.849 0.691 0.761 0.358 0.078 0.170 0.354
Near Correlation -0.019 -0.065 0.134 0.048 -0.007 0.028 0.090 0.054 0.103 -0.061 0.196 0.228 0.134
mastery Sig. (2-tailed) 0.876 0.592 0.266 0.690 0.955 0.814 0.453 0.653 0.393 0.613 0.102 0.056 0.264
 Correlation -0.128 -0.113 -0.151 -0.109 -0.094 0.076 -0.180 -0.166 -0.097 -0.094 0.056 -0.072 -0.018
Boys Language 
Arts
Mastery
Sig. (2-tailed) 0.288 0.349 0.209 0.365 0.434 0.527 0.134 0.166 0.421 0.434 0.644 0.550 0.880
Table Key
E Number of visitors from NHP
F Number of classrooms observed (yellow form)
G Number of demonstration lessons given
H Number of teachers with portfolios
I Number of man hours spent in school
AMH Average Man Hours
REG
Ja Reciprocal teaching
Jb Student centred teaching
Jc Classes with students having portfolios or journals
Km MRT(s) has/have had development sessions (Y or N)
Kl LART(s) has/have had development sessions (Y or N)
L Quality of leadership in school – see Likert scale
M Quality of MRT(s) in school – see Likert scale
N Quality of LRT(s) in school – see Likert scale
Likert Scale:
5 – Excellent 4 – Good 3 – Satisfactory 2 – Weak 1 – Poor U – unable to make a judgment
20
E. System Support 
In order to improve the success of children, teachers must be supported by an 
infrastructure at the school and national level. This includes support for professional 
development that will contribute to successful teaching and learning, effective management of 
the local learning institution, to ensure that teachers can focus on teaching, and participation of 
community members in the education of their children.
1. Professional Development. 
Training to upgrade skills and knowledge is one of the main ways that a school system provides 
support for teachers. Such training can come about through in-service courses and workshops or 
through interaction with colleagues who have specialized knowledge in a particular subject area 
such as mathematics or language arts. This indicator establishes the number of teachers that 
have engaged in professional development activities as a consequence of their participation in 
New Horizons. The indicator takes into account training in Jamaica and abroad. Schools with 
resource teachers are also used as an indicator. All professional development activities are 
coordinated with the Professional Development Unit of the MOEC.
As shown in Table 23, at the time of the initial formative evaluation data collection, no 
teacher had participated in training offered through the New Horizons project. By the end of the 
1999-2000 school year, 85% of sample teachers stated that they had participated in such training. 
Similarly, the availability of resource teachers had risen from 15% in 1998-1999 to 94% of 
schools in 1999-2000. In 2001, all teachers had participated in workshops and all had resource 
teachers.
Table 23: NHP Professional Development
Professional Development/Year 1999 2000 2001
Teachers participate in Workshops 0 85% 100%
Schools with Resource Teachers 15% 94% 100%
2. School Management 
Tracking of school resources and students is an important function of school management. Such 
tracking should be undertaken within a framework of specific objectives and activities. Thus, the 
utilization of school management plans in regard to NHP activities together with the utilization 
of the computer and accompanying administrative software, which can speed principals’ 
decision-making and ease reporting burdens, are the indicators of effective school management. 
Effectiveness of school boards is an additional indicator of school management. Measures for 
this aspect of management will be developed by the NCE.
21
As part of the NHP program, principals were asked to design development plans taking 
into consideration school needs, teacher training, curriculum design and parent/community 
involvement, especially as related to improving student literacy and numeracy. Among sample 
principals, 30% had completed this task at the time of 1999 formative evaluation data collection. 
Since most of those interviewed mentioned progress in completing the plans, it was expected that 
the number would increase rapidly. As can be seen from Table 24, all principals were 
implementing their development plans by May of 2000. 
Given that all schools had school development plans, a new indicator that was sensitive 
to implementation of the plan, as related to the objectives of NHP was developed. This indicator 
was a weighted index that examined if schools were implementing activities in the SDPs related 
to literacy and numeracy or other activities. The aggregate position of schools on the index was 
.52 in 2001.
Table 24: NHP School Management
Professional Development/Year 1999 2000 2001
School Development Plan 30% 100% 100%
Computer present 25% 68% 100%
Computer used for administration 0 20% 61%
The percentage of schools with computers increased each year and all NHP schools had 
computers in 2001. Ninety-four percent of the principals said that they had received a computer 
from NHP. With the training provided by NHP, the use of computers for administration 
increased by 41%. However, 39% of the principals do not use their computers for administrative 
purposes, suggesting that targeted training is needed in these schools to ensure that consistent 
administrative decision-making and reporting will take place.
NHP staff members hypothesized that the status of a principal might influence the 
effectiveness of a school’s efforts at implementing NHP interventions. In order to partially test 
this hypothesis, the percentage of children attaining near mastery or mastery on the 2001 GSAT 
was examined for permanent principals compared to those with other designations. As can be 
seen from Tables 25 and 26, there is a small but consistent trend favoring permanent principals 
in Mathematics. The trend also exists for boys in language arts. However, the data must be 
interpreted with caution owing to the small number of principals with “other” designations. 
22
Table 25: Mathematics Mastery Levels on GSAT by School Principal Designation
Mastery 
Level/Principal
No Mastery Maths Near Mastery 
Maths
Mastery Maths
Girls Boys Girls Boys Girls Boys
Permanent (14) 55% 69% 32% 24% 13% 7%
Other (4) 63% 75% 26% 20% 11% 5%
Table 26: Language Arts Mastery Levels on GSAT by School Principal Designation
Mastery 
Level/Principal
No Mastery Maths Near Mastery 
Maths
Mastery Maths
Girls Boys Girls Boys Girls Boys
Permanent (14) 37% 62% 38% 22% 25% 16%
Other (4) 38% 67% 38% 18% 24% 15%
3. Community Involvement
The body of research on parent participation shows positive effects brought about by parental 
emphasis on literacy and other achievement in the home. As the focus of the project is on 
improved student learning, parental participation in learning is measured. In addition, parental 
participation in management is important to assure that schooling is relevant to community 
interests. Thus, the presence of parent-teacher associations and the frequency of their meetings 
are other indicators monitored through formative evaluation. Other indicators, such as the 
number of schools with parent participation programs and training for parent and community 
leaders, will be monitored in partnership with the NCE.
Samples of NHP students were asked about parental involvement in their studies. In 
1999, these interviews were conducted as part of the NHP school survey, whereas in 2000 and 
2001, data were collected as part of the formative evaluation. Table 27 shows that there has been 
a slight increase each year in the number of students who stated that either their father or their 
mother assisted them in their reading. However, when all family members are considered, 94% 
of the children who said that they read at home do so with a family member. 
23
Table 27: NHP Community Involvement
Year 1999 2000 2001
Parent Participation in Learning 36% 42% 54%
PTA present 89% 100% 100%
PTA meets regularly 33% 94% 94%
Eighty-nine percent of the NHP schools had PTAs in 1999. However, only 33% meet on 
a regular monthly schedule. In 2000 and 2001, all of the schools had PTAs and almost all were 
meeting regularly.
24
III. CONCLUSIONS AND IMPLICATIONS
The purpose of the study was to assess the progress made by the New Horizons in 
implementing activities that will lead to increased numeracy and literacy for students who have 
had limited success in school. The comparisons made from the baseline year of 1998, or in the 
case of the qualitative data 1999, with the results of the formative evaluation in 2000 and 2001, 
allow certain conclusions and implications to be drawn that can help to guide further 
implementation of the program.
A. Conclusions 
NHP has been most successful in improving the mastery levels of Language Arts among 
students in the upper primary grades. NHP students have improved over the baseline 
in1998 and made progress in reducing the gap in performance between themselves and 
non-NHP schools in Language Arts. 
The percentage of NHP students reaching at least near mastery on the Language Arts 
GSAT increased 14.8% for girls and 16.7% for boys from the 1998 baseline to 2001. 
This compares to increases of 11.5% for girls and 16.3% for boys in non-NHP schools. 
The progress is a result of an increase of the students moving from no mastery to near 
mastery, where NHP gains were at least 6% greater than non-NHP schools.
NHP has been less successful in improving mastery levels of sixth grade students in 
Mathematics. 
The percentage of NHP students reaching at least near mastery has risen by 24.8% for 
girls and 20.2% for boys in Mathematics. However, the changes were not as great as 
those of 28.8% for girls and 24.6% for boys in non-NHP schools. Thus, despite greater 
relative gains than in Language Arts, the initial gap between NHP schools and other 
Jamaican primary schools has not been closed. Again, NHP schools had higher 
percentage change in the near mastery category than did non-NHP schools, but did not 
decrease the gap in students reaching mastery.
Despite positive gains, the percentage of NHP students who master the curriculum is low. 
This low success rate was exacerbated by poor performance of the entire primary system 
in 2001.
Only about one-third of NHP girls and one-fourth of boys exhibit even near mastery on 
the GSAT. Further, there was a decrease in performance for the system as a whole from 
2000 to 2001. In both NHP and non-NHP schools, the percentage of children reaching 
mastery in Language Arts declined by at least 4.2% from the previous year. In 
Mathematics, the decline was 0.5%, of more. The decrease is unexplained, as the tests 
were similar and although more children took the tests in 2001, standard deviations on 
the mean scores decreased. This suggests that the increase in participants did not skew 
the distribution of scores to a greater extent than in the previous year.
25
The first three grades of primary school have been the most difficult for NHP to improve 
children’s performance in Mathematics, and especially Language Arts.
Children in the third grade curriculum in NHP schools, most of who have participated in 
the three years of NHP program implementation, have lower combined near mastery and 
mastery levels than non-NHP third graders. Whereas there has been a percentage 
increase in Mathematics from the 1998 baseline for both boys (+13.5%) and girls 
(+16.5), these increases are less than the 20.6% increase for girls and the 20.3% increase 
for boys in non-NHP schools. In Language Arts, NHP third graders have had a decrease 
in the combined near mastery and mastery levels of -3.6% for girls and –1.1% for boys. 
Non-NHP schools, on the other hand, have had positive increases.
NHP has been successful in changing classroom environments so that they are organized 
to facilitate learning.
Classroom environments improved each year in NHP schools. Children’s work was 
displayed to a greater extent, teachers were positive when interacting with students, and 
in many classrooms, there was an improvement in the organization of space.
The participatory, child-center classroom approaches, emphasized by the NHP program, 
have generally not been implemented in NHP classrooms.
Instructional delivery in NHP schools remains highly traditional. Teachers initiate more 
than 90% of the interactions with children, allowing little opportunity for self-expression 
or expansion of ideas. The majority of instruction takes place in teacher-directed large 
group or seatwork contexts. Small group work is the context for less than 8% of all 
interactions, and small groups led by students are almost non-existent.
NHP has been highly successful in providing ancillary learning materials to schools. 
However, such materials are under-utilized in the classrooms.
In1999, the number of materials observed in the classroom was sufficient for only about 
20% of the students. In 2001, there are sufficient reading materials for 90% of the 
students, and mathematics materials for over half of the students readily observable. 
During lessons, such materials were actually used by less that 20% of the students. 
The concentrated effort by NHP to provide hands-on professional development and other 
technical assistance at the school level has yet to show an impact on student performance 
or teacher behavior. This may be a result of the types of activities engaged in by the 
NHP professionals when making school visits.
The number, type, and level of effort of activities carried out by NHP professionals in 
individual schools were correlated with the change in mastery levels for each school from 
2000 to 2001. No significant correlations were found. This may be the result of the 
26
types of activities engaged in by the professionals. More than 70% of the time was spent 
in observing classes and providing feedback, rather than demonstration and training.
At the school level, NHP has been successful in creating an environment to support 
improved learning of Mathematics and Language Arts.
Every indicator of system support has had a positive change from the 1999 baseline year. 
All schools have school development plans and PTAs that meet regularly. One hundred 
percent of the teachers have participated in professional development activities and all of 
the schools have resource teachers. All schools have administrative computers and two￾thirds of them are being used to support administrative procedures.
27
B. Implications 
It might be argued that improving children’s performance in schools serving those 
students who have had the least success in school, because of poverty and other factors, is 
more difficult than improving the success of more fortunate children. However, such 
improvement is the objective of NHP and measuring NHP students’ progress against non￾NHP schools is the only way to know that the gains found are the result of the NHP 
program and not simply general improvement of the Jamaica primary school system. 
Thus, the clearest case of NHP impact is for Language Arts in the upper grades, as 
measured by mastery levels on the GSAT. As the same trend is not found for third grade, 
it is likely that NHP is providing interventions for upper grade Language Arts that are 
different than for the earlier grades. NHP specialists should review their activities to 
determine interventions that are contributing to improved Language Arts performance 
and examine the applicability of such strategies to earlier grades, and perhaps to 
Mathematics. One possibility is that greater use is being made of ancillary reading 
materials provided by the project. 
NHP teachers’ lack of use of the participatory, child-centered methodologies, 
espoused by NHP and the new primary curriculum argues for a change in the professional 
development strategies employed by the NHP project. NHP specialists have focused on a 
“training of trainers” and clinical supervision, focused primarily on resource teachers. 
More direct demonstration and modeling with teachers when specialists visit the schools 
may be necessary to change classroom behavior. Given the small number of NHP 
professional staff, their expertise may have to be supplemented by additional subject 
matter or pedagogical specialists.
The administrative infrastructure for improvement in learning appears to be in
place and is an important achievement of the NHP project. Given the limited 
improvement in student performance, however, may require special training for teachers 
and administrators to make diagnosis of student performance and planning of strategies 
that will enhance student abilities in Mathematics and Language Arts and explicit part pf 
the administrative process.
The results suggest that Jamaican primary education is not yet producing the types 
of graduates that the society desires. Although there have been improvements in the 
percentage of sixth grade students with mastery of the curriculum, less than 35% have 
mastered the language Arts curriculum and fewer than 20% the Mathematics curriculum. 
The monitoring carried out under the formative evaluation has coincided not only with 
the implementation of NHP but also with the launching of a new primary school 
curriculum. It may be that when the curriculum is fully implemented in all schools, that 
improvement in student performance will be accelerated. 
28
Appendix A:
Indicators and Measures
29
INDICATORS AND MEASURES- NHP FORMATIVE EVALUATION
Strategic Objective: Increased Literacy and Numeracy Among Targeted 
Jamaican Youth
Performance Indicators:
· Language Arts scores- grades 3 and 6
· Mathematics scores- grades 3 and 6
Measures:
· % of students in grades 3 through 6 of NHP schools meeting near mastery 
criteria in Language Arts
· % of students in grades 3 through 6 of NHP schools meeting near mastery 
criteria in Mathematics
Intermediate Result 1: Improved Quality of Teaching
Performance Indicator: Index of teacher quality
Measure: Composite of the three basic components of teacher performance, 
including 1) content knowledge of students (number of students reaching near 
mastery in math and language arts over all students at a grade level); 2) 
classroom learning environment (measured on a classroom environment scale); 
and 3) teaching for learning (percentage of student initiated interactions) 
aggregated across sample classrooms, expressed as a value 
between 0 (minimum) and 1(maximum). Scores on these dimensions have been 
averaged as an overall index: 
(NMLpy + NMMpy + CE + SI) ¸ 4 where 
NMLpy = Percent of all third grade students reaching near mastery in 
language arts in the previous year
NMMpy = Percent of students reaching near mastery in mathematics in 
the previous year
CE = Average score of sample classrooms on classroom environment 
scale
SI = Percent of student-initiated interactions over the expected frequency 
of such interactions averaged across sample classrooms.
30
Lower Level Intermediate Results 1.1: Improved teaching-skills competency
Performance Indicator: Quality of teacher-student interactions
Measure: Number of interactions involving expansion and feedback divided by 
the total number of observed interactions between teachers and students.
Lower Level Intermediate Result 1.2: Improved knowledge of subject matter
Performance Indicator: Teacher mastery of intervention principles
Measure: Number of teachers who identify the principles of a given intervention 
divided by the total number of teachers in a sample who are employing the 
intervention.
Lower Level Intermediate Results 1.1a and 1.2a: Professional development program
Performance Indicator: Professional development program functioning
Measure:
· Number of teachers who have participated in training divided by total 
number of NHP teachers
· Number of NHP schools with resource teachers divided by all NHP 
schools
Lower Level Intermediate Result 1.3: Improved instructional materials
Performance Indicator: Utilization of instructional materials
Measure:
· Average number of instructional materials available per student in 
classrooms
· Average number of students utilizing materials divided by the total 
number of students present
Intermediate Level Result 2: Increased student attendance
Performance Indicators: Number of enrolled students in project schools attending 
classes
Measure: Average number of students attending class on a given day, divided by 
the number of students enrolled in sample schools, corrected for observed 
attendance, disaggregated by gender.
31
Lower Level Intermediate Result 2.1 : Special incentives program implemented
Performance Indicator: Special incentive programs functioning in classrooms
Measure: Number of sample teachers identifying incentive programs in their 
classrooms divided by all teachers in the sample.
Lower Level Intermediate Result 2.2: Improved school/parent/community 
involvement
Performance Indicators: Parents assist students with academic studies
Measure: Number of students stating that parents assist with reading/math divided 
by the total sample of students.
Lower Level Intermediate Result 2.2.1: PTAs effectively functioning
Performance Indicators: PTA=s meet regularly
Measure: Number of schools with PTAs adhering to a regular schedule of 
meetings divided by all NHP schools.
First Level Intermediate Result 3: Improved management of schools
Performance Indicators: Management plans in place
Measure: Number of schools with school development plans divided by all NHP 
schools
Lower Level Intermediate Result 3.2: Effective School Principals and 
Administrators
Performance Indicators: School administrators utilizing computer assisted 
management procedures
Measure: Number of principals using computers for school administration 
divided by all NHP principals
32
Appendix B:
USAID Reporting Tables
33
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on language arts performance, allows comparison with national average. This is 
important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 boys’ GSAT Language Arts scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 80% of the grade 6 boys were in 
the “no mastery” group in Language Arts.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: 
Year Planned Actual
1998 17.0
1999 19.0 20.0
2000 22.0 35.0
2001 25.0 34.0
2002 30.0
2003 35.0
F. Other
Comments: The slight decline is consistent with a decline for the system as a whole. This decline is likely related to an 
increased number of students, who were formerly held back, taking the GSAT and scoring in the “no mastery” level.
34
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on language arts performance, allows comparison with national average. 
Important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 girls’ GSAT Language Arts scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 57% of the grade 6 girls were in 
the “no mastery” group in Language Arts.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: Year 2000 actual differs from previously reported percentages because of corrections made 
in the database
Year Planned Actual
1998 43.0
1999 45.0 47.0
2000 48.0 63.0
2001 52.0 57.0
2002 56.0
2003 60.0
F. Other
Comments: The slight decline is consistent with a decline for the system as a whole. This decline is likely related to an 
increased number of students, who were formerly held, taking the GSAT and scoring in the “no mastery” level.
35
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on mathematics performance, allows comparison with national average. This is 
important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 boys’ GSAT Mathematics scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 90% of the grade 6 boys were in 
the “no mastery” group in Mathematics.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively, are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: 
Year Planned Actual
1998 6.0
1999 7.0 10.0
2000 10.0 25.0
2001 13.0 26.0
2002 20.0
2003 30.0
F. Other
Comments: The slight increase is consistent with that for the system as a whole. This is likely related to an increased 
number of students, who were formerly held back, taking the GSAT and scoring in the “no mastery” level.
36
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: SO Level 532-004
A. Description
Precise Definition: Percentage of students meeting near mastery in grade 6 of New Horizons schools.
Unit of Measure: Number of grade 6 NHP students meeting the criterion of near mastery/mastery on GSAT divided by 
all grade 6 NHP students.
Disaggregated By: Gender, grade level and program (NHP; non-NHP)
Management Utility: Project impact on mathematics performance, allows comparison with national average. This is 
important for determining the impact of NHP interventions in relation to overall system improvement, over the life of 
the project.
B. Plan for Data Collection
Indicator: NHP grade 6 girls’ GSAT Mathematics scores
Source: Student test data from Student Assessment Unit
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Manipulate Student Assessment Unit database to separate NHP and non-NHP students by gender and 
mastery levels on GSAT (50% correct= near mastery; 75% correct=mastery) divide by total number for each group, 
calculate percentage change from 1998 baseline and by year.
Presentation of Data: Combined percentage of near mastery and mastery in Tables of planned and actual improvement 
by year.
Review of Data: Review is performed by the institutional contractor, SO team’s annual portfolio review, and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: The criteria used to designate near mastery and mastery with the third grade diagnostic tests 
is used. Using these criteria, at the time of establishing the baseline and targets, over 85% of the grade 6 girls were in 
the “no mastery” group in Mathematics.
Known Data Limitations: The Student Assessment Unit does not designate mastery levels for sixth grade GSAT. 
Therefore, the third grade criteria of 50% and 75% for near mastery and mastery, respectively, are used in determining 
student progress. 
Actions Addressing Limits: The latest available data will be used.
E. Performance Data Table
Method of Calculations: NHP and non-NHP students separated by gender and mastery levels then divided by total 
number of NHP and non-NHP GSAT scores for each gender.
Key to Table: No key, the table is easily interpreted
Baseline & Target Notes: 
Year Planned Actual
1998 14.0
1999 16.0 24.0
2000 18.0 41.0
2001 20.0 42.0
2002 25.0
2003 30.0
F. Other
Comments: The slight increase is consistent with that for the system as a whole. This is likely related to an increased 
number of students, who were formerly held back, taking the GSAT and scoring in the “no mastery” level.
37
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: 4.1 532-004 Improved Teaching Quality
A. Description
Precise Definition: Composite of: 1) content knowledge of students; 2) classroom learning environment; 
and 3) teaching for learning, aggregated across sample classrooms and expressed as values between 0 
(minimum) and 1 maximum
Unit of Measure: Index of third grade mastery levels – mathematics and language arts, score on classroom 
environment scale and percentage of child-initiated interactions, aggregated across sample classrooms.
Disaggregated By: Unnecessary
Management Utility: To track improvement in the quality of teaching over the life of the project.
B. Plan for Data Collection
Indicator: Index of Teacher Quality
Source: Student test data from Student Assessment Unit, observational data from formative evaluation of a 
stratified, random sample of NHP schools
Data Collection: Yearly
Est. Cost of Collection: N/A
Responsible Organization: Institutional contractor’s Chief of Party and formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Aggregate each measure and average into an overall index of sample schools.
Presentation of Data: Index value between 0 – minimum and 1 – maximum in Tables of planned and 
actual performance
Review of Data: Review is performed by the SO team, the institutional contractor and other stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: Data collected by trained observers
Known Data Limitations: Diagnostic purposes of third grade tests results leading to lack of full reporting 
by schools. 
Actions Addressing Limits: The COP for the institutional contractor will ensure that adequate data are 
available prior to the R4.
E. Performance Data Table
Method of Calculations: Scores of three dimensions are averaged as an overall index
Key to Table: No key
Baseline & Target Notes: 
Year Planned Actual
1999 .43
2000 .50 .44
2001 .58 .48
2002 .65
2003 .71
2004
 
F. Other
Comments: Planned levels have not been met owing to teachers’ continued use of traditional teacher￾centered pedagogy.
38
Performance Monitoring Plan
Fiscal Year FY2003
Strategic Objective: Increased literacy and numeracy among targeted Jamaican youth
Intermediate Result: 4.3 Improved Management of Schools
A. Description
Precise Definition: Number of schools implementing School Development Plan activities in literacy and 
numeracy; plus schools implementing activities in either literacy or numeracy; plus schools not 
implementing activities in these areas divided by the total number of schools in the sample
Unit of Measure: Weighted index where (L&N=1;L or N =.5; and other activities = 0)
Disaggregated By: Unnecessary
Management Utility: To measure the integration of project interventions with school activities.
B. Plan for Data Collection
Indicator: NHP schools implement SDP activities in numeracy and literacy
Source: Principals in a stratified, random sample of NHP schools
Data Collection: Yearly
Est. Cost:
Responsible Organization: Institutional contractor’s formative evaluation team
C. Plan for Data Analysis, Reporting, Review
Data Analysis: Weight responses, sum response categories, divide by number of sample schools using 
Excel or SPSS software.
Presentation of Data: Index value between 0 – minimum and 1 – maximum.
Review of Data: Review is performed by the SO team, the formative evaluation team and other 
stakeholders
Reporting of Data: Annual performance reports and highlighted tables and narrative of R4
D. Data Quality Issues
Initial Data Qual/Assess: Data collected by trained interviewers
Known Data Limitations: None
Actions Addressing Limits: None
E. Performance Data Table
Method of Calculations: Weighted index of SDP implementation
Key to Table: None
Baseline & Target Notes: New indicator with 2001 as baseline year
Year Planned Actual
2001 .52
2002 .70
2003 .90
 
F. Other
Comments: This indicator was revised after 5-year targets were reached in 2 years with previous indicator.