Assessing Progress Related to USAID/Macedonia’s Strategic Objective 3.4: Technical Report and Results from the 2004 Administration Marc Shapiro, Zoran Stojanov, Vaska Dekova, and Jennifer Anderson December 2004 Contract Number: GDG-A-00-03-00006-00 Task Order Number: 165-00-03-00105-00 EQUIP1: Secondary Education Activity (SEA) USAID/Macedonia and EGAT/ED SO 3.4: Students better prepared for employment through education programs” Submitted by: American Institutes for Research 1000 Thomas Jefferson Street NW Washington, DC 20007 Phone: 202.403.5000 Fax: 202.403.5979 www.air.org with: International Reading Association AMERICAN INSTITUTES FOR RESEARCH® Assessing Progress Related to USAID/Macedonia’s Strategic Objective 3.4: TECHNICAL REPORT AND RESULTS FROM THE 2004 ADMINISTRATION Marc Shapiro, Zoran Stojanov, Vaska Dekova, and Jennifer Anderson December 2004 American Institutes for Research Secondary Education Activity, Macedonia “American Institutes for Research” is a registered trademark. All other brand, product, or company names are trademarks or registered trademarks of their respective owners. 1000 THOMAS JEFFERSON ST, NW|WASHINGTON, DC 20007-3835|TEL 202 403 5000| WEBSITE WWW.AIR.ORG Table of Contents Chapter 1. Introduction .................................................................................................................. 2 1.1 Goal.............................................................................................................................. 2 1.2 Design of the Study...................................................................................................... 2 1.3 Overview of Report...................................................................................................... 2 SECTION ONE: TECHNICAL REPORT................................................................................................. 3 Chapter 2. Description of projects that contribute to Strategic Objective 3.4 ............................... 4 2.1 Overview...................................................................................................................... 4 2.2 The Creative Teaching and Learning Project (CTLP)................................................. 4 2.3 Secondary Education Activity ..................................................................................... 5 2.4 E-Schools in Macedonia Project (EDC)...................................................................... 6 Chapter 3. Study Design and Instrument Development................................................................. 7 3.1 Overview...................................................................................................................... 7 3.2 Index Construction....................................................................................................... 7 3.3 Framework ................................................................................................................... 8 3.4 Administration Schedule.............................................................................................. 9 3.5 Translation ................................................................................................................... 9 3.6 Pilot Studies............................................................................................................... 10 Chapter 4. Sample Design............................................................................................................ 11 4.1 Overview.................................................................................................................... 11 4.2 Sample Description.................................................................................................... 11 Chapter 5. Study Limitations....................................................................................................... 14 Chapter 6. Operations.................................................................................................................. 15 6.1 Administration ........................................................................................................... 15 6.2 Scoring ....................................................................................................................... 15 SECTION TWO: RESULTS FROM THE 2004 ADMINISTRATION.......................................................... 17 Chapter 7. Results......................................................................................................................... 18 7.1 Overview..................................................................................................................... 18 7.2 Problem-Solving Skills............................................................................................... 19 7.2 Improved Instructional Methods................................................................................. 20 7.4 Computer Access, Computer Skills, and Computer Use for Teaching....................... 23 7.5 Career Preparation ...................................................................................................... 26 Appendix A. Construction of indices............................................................................................ 28 Appendix B. Data tables............................................................................................................... 34 Appendix C. English translations of student and teacher surveys................................................ 53 1 Chapter 1. Introduction 1.1 Goal USAID/Macedonia has asked the American Institutes for Research (AIR) to assess progress related to Strategic Objective 3.4, “Students better prepared for employment through education programs.” The goal of this study is to measure the combined effect of three USAID-funded education projects on the achievement of this objective. The three projects are Foundation Open Society Institute Macedonia’s (FOSIM) Creative Teaching and Learning Project (CTLP), Educational Quality Improvement Program’s (EQUIP1) Secondary Education Activity (SEA), and Education Development Center’s (EDC) E-Schools in Macedonia Project. The study will conduct annual data collection through 2008. This report explains the study design and provides details on the results of the 2004 administration. 1.2 Design of the Study The research instruments for this effort are a teacher’s survey, a student survey, and a test of student problem solving skills. The surveys are administered annually at secondary schools, both vocational and gymnasia that are receiving programming from one or more of the three projects. Teacher and student surveys incorporate questions designed to measure behavior, attitudes and skill levels relevant to the increased employability of students. Both surveys are administered annually. The test of student problem solving skills is designed to measure the impact of SEA and CTLP methodologies on students’ problem solving and critical thinking abilities. This component of the student assessment is administered in 2004 and 2008. During the 2004 administration, the study instruments were administered at 64 Schools, all of which are or will be part of the SEA and CTLP programming and most of which are receiving E-Schools programming. To maximize the power of the research, respondents are chosen in a stratified sampling design with random sampling within strata. Strata are based on ethnicity and school type. Gymnasia respondents also are stratified by whether they are in schools in which faculty have been exposed to the new methodologies. The surveys include a total of 1689 students and 901 teachers representing all regions and ethnic groups, although not all respondents answered enough questions (at least a third) for each respondent to be counted for that index. All national level estimates for students or teachers in USAID schools use data weighted to compensate for the stratification. 1.3 Overview of Report This report is divided into two sections. The first section is the technical report, which presents information on the design, piloting, and administration of the study instruments. The second section presents results from the 2004 administration. This results section is divided into topical areas, and both teacher and student results are discussed according to topic. Because only the CTLP project is beyond its first year of implementation, the data presented in this report are intended as interim results for the CTLP project and as baseline measures for the SEA and EDC projects. 2 SECTION ONE: TECHNICAL REPORT 3 Chapter 2. Description of projects that contribute to Strategic Objective 3.4 2.1 Overview In a country such as Macedonia, facing the challenges of a growing free-market economy, a well-educated populace is a key factor to achieving the country’s economic potential. USAID is sponsoring projects that seek to better prepare Macedonian students for the work force. Specifically, three projects contribute to USAID’s Strategic Objective 3.4, “Students better prepared for employment through education programs.” These are the Creative Teaching and Learning Project (CTLP), the Secondary Education Activity (SEA), and the E￾Schools in Macedonia Project (EDC). These projects use varied strategies to affect positive changes in the skills and abilities of Macedonian youth. Strategies include training teachers in methods that promote critical thinking in students, creating career centers in vocational schools and opportunities for vocational students to apply and practice skills learned in school, and providing information and technology resources and instruction for teachers to engender computer literacy and creativity among high school students. Working both independently and collaboratively where possible, the three projects contribute to the goal of better preparing students for employment by addressing several of the many skill areas important for workers in a modern world economy. 2.2 The Creative Teaching and Learning Project The Creative Teaching and Learning Project (CTLP), implemented by The Foundation Open Society Institute Macedonia (FOSIM), is three-year project aimed at fostering teaching methods and techniques, as well as extra-curricular activities, which promote development of critical thinkers and writers. Working in both primary and secondary schools, FOSIM is providing training based on the Reading and Writing for Critical Thinking Project (RWCT) that has been operating in Macedonia since 1997. The Strategic Objective 3.4 study focuses on CTLP’s work at the secondary level, where the project is training teachers, school principals, pedagogists, and parents in the RWCT methodologies. In addition to this professional development element, the project also provides support for establishing Media Laboratories at four secondary schools, as well as training and materials needed to establish after school activities, such as Student Democracy Clubs, in four additional schools. It is further hoped that through the work of the project networks will be created among teachers committed to creative teaching and learning. The goal of the CTLP project is to develop the critical thinking and writing skills of high school students, including their abilities to form original opinions, choose rationally between competing ideas, solve problems, and debate ideas responsibly—all skills relevant to active participation in a pluralistic, democratic society. This goal is achieved through the direct and indirect effects on pupils generated from teacher training activities. The CTLP-based teacher training programs are not curriculum specific, but rather focus on strategies such as instructional arrangements, program processes, instructional practice, and pupil assessment that cut across curriculum areas. Such programs can directly impact students’ knowledge, skills, and behavior, while indirectly affecting these same target areas by changing teacher reactions, attitudes, behavior, and knowledge. 4 Project implementation began in September 2002 in four secondary schools. A further eight secondary schools joined the program in 2003-2004, and six more secondary schools are slated to join the program in the 2004-2005 school year. 2.3 Secondary Education Activity The SEA project, which also has a strong teacher-training element, similarly depends on this model of direct and indirect student impact. The SEA project has three components, two of which contribute directly to Strategic Objective 3.4 and are subject to the evaluation. The third component, providing support to the Ministry of Education to develop a licensure program for secondary school directors, is not assessed under the Strategic Objective 3.4 evaluation. SEA’s teacher-training component is part of a five-year project implemented jointly by the American Institutes for Research and the International Reading Association, and is aimed at improving students’ experiences in 50 vocational schools around Macedonia. The project hopes to make students’ vocational school experiences more relevant to their future employment. The SEA teacher-training component differs from CTLP in that it focuses on bringing contextual learning methods to secondary vocational classrooms. Contextual learning methods help relate theoretical training to practical applications, with the ultimate goal of improving students’ critical thinking and problem-solving skills. To this end, SEA is training master teachers from each school, as well as pedagogists and school principals in contextual learning methods. This then provides the groundwork for the master teachers to train the other teachers in their schools. Training of the first cohort of master teachers from 15 schools began in May 2004. At present, training for this cohort is scheduled to be completed in February 2005, at which point the second phase of training the other teachers in the school may begin. However, the SEA team (at the time of writing this report) is in the process of setting a schedule to accelerate training plans for all cohorts. Still, the full impact of having all teachers trained in all project schools will not occur until at least spring of 2006. The other component of the SEA project relevant to Strategic Objective 3.4 is the career development component. The goal of this component is the development of students’ practical entrepreneurial and business skills and knowledge about their career fields. School￾based career centers are being established in each of the 50 project schools to provide students with the information they need to make informed career choices. In addition, SEA is supporting the creation of supervised work experiences (which can include school-based real and virtual firms as well as documenting real work outside school) and competitive, career development events. Real firms that sell products or virtual firms that exist only as a simulation are methods through which students are able to receive professional qualification within the frames of practically oriented training processes such as sales, management, and relations with state institutions (courts, trade and statistical registers, tax and bank services, etc.). In the network of virtual firms no real money and goods are operating, which minimizes the consequences of eventual mistakes and provides opportunities to evolve trainees’ skills. Such virtual and real firms foster development of students’ problem-solving and critical thinking skills, and provide opportunities to put acquired IT skills to use. Installation of career centers and procurement of necessary equipment for real and virtual firms will begin fall 2004. 5 The SEA project is additionally working both with schools and local businesses to increase student options for real-life practical learning opportunities such as internships. Through internships and competitive career development events, SEA hopes to strengthen the link between schools and industry in the context of a transition economy. 2.4 E-Schools in Macedonia Project (EDC) The last of the three Strategic Objective 3.4 projects is the E-Schools in Macedonia Project, implemented by the Education Development Center (EDC). This two-year project focuses on bringing IT resources and teaching methods to primary and secondary schools to encourage the development of IT literacy in Macedonian youth. At the secondary school level, EDC is providing Internet connectivity and relevant applications in order to enable school computer labs to become effective resource and learning centers. This is complemented by the provision of teacher-training in methodologies for IT literacy instruction and the use of IT for active learning. EDC activities are being implemented on a continual roll-out basis throughout the life of the project. The first secondary school computer lab was opened November 4, 2003. 6 Chapter 3. Study Design and Instrument Development 3.1 Overview The study measures progress toward Strategic Objective 3.4 by gathering data on all three projects (SEA, CTLP and EDC) that contribute to enhanced employability of secondary students. It does this by focusing on the following six project-related elements: teacher attitudes, instructional practice, teacher and student computer access, career development activities, computer skills and student problem-solving skills. Data on the first five elements—teacher attitudes, instructional practice, computer access, computer skills and career development—is gathered through two instruments: a teacher survey and a student survey. Data on the sixth element, problem-solving skills, is gathered through a student assessment piece. For teachers, we measure three project related indices on a scale with a maximum of 100 as well as two other indices to measure obstacles to use. The three indices are: teacher attitudes, teaching practices, and computer use for teaching. The teacher attitudes index measures teachers’ attitudes towards teaching practices synonymous with SEA and CTLP methodologies versus more traditional practices. The instructional practice index measures the extent to which teachers are implementing SEA and CTLP methods or more “traditional” teaching practices such as repetition or teacher-centered instruction. This includes asking teachers how often students perform certain tasks and how student performance is evaluated. A third index measures teacher use of computers for teaching purposes. The teacher survey also includes questions that form two other diagnostic indices that measure teachers’ access to computers at school and the obstacles teachers face in implementing new instructional practices. Five project-related indices are measured for students. Like teachers, student indices include an index on use of the instructional methods that compose the CPTC and SEA programs as an index on computer access. Three student indices that differ from teacher indices measure computer skills, problem-solving skills, and career preparation. 3.2 Index Construction The instructional practice index incorporates five higher-level objectives underlying CTLP and SEA programming. These objectives are 1) problem-solving skills 2) collaborative learning, leadership, and communication skills, 3) assessment and self assessment methods, 4) research and project based learning, and 5) community-based and work-based learning. Inadequate facilities and lack of appropriate programs or systems for measuring technical ability on computers in Macedonia precludes an actual assessment of computing ability. For this reason the index on student computer skills is a self-reporting device broken down into four subcategories of skill. These subcategories are basic computing skills, working with documents, multi-media and research. The SEA project is the only one of the three projects to address career development activities, and thus the index of career development activities is analyzed only for vocational schools. The index is broken into the following three subcategories: practical experience (including working in virtual and school firms), career guidance, and job-seeking skills such as developing portfolios or CVs. 7 As stated above, problem-solving skills are assessed via a student test, which is administered along with the student survey. The test generates one index score. Test items were taken from portions of the problem-solving section of the OECD’s 2003 Programme for International Student Assessment, which has already been validated and tested internationally. Full details about which survey items make up each of the teacher and student indices may be found in Appendix A. Tests of the reliability of the indices constructed indicated strong reliability levels with Chronbach alpha statistics all near .80 to over .95. 3.3 Framework The study collects data from students and teachers before the intervention for the SEA and EDC projects (with the exception of the first set of master teachers taught under SEA). Over time, the study will continue to collect data from teachers and students whose schools have not yet experienced these interventions as well as from those who have. This is a mixed study design that allows for pre- and post-intervention comparisons as well as comparisons between control and experimental groups in future years. These comparisons will be made with different cohorts of students and teachers rather than as panels studied across time. The study begins in the middle of the CTLP intervention, after two years of project training already has occurred. This will allow for pre- and post-intervention comparisons in gymnasia schools only for the one cohort of schools that has not already experienced the CTLP training. The 6 schools with gymnasia that have not received training serve as controls for the first year. We do not include as controls any of the other vocational schools that will not be a part of the SEA program. This design limits the effects of contamination from other similar programs run in the schools that are not receiving an intervention. The Macedonian school system has been exposed to numerous professional development programs with some elements similar to CTLP and SEA, especially with vocational education projects. Teachers and schools that have experienced these interventions can be expected to score higher on questions designed to measure CTLP and SEA characteristics. Teachers at schools that have experienced no intervention, in particular those served by SEA, can be expected to score lower, making baseline level comparisons difficult. (Expected differences in baseline scores among students and teachers from gymnasia and vocational schools is another reason gymnasia and vocational schools are not appropriate comparisons with each other.) More importantly, one cannot “control” for changes over time from USAID’s programs if other projects, such as the European Union’s VET III projects, which also contain elements to develop problem-solving skills, are ongoing in these other schools. This methodology also appears appropriate to build a national estimate of the effect of S.O. 3.4 because these projects are not designed to affect directly schools outside of these programs. In this study, CTLP schools and SEA schools also are not used as comparisons with each other. CTLP is conducted in gymnasia while SEA is conducted at vocational schools. Vocational schools traditionally have received less attention than gymnasia or have a history of poor results with other projects. Also, the vocational schools have few or no other ongoing 8 projects. Therefore, the SEA schools can be expected to have a lower starting point than the CTLP schools. 3.4 Administration Schedule The teacher and student surveys are administered annually at the end of each school year starting in 2004. The student problem solving assessment will be administered in 2004 and 2008. The reason for administering the problem solving assessment only twice is twofold. First, the project design relies on a train-the-trainer model for the dissemination of CTLP teaching techniques. This means it will take several years for schools to build up a core number of teachers implementing the methodologies. Even then, not all teachers are expected to adopt these methods so that student exposure to the new methodologies will be inconsistent and incomplete. For instance, if, on average, only a quarter of the teachers trained by USAID’s projects adopt the practices within the first two years after their training, fourth-year graduating students will have been exposed to these teachers and their methods in no more than one eighth of their classes (year 1 and 2 x 0 percent of teachers + year 3 and 4 x 25 percent of teachers = 12.5 percent = 1/8). Second, studies suggest that the development of the skills being measured in this study takes time, especially in the case of developing problem solving skills even assuming 100 percent teacher participation.1 This limits the value of annual assessment. The limited effect on students in the early years of the project is compounded by the previous issue, that by the third year of the SEA project (two years after the baseline), vocational students will have had limited exposure to more teachers using methodologies that help spur problem solving capabilities. Because the number of students exposed to these methodologies is expected to increase most dramatically in the final two years of the project, assessment is most efficient and likely to show the expected results in the fifth year. 3.5 Translation Due to the limited time available between survey and assessment construction and delivery, the study was unable to translate and back translate instruments. Instead, survey instruments were translated into Macedonian and Albanian by two independent translators for each language. The two separate translated versions were merged into one during a session in which both translators and a third or fourth staff member, involved in developing the survey, discussed and agreed on the correct translation item-by-item. The resulting translated survey instruments for each language were compared by an independent party fluent in both languages and with experience with both survey instruments and the Macedonian school system to assure consistency. Pilot testing occurred in both languages (see next sub-chapter), and, where applicable, required changes were translated into the other language with an independent party again serving to reconcile meanings. These changes then were back translated into English and discussed. The emphasis in the translation process was comprehensibility in the native 1 An assessment of the impact of CTLP type teaching methodologies on teaching practice in four countries (including Macedonia) found that, after four years of programming, participants integrated critical thinking principles in teaching practices at levels greater than that found in the control group at statistically significant levels. (see: Heyman, Cory and Arthur Evans, (2001). “The 2000-2001 Evaluation of the Reading and Writing for Critical Thinking Project.” Washington, DC, September 5.) 9 languages and accuracy with the intent of the question rather than exact translations from and back to English. The same procedure was used for translating the problem-solving skills assessment with the exception that item-by-item pilot testing was not required. An independent party with expertise in administering assessments in Macedonia for PISA reconciled the translations across languages. 3.6 Pilot Studies Initial pilot testing with international and vocational students and teachers occurred with the English version in the United States. After translating and merging the two versions of the surveys, pilot testing was organized to increase the reliability of the instrument by making questions easier to understand for students and teachers. There was also a concern to ensure that questions were relevant to students’ and teachers’ in-school experiences. Pilot testing of the student survey was organized with seven Macedonian and six Albanian students. Testing was held in sessions with one or two students. After answering the questions, students were asked to give comments on the relevance of questions to their school experience, on their ability to understand specific questions, and their reaction and attitude towards the type and number of questions. Comments and suggestions collected from these sessions were used to develop the final version of the survey form. The same procedure was used for pilot testing the teacher instrument. The survey instruments were developed under a compressed time schedule between approval of project indicators and the end of school year, when the baseline measurements were required. Traditionally, pilot testing may occur over a considerable span of time. This resulted in more limited pilot testing than preferred, but the pilot testing that did occur solved many potential problems. Some problems that were not avoided, such as student nonresponse for specific questions, are discussed later. Our analyses indicate, however, that few major problems were encountered and that high reliability for the indices were achieved overall, as discussed earlier. 10 Chapter 4. Sample Design 4.1 Overview The study instruments were administered at 64 Schools, all of which are receiving SEA and CTLP programming, and most of which are receiving EDC programming. To maximize the power of the research, respondents are chosen in a stratified sampling design with random sampling within strata. 4.2 Sample Description In vocational schools, students are stratified in the first year of the study by ethnicity. Although considered a significant minority in the country, there are too few students of Turkish ethnicity to constitute their own strata for statistical inference. They therefore are grouped with Albanians in the data analysis. Turkish students were given the Macedonian language version of the survey and assessment, however. Ethnicity is thus defined as 1) Albanian or Turkish or 2) Macedonian or other ethnicities. In the first year, Gymnasia students are stratified by ethnicity as well as whether they are in schools in which faculty have been exposed to the new methodologies. This is the only part of the study for which an experimental group is used as a comparison with a control group in the first year of testing. Schools were stratified implicitly by size during sample selection. Information on student occupational cluster2 was not available in time for sampling. Sampling of the teacher population varied by school type. Teachers in the vocational schools were stratified by whether they were in a master teacher group (of which all members were surveyed) and their ethnicity. Teachers in the gymnasia were stratified according to whether or not they were in schools that had received training from CTLP, and by their ethnicity. This stratification will change in future years relative to changes in the projects’ statuses. Because it was not possible to identify teachers who did not complete the school year before testing, they were excluded after the sample was selected if they had not completed the school year. Albanians and Turks comprise about 20 percent of the students in Macedonia’s schools and 18 percent of the teachers. To assure accurate estimation among these respondents, we over sampled so that the sample drawn consisted of 30 percent Albanian and Turkish students and 33 percent Albanian and Turkish teachers. The sample drawn for students included an additional 20 percent of students by stratum to serve as potential replacements for absent students. The potential replacements were surveyed at the same time as the other students for simplicity of administration, but their results were used only as replacements within a stratum. The vocational schools include many different occupational clusters. In order to avoid potential biases caused by low response rates within occupational clusters, vocational classes in which over a third of students were absent were surveyed a second time. In the 2004 testing, were we unable to get the at least two-thirds of the class for only one class. We achieved our overall goal of over 85 percent response rate by stratum. For the first year, the response rate was over 90 percent for all strata except for Albanian and Turkish gymnasia students in schools that have not been served yet by CTLP. 2 The use of the term cluster follows the use of the term by the MOES and does not signify the statistical approach to clustering. 11 Post-stratification weights were calculated as one over the selection probability for that stratum. Each year, the study will administer the teacher survey to between 700 and 1,000 teachers and the student survey to between 1,000 and 1,200 students with the exception of 2004 for which we surveyed approximately 1,750 students to broaden the control group especially for the CTLP schools. These sample sizes are enough to pick up small to moderate changes in indices broken down by single variable (e.g., results by ethnicity, gender, etc) but not necessarily detailed cross tabulations (e.g., results by ethnicity and by gender simultaneously). AIR drew on multiple sources to develop the survey instruments. Among them were surveys of teaching practices developed for AIR’s evaluation of the federal Eisenhower Professional Development Program in the United States; classroom observation protocols developed by Newmann, Secada, and Wehlage to evaluate the alignment of instruction and assessment3 ; and the rubrics developed by Crawford and Mathews for use in Reading and Writing for Critical Thinking (RWCT) classrooms. Instruments developed by RWCT and CTLP also were used, and new questions were designed at AIR. The student problem-solving skill assessment was constructed from selected problem-solving units administered as part of OECD’s Programme for International Student Assessment 2000 test. The evaluation is concerned with the change in problem solving skills generally, and not defining separate scores for each sub-construct measured in the full assessment. Moreover, the process was already long for students when considering the length of the student survey. For these two reasons, only six units from the assessment were included. Most of these items contained more than one question. Scoring of student responses for these items followed OECD’s established methodology. When students are unfamiliar with standardized testing, reliability of responses will be higher with fewer options from which to choose. Due to low familiarity among Macedonian students with standardized testing, all questions were modified so that students generally faced a maximum of three columns. This limits somewhat the ability to discern changes in student responses over time to individual questions. AIR responded by adding questions where appropriate to gauge changes over time and to further allow the ability to measure reliability of responses. Item nonresponses for each individual were excluded from the calculations for index scores. If an individual answered fewer than a third of the questions for an individual index, that individual’s index score is not used in the analysis. This occurred for only a few individuals and indices, and the item nonresponse in general. Item nonresponse was higher among students especially toward the end of the survey, which probably is, in part, a function of length of survey after having completed the assessment. Item nonresponse was particularly high for question 21 asking students about use of computers at different locations for which many students apparently treated no response as a way of indicating no use of computers. Question 6c, which asks about older siblings is another example of nonresponse that should instead have been categorized as lack of a sibling. As one might expect, although there is a 3 Fred Newmann, Walter Secada, and Gary Wehlage, A Guide to Authentic Instruction and Assessment: Vision, Standards, and Scoring. Madison, WI: University of Wisconsin Center on Organization and Restructuring of Schools, 1995. 12 fourth column option to select if a student does not have a sibling, since many Macedonian students are unfamiliar with standardized tests, they did not use that column as a choice. 13 Chapter 5. Study Limitations The study faces a number of limitations. First, the study design does not link the student sample to the teacher sample. This was not possible due in part to concerns about teacher honestly if responses were not anonymous and due to the difficulty in linking secondary school students who have multiple teachers per year anyway. This means there will be no way of identifying whether or not a student taking the survey has been taught by a teacher who has received SEA or CTLP intervention. It will not be possible, therefore, to establish a direct causal relationship between individual teacher participation in SEA or CTLP programming and student outcomes on the assessment of problem solving skills. This limitation complicates the interpretation of differences in results between student and teacher surveys and creates a greater potential for discrepancies in results. Second, there is no independent, objective measure of the teachers’ self-reported use of the newer instructional methods. USAID-Macedonia required AIR to collect data to provide national-level estimates of the proportion of teachers using the methods that are a part of these projects. It was decided that a scientifically valid approach to measuring changes in instructional methods across hundreds of teachers in 64 schools would be cost prohibitive. It also would require more time than was available before baseline results were requested. Due to these considerations, objective measures of teacher use of these methods are not a part of the study design. Instead, aggregate teacher self-reported use of the methods will be compared with aggregate student responses. Third, the “power” of statistical tests depends on the number of students and teachers surveyed and the expected magnitude of change or difference. When comparing sub-groups, the study will have a lower probability of finding significant results even if changes actually are occurring. In order to conform to survey norms, the study will not compare low power groups. Finally, the types of teacher methodologies that are involved in these trainings are not entirely foreign to many Macedonian teachers. Some teachers who are interested and motivated in improving their methods may already have had some exposure to these methods. This raises the index scores in the baseline year for the vocational schools and may attenuate the ability of the tests to measure some potential effects of the projects. 14 Chapter 6. Operations 6.1 Administration Administration of the student surveys and student problem-solving test was performed by a trained team of 22 test administrators chosen from current students or graduates from a local university based on their study of and familiarity with social science research methodology. Test administrators first underwent a day of training. They then performed or observed a pre￾test of the survey in a classroom and then participated in a day of follow-up training. After this, they were deployed to schools around the country. A representative from the school administration or faculty was instructed to accompany each test administrator throughout the period. Consistency of test administration was assured through the use of school quality monitors. Briefed on the duties and responsibilities of test administrators, these monitors were deployed throughout the testing period such that each test administrator was monitored at least once during the first two days of testing and randomly thereafter. Quality monitors completed report forms for each test session monitored and returned these forms to the study team for review. Test administrators were informed if there were any suggestions for improvements in their delivery. Quality monitors also indicated any problems at the school that might affect student performance on the assessment or other issues of potential concern to the analysis. There were no cases in which a test administrator’s performance was deemed insufficient to exclude a whole class. Students first took the problem-solving skills test, which lasted 36 minutes. This was followed by a short break and then the survey. The problem-solving skills test was not counted if the student left any time during the test period. Unlike the student survey, where sessions were organized and held by test administrators, the teacher surveys were distributed directly to school coordinators in each school (to either the pedagogist or psychologist). The school coordinator was responsible for distributing the surveys to the selected teachers, collecting the completed surveys, and maintaining the safety and security of the surveys until SEA staff members visited the school to retrieve the surveys. Teachers selected to participate in the survey were given three days to complete the survey forms. 6.2 Scoring Both the teacher and student surveys were formatted such that the survey forms could be scored via optical scanner. Respondents provided answers by filling in ovals corresponding to response options. Following the administration of the student and teacher surveys, the survey forms were collected and shipped to AIR’s scanning facility in Sacramento, California. Data from the scanned forms was then electronically provided to staff in Macedonia for analysis. The process of constructing values for each index from the survey data is explained in detail in Appendix A. Scoring of the problem-solving test was organized with the assistance of experts from Belgrade, who had been trained to score the PISA test in 2002. These experts were invited to 15 train locally hired markers and observe the scoring process. Test markers and supervisors received instruction with intensive practical training on marking different types of answers. Test markers were organized into two groups. One group, consisting of 7 test markers and one supervisor, carried out the marking of Macedonian tests, while the other group of 4 markers and one supervisor marked the Albanian tests. Twenty-five percent of tests were double-checked by a second marker, while the remaining tests were double-checked by supervisors. Supervisors were also responsible for unifying different markers’ criteria in scoring similar answers, in order to maintain the same criteria for all markers and all subjects. 16 SECTION TWO: RESULTS FROM THE 2004 ADMINISTRATION 17 Chapter 7. Results 7.1 Overview The results of the 2004 assessment provide information on the progress toward Strategic Objective 3.4, “Students better prepared for employment through education programs” for the CTLP project and baseline measures against which subsequent years’ results will be compared for all three projects. As explained earlier, the task of measuring the combined effect of three different projects has been achieved through the use of index scores appropriate to the Intermediary Results and each project’s goals. The surveys were designed such that both teachers and students report on activities and elements of school life that are or will be affected by the efforts of the three USAID-funded education projects. Various strategies are being taken by these projects to better prepare students for future employment. The questions in the surveys, and the corresponding indexes derived from the survey questions, reflect these various strategies. Results from the student problem-solving test provide a measure of student problem-solving skills, which are being addressed through the teacher-training activities of both CTLP and SEA and, indirectly, by EDC. The discussion of results is organized by topic. These topics broadly correspond to the foci of the three projects. The first chapter in this section, students’ problem-solving skills, provides data from the student problem-solving test. The second chapter in this section provides results related to teachers’ use of improved instructional methods, their attitudes toward these methods, and students’ reports of their teachers’ use of such methods. These index measures are discussed second because they are the most immediate predictors of changes in students’ problem-solving skills. According to the conceptual approaches underlying CTLP and SEA, positive changes over time in teachers’ use of these methods, as measured by these index scores, will result in increases in the students’ problem-solving abilities and employability. The next chapter in this section discusses computer access, teachers’ use of computers for teaching, and students’ computer skills. These indices reflect efforts of the EDC project. It is hoped, however, that these index scores also will provide a synergistic effect on student problem solving skills through improved information and communication technology skills. Further, the use of computers increasingly are used as teaching tool and are integral to many career preparation related activities providing the potential for additional synergies with the ongoing SEA project. The last chapter in this section presents results for the career preparation index, which is a component only of the SEA project. All following chapters will present results in the form of indices and percentage responses to specific survey questions to provide a clearer image of the baseline condition. All indices are scaled to 100, with higher scores indicating greater exposure or higher self-reported skill levels. All differences reported are statistically significant at or below the .05 level unless otherwise indicated. Tables at the end of each section provide the t-test data for differences highlighted in that section. 18 7.2 Problem-Solving Skills The ultimate goal of USAID-sponsored teacher-training activities is to improve student problem solving skills in order to increase their employability. These skills are important tools that students need to become productive members of Macedonia’s workforce and that prepare Macedonia’s workforce for the careers of the future. The goal of the problem￾solving skills test is to measure changes in students’ problem-solving abilities between the baseline study year and the final year of the SEA project that result from the projects’ interventions. The scores presented here represent student performance on the problem-solving test and establish a baseline against which scores from the 2008 administration of this assessment will be compared. The results presented here primarily are designed as baseline measures for comparison with 2008 results rather than as analytical tools to examine potential problems with the school system, which is not within the scope of this study. In order to measure well a wide range of problem-solving scores, the test is designed so that the maximum score is expected to be reached infrequently by individual students. The aggregate scores reported are not internationally comparable. Figure 1. Average student problem-solving scores 22.1 17.2 27.8 0 10 20 30 40 50 60 70 80 90 100 All students Vocational Gymnasia Score The national average score for students in Macedonia is 22.1 out of 100. Gymnasia students performed better on this assessment than their peers in vocational schools. The average gymnasia score is 27.8, compared to 17.2 for vocational students. As a group, Macedonian students (vocational and gymnasia) scored higher than their Albanian counterparts, with an average score of 25.8 versus 14.1 for Albanian students. As a preliminary measure of the partial, early effect of the USAID-Macedonia interventions on S.O. 3.4, we compare results from the gymnasia programs in which CTLP interventions have occurred with results from other gymnasia. Among gymnasia students, a statistically significant difference in index scores is found between students in schools that have been receiving CTLP interventions for two years (since the 2002/2003 school year) and students in all other gymnasia. The latter group includes, in addition to schools that have not received any interventions, schools that have only begun to receive CTLP interventions in the 2003- 2004 academic year. 19 No difference is found, however, between all schools having received interventions from CTLP and the control group. The lack of a statistically significant difference when including the schools that have most recently received the CTLP intervention is hypothesized to be a function of the expected lag between the time of initial teacher training, changes in teacher methods, and resulting effects on student test scores. No baseline testing of problem solving skills was conducted in these schools before the CTLP intervention, however. Thus, it cannot be determined whether the students in schools receiving training from CTLP beginning in 2003-2004 began with a lower baseline skill level. There is some evidence supporting this contention, because the problem solving skills scores for the students in these schools are slightly lower than those for the control schools, although not at a point at which the difference is statistically significant. T-test information Index score Standard deviation Vocational students 17.2 (12.4) Gymnasia students 27.8 (18.6) Difference -10.6* Gymnasia students (CTLP 2002/2003) 31.7 (19.6) Gymnasia students with no intervention or CTLP intervention in 2003/2004 27.0 (18.4) Difference 4.7* Macedonian students 25.8 (17.6) Albanian students 14.1 (9.6) Difference 11.7* *Statistically significant difference at the 0.05 level. 7.2 Improved Instructional Methods A major thrust of both the CTLP and SEA projects is instructing teachers in improved instructional methods aimed at enhancing students’ critical thinking and problem solving skills. The teachers’ use of improved instructional methods index was created to measure the extent to which teachers are employing the methods, strategies, and approaches they have learned. This index measures the frequency of self-reported use of instructional practices that are consistent with the contextual learning approaches taught through USAID-funded education programs in Macedonia. The Teacher Instructional Practice Index incorporates the five higher-level objectives underlying CTLP and SEA programming: 1) problem-solving skills, 2) collaborative learning, leadership, and communication skills, 3) assessment and self￾assessment methods, 4) research and project-based learning, and 5) community- and work￾based learning. The index is composed of 26 items. At present, CTLP activities have been initiated in two cohorts of schools, with a third group beginning in the 2004 school year. SEA activities, however, have only just begun. The baseline score presented here for the improved instructional methods index therefore are expected to increase as CTLP’s third cohort of schools begins participation and as SEA’s train-the-trainer model expands to all 50 project schools. The baseline mean index score for all teachers on the Teacher Instructional Practice Index is 52.3. As an early gauge of future changes in teaching methodologies, a second index, the Teacher Attitudes Index, measures the extent to which teachers agree with or view favorably several 20 of the key practices, methodologies, and beliefs associated with the contextual learning approaches taught through the education programs funded by USAID/Macedonia. Teachers’ positive attitudes toward the methods, measured by the Teachers Attitudes Index, are expected to increase as the two projects progress. Changes over time in the Teacher Attitudes Index are hypothesized to predict increases in teacher use of the methods that compose the Teacher Instructional Practice Index. Interestingly, support is not found for the hypothesis that differences in attitudes (Teacher Attitudes Index) between groups in the baseline year translate into differences in teaching practice (Teacher Instructional Practice Index) across the board. The one exception is teachers who teach Macedonian students, compared to those who teach Albanian students. Teachers of Macedonian students have an average of 66.1 on the Teacher Attitudes Index, versus 58.4 for teachers of Albanian students. Teachers of Macedonian students also report using the sponsored instructional methodologies to a greater extent, scoring 54.0 on the Teacher Instructional Practice Index compared to 48.6 for teachers of Albanian students. Other significant differences in the Teachers Attitudes Index: • Teachers at or below the age of 40 hold more favorable views toward the improved instructional methodologies than teachers above age 40. • Teachers of vocational or both vocational and gymnasia classes have more favorable attitudes towards these methods than teachers who teach only gymnasia classes. The teacher survey also asked teachers whether any particular factors posed as obstacles to their use of the instructional practices taught through CTLP and SEA. These questions are meant primarily as diagnostics for future years, but the current responses are interesting and suggest potential obstacles for the future. About 60 percent of all teachers surveyed reported that having too many students in the classroom or having and insufficient classroom size was an obstacle to their use of these methods (see figure 2). Indeed, teachers with 30 or more students in their classes scored higher on a combined “obstacles” index than teachers with fewer than 30 students, indicating that they identify more obstacles to use of the methods and approaches. The lack of equipment or materials, cited by 58 percent, also appears to be an obstacle that may hamper use of these methodologies in the future for SEA. 21 Figure 2. Obstacles to use of improved instructional methods 59,9 58,2 40,8 24,2 20,5 20,2 11,8 8,7 0 1020 30405060 70 8090 100 Too many students in classroom or insufficient classroom size I do not have the necessary equipment or materials There is not enough flexibility in curriculum to make room for these instructional practices Inadequate teacher training support on how to use these instructional practices in the classroom I have not been trained in most of these instructional practices Classtime is too short These instructional practices do not apply to most of my courses I don't have time to practice these isntructional practices Percent of teachers responding that this factor was an obstacle Data from the teacher survey is complemented by information gathered from the student survey. Students were asked several questions about their teachers’ use of instructional methods, approaches, and strategies which correspond to the training being provided by USAID-sponsored projects. This Student-reported Instructional Practice Index incorporates five higher-level objectives underlying CTLP and SEA programming mentioned above. The index is composed of 33 items. Higher scores on this index indicate that students are exposed to the instructional methods and strategies encouraged by the CTLP and SEA projects more frequently. The national baseline score for the Student-reported Instructional Practice Index is 47.1. Vocational students provide higher scores on this index for their teachers than do gymnasia students. The score for vocational students is 47.9 compared to 46.2 for students in gymnasia schools. As the SEA and CTLP teacher-training programs reach greater numbers of teachers in the next four years, the Student-reported Instructional Practice Index scores are expected to increase. T-test information Index score Standard deviation Teacher instructional methods index Teachers of Macedonian students 54.0 (12.5) Teachers of Albanian students 48.6 (15.0) Difference 5.4* Teachers’ attitude index Teachers of Macedonian students 66.1 (12.4) Teachers of Albanian students 58.4 (16.7) Difference 7.7* Younger teachers (below age 40) 65.7 (12.3) Older teachers (age 40 or older) 63.3 (13.6) Difference 2.4* 22 Gymnasia only teachers 62.9 (13.2) Vocational or vocational and gymnasia teachers 66.8 (11.6) Difference -3.9* Obstacles to use of methods index Class size 30 or more students 43.2 (23.9) Class size less than 30 students 32.9 (20.3) Difference 10.3* Student-reported instructional methods index Vocational students 47.9 (11.3) Gymnasia students 46.2 (10.9) Difference 1.7* *Statistically significant difference at the 0.05 level. 7.4 Computer Access, Computer Skills, and Computer Use for Teaching Computers are becoming increasing integral to the modern school experience, both as a tool used by teachers and as a portal for students to access vast information resources. Computers also are important to the work environment, making it crucial for students to gain the computer skills to allow them to be productive in the workplace. Already computers have a presence in the Macedonian education setting, although this presence is still somewhat limited. Just fewer than 30 percent of teachers (29.6 percent) report that they have a computer in their classroom or the annex to their classroom. Thirty four percent of students who responded to the question report using a computer at school at least once a month and about half of those used computers at school at least once a week. However, this probably is a slight overestimate. This is because many students indicated to the test administrators during or after testing that they did not respond to indicate that they never used computers at school. If half of the students who did not respond to this question (450) did so because they never used computers at school, then only thirty percent of Macedonian secondary students use computers at school at least once a month. Still, a great number of students and teachers do not have adequate access to computers or to the Internet. More than 80% of students report that it is “somewhat difficult” or “very difficult” to get access to computers at school, or that they have no access at all to computers. Indices created from both the teacher and student surveys provide an indication of the level of computer access in Macedonian schools. The Teacher Computer Access Index is comprised of five items, which ask teachers whether they have access to a computer in their classroom or classroom annex; the frequency with which they have access to various computer resources such as printers, Internet, or school computer labs; and whether too little access to computer labs is an obstacle to the use of computers for teaching. Nationally, the average score for the teacher computer access index is 31.1. The Student Computer Access Index includes items that ask students how often they use computers at school, home, or other places and how frequently computers are made part of the learning experience both inside and outside of the classroom. The index also reflects students’ access to the Internet at school, home, and other places. This index is composed of 13 items. The average computer access index score for all students is 55.0. Students in gymnasia schools score higher on the computer access index than students from vocational schools, 23 with an average score of 57.1 compared to 53.3 (figure 3). Albanians in gymnasia report slightly higher access to computers than do Macedonians, while the opposite is true among vocational students. Figure 3. Index scores for computer access, by student characteristics 53.3 54.0 51.5 57.1 55.5 60.5 55.0 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 Total Total Macedonian Albanian Total Macedonian Albanian All students Vocational students Gymnasia students Index score Access to computers can impact both teachers’ abilities to use computers for teaching, and students’ abilities to gain important computer skills. The computer use for teaching index measures how often teachers ask their students to use computers for various tasks, assignments, and projects. This index is composed of nine items. The average index score for all teachers is 18.0. As shown in table 1, some of the more popular ways in which teachers have students use computers are for reports and term papers, Internet research, and the creation of graphs or charts. Differences in the extent to which teachers use computers for teaching are seen between teachers who instruct only gymnasia students, and those who teach vocational students or a mix of vocational and gymnasia students. The mean index score for gymnasia-only teachers is 19.7 compared to 16.0 for teachers of vocational or vocational and gymnasia students. 24 Table 1. Number of courses in which teaches ask students to use computers for the following tasks/assignments. Not relevant to my classes None One or two Three or four Five or more n a) Research using the Internet 18.0 35.2 18.9 9.5 18.4 800 b) Reports and term papers 14.5 30.4 30.4 12.6 12.1 756 c) Multimedia projects or music 30.5 45.0 12.0 6.7 5.8 717 d) Web pages, web sites, etc. 24.6 49.8 12.8 7.1 5.7 719 e) Artwork, photography or video products 14.1 33.8 22.0 14.0 16.1 740 f) Graphs or charts 15.9 33.4 20.0 12.4 18.3 728 g) Calculations or information using a spreadsheet 31.7 46.6 12.6 3.9 5.2 713 h) Reports or information derived from databases 24.0 43.2 19.4 8.5 4.9 720 i) Slid show presentations 17.0 47.6 17.2 8.9 9.3 733 Students’ acquisition of important computer skills is measured through the student computer skills index. This index reflects students’ responses to questions asking about their ability to use various types of computer programs and their ability to perform tasks related to research, communication, and document creation. This index is composed of 23 items. The average index score for all students is 41.1. Table 2 shows students’ self-reported skill level in regard to common computer-related tasks. Table 2. Percent of vocational students reporting whether they have participated in the following activities during school. I don’t know what this is I can do this but sometimes need help I can do this completely by myself n a) Use word processing software (e.g., Word®, Works®, WordPerfect®, etc.) to create or edit a document such as a paper or letter 26.2 41.8 32.0 1613 b) Use databases (e.g., Access®, Oracle®, etc.) to add data, sort information or create reports 46.3 42.7 11.0 1593 c) Use a spreadsheet (e.g., Excel®, Works®, etc.) to enter and calculate numbers 19.2 40.6 40.2 1598 d) Attach files to and send email messages 29.9 30.2 39.9 1594 e) Find information on the Internet using a search engine 18.2 35.2 46.6 1585 f) Maintain a website (use programming software, upload materials, etc.) 53.5 32.6 13.9 1579 The E-Schools project is working to bring IT resources, including Internet connectivity and relevant applications, to both gymnasia and vocational secondary schools. The project is also teacher-training in methodologies for IT literacy instruction. As result of these efforts it is 25 expected that index scores for both computer access and computer skills will increase in future years. T-test information Index score Standard deviation Student computer access index Vocational students 53.3 (14.6) Gymnasia students 57.1 (16.7) Difference -3.8* Teachers’ use of computers for teaching index Gymnasia only teachers 19.7 (20.4) Vocational or vocational and gymnasia teachers 16.0 (19.3) Difference 3.7* *Statistically significant difference at the 0.05 level. 7.5 Career Preparation The Career Preparation Index measures the extent to which students have been exposed to and have participated in activities aimed at preparing them for employment. It reflects students’ responses to items asking them the frequency and extent to which they have been taught job-seeking skills at school, have learned about job markets and professions, and have participated in activities that provide practice in career-related skills. The Career Preparation Index is comprised of 19 items. This index is used to analyze only the vocational students because career preparation is an element of only the SEA project, which focuses on vocational schools. The average index score for all vocational students is 44.8. This score is expected to increase as the SEA project works to create career centers in all of its 50 project schools. A more detailed picture of the baseline condition is revealed in student responses to specific survey questions. Over half (56.2%) of vocational students have been taught how to interview for a job, while nearly three quarters of vocational students (73.8%) have received instruction in how to write a resume or CV. When asked whether school had helped them become better informed about job markets in potential future professions, 59.8% of students responded that they had learned about job markets in their local area or municipality. A key element in SEA’s support to vocational schools is assistance in creating real or virtual firms within schools so as to offer students opportunities to utilize the skills and knowledge being taught in the classroom. At present, 12.9% report having participated in a real school business or firm. Less than half of all vocational students have participated in career guidance discussions or activities. 26 Table 3. Percent of vocational students reporting whether they have participated in the following activities during school. No Yes Don’t know what this is n a) School business or firm 61.7 12.9 24.4 877 b) Virtual school business or firm 61.8 5.1 33.1 869 c) Practical work outside the school 31.8 67.0 1.2 869 d) Participate in career guidance discussions/activities 55.5 42.1 2.4 879 e) Role playing about different job situations 72.0 25.6 2.4 883 27 Appendix A. Construction of indices To construct the indices presented in this report, the study team first developed a system for assigning points to survey responses. For every item or sub-item, a point value is assigned to each possible response. These point values range from 0 or 1 to 2, 3, or higher, depending on how many possible responses there are. At all times, point values are assigned to reflect an answer’s degree of consistency with the goals, methods, and attitudes associated with the CTLP, SEA, and EDC projects, with higher numbers representing greater consistency. In the student survey, higher values also reflect higher self-reported skill levels and greater exposure to the instructional methods taught by these USAID-sponsored projects. Point scales for items may be increasing or decreasing based on the phrasing of the question. To illustrate, where questions ask the frequency of use of project-supported methods, a response of “very often or always” would receive a higher point value than “never” or “sometimes.” The opposite is true, however, if questions ask about methods that are inconsistent with project goals. In such cases, a response of “never” would receive the highest point value. After coding survey responses in this manner, the study team then chose relevant items, or sub-items, to be included in each of the indices. The maximum value for each index was computed by summing the maximum point values of the component items or sub-items. The maximum value of each index was scaled to 100 for reporting purposes, with respondents’ scores for that index being scaled by the same factor. The following paragraphs discuss the survey questions included in each of the indices presented in this report. Student: Career Preparation Index The index of career preparation is derived from student responses to four questions asking them if they had been taught skills or been involved in activities related to career preparation. Students were asked if school had helped to inform them of the job market or potential professions in their local area or municipality, across Macedonia, and internationally. Students responded yes or no to whether, in school, they had been taught how to interview for jobs; write a resume; search for and find jobs; write a letter to apply for a job; determine the skills needed for a future profession; understand manuals, diagrams, memoranda, and other job-related documents; and assess his or her performance in a job. A third question asked students whether they had ever participated during school in a school business or firm, a virtual school business or firm, practical work outside the school, career guidance discussions/activities, and role playing about different job situations. Student could respond “yes” “no,” or “don’t know what this is.” These same answer categories were available for the final question in this index, asking students if they had ever developed a portfolio, job shadowed, prepared a business plan, or participated in career exploration activities. 28 Student: Computer Access Index The computer access index is derived from student responses to five questions. Students were first asked the frequency with which they used computers at home, school, or other places. The three response categories were “less than once a month,” “between once a week and once a month,” and “at least once a week.” Students were asked how many times during their years in high school they had used computers during or outside of class period for the following subjects: science or mathematics; humanities, social sciences, or languages; and vocational courses (for vocational students only). Response categories were 0-5 times, 6-20 times, or more than 20 times. The third question asked students how many of their high school professors had required them to use a computer either during class or outside of class at least two times during the school year. Gymnasia students were asked to answer this question for their gymnasia courses. Vocational students were asked to answer this question for both their general education courses and their vocational education courses. The response categories for this question were “0-2 professors,” “3-10 professors,” “11 or more professors,” and “Does not apply to me.” The fourth question asked students how difficult it is for them to use a computer at school when they want to. Students could respond “Easy to get access to school computer,” “Somewhat difficult,” or “Very difficult or no access to school computers.” The final question in this index asked students whether they had access to the Internet at home, at school, or other places. Students responded “yes” or “no” to this question. Student: Computer Skills Index The student computer skills index is derived from student responses to a series of questions asking them to identify their proficiency in performing computer-related tasks. For each question, the possible responses were “I don’t know how to do this,” “I can do this but sometimes need help,” and “I can do this completely by myself.” The tasks were listed according to category. Under general tools, students were asked if they can start and run programs or software, delete a computer document or file and copy a file from one location to another, and print documents. Tasks associated with documents, databases and programming were: use word processing software (e.g., Word®, Works®, WordPerfect®, etc.) to create or edit a document such as a paper or letter; use databases (e.g., Access®, Oracle®, etc.) to add data, sort information or create reports; use a spreadsheet (e.g., Excel®, Works®, etc.) to enter and calculate numbers; use a word processor, spreadsheet, or other program to create graphs, charts, or tables; create macros, de-bug computer programs, or write computer programs; and diagnose and fix problems with a computer. Multi-media tool tasks: Create a slide show presentation (e.g., PowerPoint®); use image￾editing software (PhotoShop®, etc.) to alter or enhance photographs or images; create a multi￾media presentation (that might include text, sound, pictures or video) (e.g., using Hyperstudio® or HyperCard®); use web authoring software (e.g., Dreamweaver, Netscape Composer®, Frontpage, or other software) to create web pages; upload images to a computer from a scanner, camera, or other source; use MIDI software to create or alter music; and, use Computer Aided Design software or Computer Aided Manufacturing software. Research and communication tasks: attach files to and send email messages; find information on the Internet using a search engine; chat, use instant message, or subscribe to newsgroups; download files from the Internet and access them from you computer; maintain a website (use 29 programming software, upload materials, etc.); use probes (e.g., temperature sensor, motion detector) and probe software to collect and analyze information; and, help set up or troubleshoot problems with computer networks. Student: Student-reported Teacher Use of Improved Instructional Methods Index The student-reported teacher use of improved instructional methods index is derived from student responses to several questions in the survey. Students were asked how many of their professors, throughout their years of high school, were using the following activities: having students work on assignments with others in small groups; having students discuss among themselves what they read or observe; some or all students help lead their group to complete the project; having students work together to complete classroom assignments; having students explain how to reach solutions or decisions in writhing or orally; having students debate ideas with or explain their reasoning to the class; encouraging all students to participate in class discussions; and, encouraging student presentations or interesting writing assignments. Possible response categories were “0-5 professors,” “6-10 professors,” or “11 or more professors.” The response categories were the same for a second question asking students how many of their professors during all their years of high school were using the following specific strategies: class work that includes problems or issues with no immediately obvious solution or answer; assignments that require students to organize information (comparing, classifying, ordering, etc.); assignments that require students to figure out for themselves the best way to deal with complex issues or situations; assignments that require troubleshooting why something does not work properly or diagnosing why a situation does not go as planned; encouraging students to discuss what they already know about a topic before the lesson begins; having students write questions they would like answered before the professor starts a presentation; and, having students create their own questions based on their lessons. The same response categories were again available for a question asking students for how many of their professors, during all their years of high school, had the following occurred in class: “my professors show me what to do to improve my work,” “I review and revise my own work,” “professors involve students in determining how well the students are doing in class,” and “I set my own goals for learning.” Further questions asked students to what extent they had been taught to develop particular skills through school or classroom activities. Students could respond “not at all,” “somewhat,” or “a lot” for the following skills: how to begin, conduct, and complete large projects (at least one week long); how to speak in front of others; how to identify and use resources (human, financial, informational) for projects; and, how to write reports or technical documents. Students were asked whether or not they had been involved in the following types of projects through school: projects in general education or gymnasia courses that last a week or more; projects in vocational education courses that last a week or more (an additional response category of “does not apply to me” was available for this item); projects that involve interaction with community members; projects that involve interaction with businesses; and, research using computers, libraries, or community resources. Lastly, students responded whether or not they had ever been taught the following skills through school: how to interview for jobs; how to write a resume; how to search for and find jobs; how to write a letter to apply for a job; how to determine the skills needed for a future 30 profession; how to understand manuals, diagrams, memoranda, and other job-related documents; and, how to assess one’s own performance in job. Student: Problem-solving Skills Index The student problem-solving skill index was created using student scores on the problem￾solving test that was administered to students. This test was constructed from six problem￾solving units administered as part of OECD’s Programme for International Student Assessment 2000 test. Most of these items contained more than one question. Scoring of student responses for these items followed OECD’s established methodology. The six items are not included with this report, as these same questions will be used again in the 2008 administration. Teacher: Attitudes Toward Improved Instructional Methods Index The teacher attitudes toward improved instructional methods index is derived from teacher responses to whether they “strongly agree,” “agree,” “disagree,” or “strongly disagree” with the following statements: “it is best when students work on assignments alone to show how much they know;” “all students should be encouraged to participate in class discussions even if they don’t know the answer;” “teachers know more than students and should just explain the facts directly;” “teachers should provide feedback to students on assignments to show them how to improve their work;” “classes should be focused on problems with specific, correct answers and ideas that students can grasp quickly;” “student projects often result in students learning ‘incorrect knowledge’;” and, “all real learning from school happens in the classroom.” Teacher: Use of Improved Instructional Methods Index The teacher use of improved instructional methods index is derived from teacher responses to four questions. The first of these asked teachers to indicate how often and when they ask students questions as well as whether they are trying to accomplish the following goals: see if students know the correct answer; elicit/draw out students’ ideas and opinions; get students to justify and explain their reasoning; and, have students relate what they are working on to their own experiences. The response options were “never,” “sometimes,” “often,” and “very often or always.” Teachers were asked how important various factors were in determining students’ grades in the subjects taught by that teacher. This question did not pertain to the matura or final exams. Teachers were asked to indicate the importance of the following: questioning; performance on experiments, projects, or other hands-on experiences; written reports or essay; and, oral presentations other than questioning. Response options for this question were “not important,” “minor importance,” “moderate importance,” and “very important.” Teachers were further asked to indicate which of the following activities are done by students in some of their classes (teachers were instructed to mark all that apply): students conduct projects that require them to interact with community members outside of school; students conduct projects that require them to interact with businesses outside of school; students conduct projects were they must use libraries or community resources; students conduct projects that last a week or more; students review and revise their own work; students have opportunities to set their own goals for learning or criteria for grades; students learn how to 31 understand job-related documents such as manuals, diagrams or memoranda; and, students learn how to assess their own performance in a job. The final question in this index asked teachers to indicate how often, if at all, they included the following instructional practices in some of their classes: class work includes problems or issues with no immediately obvious solution or answer; assignments require students to troubleshoot why something does not work properly or diagnose why a situation does not go as planned; students discuss what they already know about a topic before the lesson begins; students write questions they would like answered before the teacher starts a presentation; students create their own questions based on their lessons; students work on assignments in groups; students break into groups to discuss among themselves; students debate ideas or discuss their reasoning with other students; students compare their thoughts to those of authors or others; and, providing feedback on student assignments on how to improve. The scores for this index should be interpreted with care, as they are based on self-reports and cannot be verified independently. Teacher: Computer Use for Teaching The teacher computer use for teaching index is derived from teacher responses to a single question asking how often they asked their students to perform various tasks or create products using computers. Frequency was determined by the number of courses in which teachers asked students to use computers for: research using the Internet; reports and term papers; multimedia projects or music; web pages, web sites, etc.; artwork, photography or video products; graphs or charts; calculations or information using a spreadsheet; reports or information derived from databases; and, slide show presentations. Response categories were “not relevant to my classes,” “none,” “one to two courses,” “three or four courses,” or “five or more courses.” Teacher: Obstacles to Use of Improved Instructional Methods The obstacles to use of improved instructional methods index is derived from teacher responses to whether any of the following potential obstacles to using such practices applied to them: “I have note been trained in most of these instructional practices;” “these instructional practices do not apply to most of my courses;” “I don’t have time to practice these instructional practices;” “inadequate teacher training support on how to use these instructional practices in the classroom;” “there is not enough flexibility in curriculum to make room for these instructional practices;” “I do not have the necessary equipment or materials;” “too many students in classroom or insufficient classroom size;” and, “class time is too short.” Teacher: Computer Access The teacher computer access index is derived from responses to three questions related to teachers’ access to computers and related accessories. Teachers were asked whether or not they have a computer in their classroom or annex to their classroom. They were further asked how often they have access to computers reserved for teachers, computers in the school computer lab, computer printers at school, and the Internet at school. Possible responses were “never tried to get access or not interested,” “never,” “less than once a month,” “one to 32 four times a month,” or “more than once a week.” Finally, teachers responded whether or not “too little access to computer labs” is an obstacle to their use of computers for teaching. 33 Appendix B. Data tables Teacher Survey Survey response distributions, expressed as a percent of total responses. 1. What is your age? 30 years or younger 31-40 years 41-50 years 51-60 years 61+ years n 15.1 30.9 27.5 23.3 3.3 881 2. What kind of students do you teach? (Mark all that apply.) Gymnasium Vocational Both n 59.4 28.0 12.6 875 3. What kind of subjects do you teach? (Mark all that apply.) Natural sciences or mathematics Humanities, social sciences, or languages (e.g., psychology, history, literature, arts) Information technology/ computers Vocational subjects Practical training n 32.8 48.9 3.7 15.3 9.2 881 4. What is the highest level of schooling that you have completed? Secondary degree Some higher education University degree Some post￾baccalaureate education or completed degree n 0.9 2.6 89.5 7.0 878 5. How many years have you been a classroom teacher? 0-5 years 6-10 years 11-20 years 21 years or more n 21.9 23.6 23.8 30.7 877 6. What one language do you speak the most often at home? Macedonian Albanian Turkish Roma Serbian Vlach Other n 67.2 30.8 1.3 0.1 0.1 0.1 0.4 860 34 7. What languages do you teach in? (Mark all that apply.) Macedonian Albanian Turkish English Other n 67.2 28.8 1.5 8.7 4.9 891 8. What is the size of most of your classes? 15 or fewer students 16-24 students 25-29 students 30-34 students 35-39 students 40 or more students n 2.8 4.1 10.8 56.3 22.5 3.5 832 9. What is your gender? Male Female n 45.5 54.5 873 10. Please indicate the degree to which you agree with each statement. Strongly agree Agree Disagree Strongly disagree n a) It is best when students work on assignments alone to show how much they know. 27.9 57.2 13.2 1.7 859 b) All students should be encouraged to participate in class discussions even if they don’t know the answer 55.6 41.2 2.4 0.8 866 c) Teachers know more than students and should just explain the facts directly. 19.5 31.1 38.9 10.5 853 d) Teachers should provide feedback to students on assignments to show them how to improve their work. 66.8 31.9 0.7 0.6 865 e) Classes should be focused on problems with specific, correct answers and ideas that students can grasp quickly. 46.8 43.8 8.1 1.3 861 f) Student projects often result in students learning “incorrect knowledge." 2.7 18.7 67.1 11.5 845 g) All real learning from school happens in the classroom. 6.3 27.1 53.4 13.2 855 35 11. Teachers may have one or more goals when asking students questions. When you ask students questions, how often do you try to accomplish the following goals? Never Sometimes Often Very often or always n a) See if students know the correct answer 1.3 13.3 48.9 36.4 854 b) Elicit/Draw out students’ ideas and opinions 0.9 7.9 39.9 51.3 859 c) Get students to justify and explain their reasoning 0.5 8.4 43.5 47.6 853 d) Have students relate what they are working on to their own experiences 1.4 13.8 37.0 47.8 855 12. Not including the matura and final exams, how important are the following in determining your students’ grades in the subjects you teach? Not important Minor importance Moderate importance Very important n a) Questioning 3.0 9.5 47.3 40.2 862 b) Performance on experiments, projects, or other hands-on experiences 1.2 6.0 31.0 61.8 850 c) Written reports or essays 3.8 11.4 40.9 43.9 845 d) Oral presentations other than questioning 1.7 6.0 30.3 62.0 844 13. Please indicate if your students do the following in some of your classes: (Mark all that apply.) Yes n a) Students conduct projects that require them to interact with community members outside of school 32.1 901 b) Students conduct projects that require them to interact with businesses outside of school 22.5 901 c) Students conduct projects where they must use libraries or community resources 53.6 901 d) Students conduct projects that last a week or more 32.7 901 e) Students review and revise their own work 61.0 901 f) Students have opportunities to set their own goals for learning or criteria for grades 47.9 901 g) Students learn how to understand job-related documents such as manuals, diagrams or memoranda 37.0 901 h) Students learn how to assess their own performance in a job 57.8 901 36 14. Please indicate how often, if at all, you include the following instructional practices in some of your classes. Never Less than once a month 1-3 times a month At least once a week n a) Class work includes problems or issues with no immediate obvious solution or answer 38.4 31.4 13.3 12.3 765 b) Assignments require students to troubleshoot why something does not work properly or diagnose why a situation does not go as planned 20.3 35.7 21.3 16.1 765 c) Students discuss what they already know about a topic before the lesson begins 5.9 18.3 21.6 45.3 796 d) Students write questions they would like answered before I start a presentation 22.8 24.7 19.4 20.1 766 e) Students create their own questions based on their lessons 4.7 13.5 20.3 51.6 792 f) Students work on assignments in groups 7.4 24.3 27.1 32.5 786 g) Students break into groups to discuss among themselves 11.5 24.7 25.1 30.3 783 h) Students debate ideas or discuss their reasoning with other students 6.6 26.3 30.0 28.3 783 i) Students compare their thoughts to those of authors or others 17.7 31.3 21.5 22.4 776 j) I give feedback on student assignments on how to improve 0.7 7.6 20.4 71.3 815 15. What is your level of interest in the instructional practices mentioned in questions 13 and 14? High – I am interested in many of these instructional practices Medium – I have some interest in some of these instructional practices Low – I am not interested in most of these instructional practices n 65.2 33.1 1.7 840 16. Indicate whether any of the following potential obstacles in using the instructional practices described in questions 13 and 14 apply to you. (Mark all that apply.) Yes n a) I have not been trained in most of these instructional practices 20.5 901 b) These instructional practices do not apply to most of my courses 11.8 901 c) I don’t have time to practice these instructional practices 8.7 901 d) Inadequate teacher training support on how to use these instructional practices in the classroom 24.2 901 e) There is not enough flexibility in curriculum to make room for these instructional practices 40.8 901 f) I do not have the necessary equipment or materials 58.2 901 g) Too many students in classroom or insufficient classroom size 59.9 901 h) Class time is too short 20.2 901 37 17. How often did you have the following types of interaction this year? Never or does not apply Less than once a month 1-3 times a month At least once a week n a) Discussions with other teachers about how to teach a particular concept to a class 9.2 37.4 30.3 23.1 835 b) Discussions with other teachers about ideas for student projects or sharing examples of student projects 16.8 41.6 26.5 15.1 800 c) Discussions with teachers at other schools through professional associations 51.9 36.8 7.5 3.8 744 d) Discussions with teachers at other schools through associations of schools with the same vocations 52.4 37.0 7.0 3.6 759 e) Discussions with representatives of local companies or industry 80.3 14.5 2.6 2.6 736 18. Please indicate the degree to which you agree or disagree with each statement about your school. Strongly agree Agree Disagree Strongly disagree n a) Teachers at my school have the pedagogical training they need to improve their teaching 28.4 53.6 13.9 4.1 825 b) I enjoy being a teacher 67.4 27.9 3.1 1.6 828 c) Students’ training in my school is better now than it was five years ago 15.2 43.5 31.4 9.9 818 d) Students are more likely to get jobs when they graduate because of what they have learned in school 40.6 53.6 3.4 2.4 828 e) Students are more likely to get into university when they graduate because of what they have learned in school 20.1 40.8 30.6 8.5 823 19. Do you have a computer in your classroom or the annex to your classroom? Yes No n 29.6 70.4 901 20. How often did you have access to the following devices this year? Never tried to get access or not interested Never Less than once a month 1-4 times a month More than once a week n a) Computer reserved for teachers 10.3 37.8 18.6 12.1 21.2 823 b) Computer in the school computer lab 10.3 44.2 17.5 12.3 16.7 787 c) Computer printers at school 7.6 37.9 24.5 14.8 15.2 792 d) Internet at school 11.2 53.3 15.3 8.3 11.9 792 38 21. In how many of your courses this year did you ask students to produce the following using computers? Not relevant to my classes None One or two Three or four Five or more n a) Research using the Internet 18.0 35.2 18.9 9.5 18.4 800 b) Reports and term papers 14.5 30.4 30.4 12.6 12.1 756 c) Multimedia projects or music 30.5 45.0 12.0 6.7 5.8 717 d) Web pages, web sites, etc. 24.6 49.8 12.8 7.1 5.7 719 e) Artwork, photography or video products 14.1 33.8 22.0 14.0 16.1 740 f) Graphs or charts 15.9 33.4 20.0 12.4 18.3 728 g) Calculations or information using a spreadsheet 31.7 46.6 12.6 3.9 5.2 713 h) Reports or information derived from databases 24.0 43.2 19.4 8.5 4.9 720 i) Slide show presentations 17.0 47.6 17.2 8.9 9.3 733 22. How often do you do the following? Never Once or twice a year Up to once a month At least once a month n a) Use a computer or the Internet as part of your instruction 56.8 15.3 11.6 16.3 798 b) Make handouts for students using a computer 23.8 20.7 21.8 33.7 793 c) Create a test or assignment using a computer 21.3 24.4 21.6 32.7 804 d) Email for professional use 68.4 12.5 7.7 11.4 754 23. What is your level of interest in using computers for your teaching or preparation? High – Very interested Medium – Moderate interest Low – No interest n 55.1 37.9 7.0 849 39 24. Indicate whether any of the following potential obstacles to using computers in teaching apply to you. (Mark all that apply.) Yes n a) Too little access to computer labs 46.4 901 b) Too few computers for my students 49.3 901 c) Not comfortable enough with computers to use them for teaching 31.7 901 d) Not sure how to make technology relevant to my subject 19.9 901 e) Insufficient training on how to use technology for classes 37.0 901 f) Computers are too unpredictable or outdated, or the software is inadequate or does not work right 12.0 901 g) No access to printer or the printer frequently is not working or out of paper 26.5 901 h) There is not enough flexibility in curriculum to make room for technology 37.2 901 25. Who in your school makes decisions about the following issues (not according to law but in everyday experience). (Mark all that apply for each row.) Do not know MOES or BDE School Board School Director The Mayor Teacher Council Individual teachers n a) Suggesting which school director to hire 25.3 26.6 28.0 4.1 3.0 9.3 5.1 901 b) Appointing the director 17.4 45.8 16.2 2.5 3.1 6.7 2.7 901 c) Deciding which profiles, smer, or nasoka are offered4 8.4 34.3 10.6 20.3 1.4 27.9 5.3 901 d) Determining course content 11.9 53.6 5.4 4.1 1.3 11.1 9.1 901 26. Your school director currently exercises which of the following responsibilities? (Mark all you believe apply, if any.) Paying teacher salaries Hiring teachers Formulating the total school budget Deciding on budget allocations within the school n 24.2 74.6 33.0 54.4 901 27. Mark which of the following two groups, if either, are informed about school budgets and spending choices by the school director or school board? Teachers Parents Both Null n 51.5 5.9 9.1 33.5 901 4 Profiles are vocational professions that include a set of courses. Smer or nasoka are the parallel for gymnasia. 40 28. Which of the following actually help teachers at your school in their professional development? (Mark all that apply.) Yes n a) MOES inspectors 14.3 901 b) BDE advisors 33.3 901 c) School Director 30.5 901 d) School Pedagogue 37.5 901 e) Teacher Council 15.3 901 f) Parent Council 5.4 901 g) Other teachers within school 27.6 901 h) Union of Employees in Education, Culture, and Science 5.2 901 i) School associations 8.2 901 j) Foreign associations 33.3 901 29. Which of the following does your Teacher Council of Councils do? (Mark all that apply.) Yes n a) We do not have a Teacher Council or it is inactive 4.9 901 b) I do not know much about the Teacher Council 7.0 901 c) It makes recommendations to school board or school director 54.8 901 d) It negotiates or makes decisions together with school board or school director 44.3 901 e) It fights with school board or school director 14.5 901 f) It does whatever the school board or school director tells it to do 17.8 901 g) It recommends school budget priorities 16.9 901 h) It coordinates with other school associations or bodies (e.g., parent council or other teacher councils) 29.4 901 i) It coordinates with Union of Employees in Education, Culture, and Science 28.3 901 41 30. What changes do you think are needed to improve the interaction between school directors and teacher councils? (Mark all that apply.) Yes n a) No changes are needed 10.1 901 b) Changes in the laws to better define the role of teacher councils 45.7 901 c) Changes in the regulations to better define the role of teacher councils 40.0 901 d) Training for teacher councils about how to assert their roles 52.3 901 e) Training for school directors about the proper roles of teacher councils 39.1 901 f) Other 5.9 901 31. Please check all of the professional development programs in which you have participated in the last six years? (Mark all that apply.) Yes n a) I have not participated in professional development courses or workshops in the last 6 years 25.3 901 b) Phare (VET I) 9.9 901 c) GOPA (VET II) 3.4 901 d) GTZ 1.5 901 e) Junior Achievement 3.7 901 f) ECO NET 3.1 901 g) SOROS (Reading and Writing for Critical Thinking) 29.3 901 h) Secondary Education Activity (SEA) 16.2 901 i) E-Scola (Computer-IT) 11.6 901 j) World Bank Education Modernization 2.2 901 k) CRS (Civic Education) 7.3 901 l) CRS (Computer-IT) 8.4 901 m) Other 8.5 901 42 32. Have you received the Secondary Education Activity (SEA) training for teacher trainers? Yes No n 45.517.1 82.9 749 33. For those who have participated in SOROS or Secondary Education Activity (SEA) professional development programs, please indicate the degree to which you agree with each statement. [Responses shown are only those from teachers in schools where CTLP is active, or teachers who have begun to receive training as SEA master teachers.] Strongly agree Agree Disagree Strongly disagree n a) Techniques I learned through SOROS or SEA have helped me improve my teaching 27.3 64.2 7.8 0.7 271 b) Techniques I learned through SOROS or SEA improved my pupils’ learning 18.4 62.9 17.6 1.1 267 c) Individual pupil involvement during lessons has improved due to techniques I learned through SOROS or SEA 17.1 68.8 12.9 1.2 263 d) My pupils’ retention of information has improved when I use techniques I learned through SOROS or SEA 17.3 63.8 18.1 0.8 260 Student Survey Survey response distributions, expressed as a percent of total responses. 1. What is your age? (Mark only one box.) 15 16 17 18 19 20 n 0.0 0.2 9.7 76.8 12.8 0.5 1683 2. What is your gender? (Mark only one box.) Male Female n 49.4 50.6 1681 3. What is your ethnic group? (Mark only one box.) Macedonian Albanian Turkish Roma Serbian Vlach Other n 64.9 29.6 2.5 0.6 1.2 0.5 0.7 1679 43 4. What language do you speak most often at home? (Mark only one box.) Macedonian Albanian Turkish Roma Serbian Vlach Other n 67.2 29.5 2.0 0.4 0.5 0.2 0.2 1683 5. What is the size of the place where you live? (Mark only one box.) Village or rural area (less than 10,000 people) Town (10,000 – 50,000) Small city (50,000 –100,000) Skopje n 32.4 18.0 32.8 16.8 1676 6. What is the highest educational level each of the following persons completed? If you are not sure, please give your best guess. (Mark only one for each row.) Primary education or less Secondary education Some higher education or university degree Do not have this family member or do not know n a) Father or male guardian 15.9 53.6 29.3 1.2 1661 b) Mother or female guardian 31.3 48.1 19.4 1.2 1648 c) Oldest brother or sister 7.9 37.2 22.0 32.9 1541 44 7. Which of the following does your family have at home? (Mark only one for each row.) Have Do not have n a) Daily newspaper 80.0 20.0 1637 b) Dictionary 87.2 12.8 1646 c) Encyclopedia or other reference books (Either of these two) 67.6 32.4 1620 d) Magazines 93.5 6.5 1645 e) Puzzles, crossword puzzles, or games that involve strategy (any of these) 79.7 20.3 1614 f) A specific place to study 79.4 20.6 1634 g) Computer 45.8 54.2 1625 h) Cell phones for the older children 85.5 14.5 1651 i) CD player 65.3 34.7 1625 j) Cable television 56.8 43.2 1626 k) Satellite television 29.8 70.2 1621 l) High speed internet connection 21.2 78.8 1602 m) DVD player 30.4 69.6 1616 n) Automobile 75.0 25.0 1657 8. In total, how many professors did you have during all years of high school? (Mark only one.) 29 or less 30-34 35-39 40-45 46 or more n 67.9 17.4 7.3 4.6 2.8 1676 45 9. How much do you disagree or agree with each of the following: (Mark only one for each row.) Strongly agree Agree Disagree Strongly disagree n a) School has been a waste of time 1.0 2.3 27.8 68.9 1667 b) I am good at most school subjects 28.1 56.0 14.5 1.4 1656 c) School has helped give me confidence to make decisions 40.3 47.8 8.9 3.0 1658 d) School has taught me things that could be useful in a job 50.9 42.2 4.2 2.7 1664 e) I learn things quickly in most school subjects 21.2 53.4 23.2 2.2 1660 f) School has done little to prepare me for adult life when I leave school 8.9 27.7 48.2 18.2 1662 g) School has helped me learn how to solve problems and make decisions 25.6 52.9 17.8 3.7 1669 h) I do well in tests in most school subjects 17.1 50.7 29.4 2.8 1673 10. Throughout your years of high school, how many of your professors were using the following specific methods? (Answer all questions truthfully based on your actual experiences in class. Mark only one for each row.) 0-5 professors 6-10 professors 11 or more professors n a) Having students work on assignments with others in small groups 79.4 17.3 3.3 1658 b) Having students discuss among themselves what they read or observe 58.6 34.7 6.7 1641 c) Having students help their group to complete a project 60.9 29.5 9.6 1632 d) Having students work together to complete classroom assignments 55.5 32.6 11.9 1641 e) Having students explain how to reach solutions or decisions in writing or orally 32.0 44.1 23.9 1633 f) Having students debate ideas with or explain their reasoning to the class 50.5 33.2 16.3 1637 g) Encouraging all students to participate in class discussions 45.1 34.0 20.9 1637 h) Encouraging student presentations or interesting writing assignments 66.0 25.2 8.8 1641 46 11. During all your years of high school, how many of your professors were using the following specific strategies? (Mark only one for each row.) 0-5 professors 6-10 professors 11 or more professors n a) Class work that includes problems or issues with no immediate obvious solution or answer 77.6 20.0 2.4 1648 b) Assignments that require students to organize information (comparing, classifying, ordering, etc.) 48.6 42.5 8.9 1646 c) Assignments that require students to figure out for themselves the best way to deal with complex issues or situations 44.5 40.0 15.5 1639 d) Assignments that require troubleshooting why something does not work properly or diagnosing why a situation does not go as planned 50.9 35.2 13.9 1635 e) Encouraging students to discuss what they already know about a topic before the lesson begins 50.5 34.4 15.1 1640 f) Having students write questions they would like answered before the professor starts a presentation 67.0 24.4 8.6 1632 g) Having students create their own questions based on their lessons 44.4 34.6 21.0 1641 12. During all your years of high school, for how many of your professors have the following approaches occurred in your classes? (Mark only one for each row.) 0-5 professors 6-10 professors 11 or more professors n a) My professors show me what to do to improve my work 27.6 42.4 30.0 1672 b) I review and revise my own work 58.0 30.0 12.0 1649 c) Professors involve students in determining how well the students are doing in class 39.7 40.5 19.8 1652 d) I set my own goals for learning 60.1 28.5 11.4 1655 13. This question relates ONLY to professors (if any) who you mentioned in questions 10, 11, and 12 as professors who use or encourage some of the activities covered in these questions. How often do they use or encourage these activities? (Mark only one for each row.) Less than once a week Between one and three times a week Almost every day I did not have this type of professor None of my professors in these subjects used these methods n a) My general education or gymnasia professors (all students) 36.8 38.4 13.9 0.0 10.9 1493 b) My vocational subject professors (Vocational students only) 18.8 28.2 19.5 25.0 8.5 1378 47 14. To what extent are you taught to develop each of the following skills through school or classroom activities? (Mark only one for each row.) Not at all Somewhat A lot n a) How to begin, conduct, and complete large projects (at least one week long) 26.5 65.6 7.9 1666 b) How to speak in front of others 15.0 54.1 30.9 1671 c) How to identify and use resources (human, financial, informational) for projects 28.7 55.1 16.2 1658 d) How to write reports or technical documents 32.1 53.3 14.6 1660 15. Have you ever been involved in the following types of projects through school? (Mark only one for each row.) No Yes Does not apply to me n a) Projects in my general education or gymnasia courses that last a week or more 79.5 20.5 1589 b) Projects in my vocational education courses that last a week or more 53.5 21.8 33.7 1639 c) Projects where I interact with community members 65.6 34.4 1608 d) Projects where I interact with businesses 84.4 15.6 1603 e) Research using computers, libraries, or community resources 63.5 36.5 1618 16. Did school help you become more informed about the job market in a potential future profession in the following locations? (Mark only one for each row.) [Responses shown here are only those of vocational students.] No Yes n a) In your local area or municipality (locally) 40.2 59.8 892 b) Across Macedonia 55.0 45.0 888 c) Internationally 72.1 27.9 883 48 17. Have you ever been taught the following skills through school? (Mark only one for each row.) [Responses shown here are only those of vocational students.] No Yes n a) How to interview for jobs 43.8 56.2 893 b) How to write a resume 26.2 73.8 887 c) How to search for and find jobs 39.8 60.2 891 d) How to write a letter to apply for a job 36.7 63.3 890 e) How to determine the skills needed for a future profession 35.4 64.6 886 f) How to understand manuals, diagrams, memoranda, and other job-related documents 55.1 44.9 891 g) How to asses my performance in a job 36.0 64.0 893 18. Have you ever participated in the following activities during school? (Mark only one for each row.) [Responses shown here are only those of vocational students.] No Yes Don’t know what this is n a) School business or firm 61.7 12.9 24.4 877 b) Virtual school business or firm 61.8 5.1 33.1 869 c) Practical work outside the school 31.8 67.0 1.2 869 d) Participate in career guidance discussions/activities 55.5 42.1 2.4 879 e) Role playing about different job situations 72.0 25.6 2.4 883 19. Have you ever participated in the following activities? (Mark only one for each row.) [Responses shown here are only those of vocational students.] No Yes Don’t know what this is n a) Develop a portfolio 53.4 4.1 42.5 885 b) Job shadow 39.2 59.1 1.7 890 c) Prepare a business plan 67.0 32.1 0.9 887 d) Career exploration 58.0 40.4 1.6 890 20. How many times in the previous two full school weeks did you skip class? (Mark only one box.) 0 1-2 3-4 5 or more n 32.5 40.3 12.4 14.8 1673 49 21. How often do you use a computer in these places? (Mark only one for each row.) Less than once a month Between once a week and once a month At least once a week n a) At home 32.4 13.5 54.1 1263 b) At school 65.9 16.7 17.4 1265 c) At other places 29.1 31.3 39.6 1541 22. During your years in high school, about how many times have you used computers either as part of the class or outside of the class period for the following subjects? (Mark only one for each row.) 0-5 times total 6-20 times total More than 20 times n a) Science or Mathematics 80.1 10.3 9.6 1541 b) Humanities, Social Sciences, or Languages (e.g., psychology, history, literature) 77.4 16.5 6.1 1516 c) Vocational courses (vocational students only) 58.8 20.3 20.9 1159 23. How many of your professors in high school have required you to use a computer either during class or outside of class at least two times during the school year? (Mark only one for each row.) 0-2 3-10 11 or more Does not apply to me n a) Gymnasia courses (gymnasia students only) 49.2 18.3 5.1 27.4 1356 b) General education courses (vocational students only) 48.0 19.0 7.2 25.8 1298 c) Vocational education courses (vocational students only) 43.5 21.5 9.3 25.7 1317 24. When you are at school, how difficult is it to get to use a computer when you want to? (Mark only one box.) Easy to get access to school computers Somewhat difficult Very difficult or no access to school computers n 11.7 31.2 57.1 1648 25. Do you have access to the Internet in the following places? (Mark only one for each row.) No Yes n a) At home 69.6 30.4 1586 b) At school 82.5 17.5 1576 c) At other places 10.2 89.8 1644 50 26. Please indicate how well you can do each of the following tasks on a computer. (Mark only one for each row.) I don’t know how to do this I can do this but sometimes need help I can do this completely by myself n GENERAL TOOLS 12.6 44.5 42.9 1644 a) Start and run programs or software 16.4 33.4 50.2 1636 b) Delete a computer document or file and copy a file from one location to another 18.5 30.8 50.7 1641 DOCUMENTS, DATABASES, AND PROGRAMMING a) Use word processing software (e.g., Word®, Works®, WordPerfect®, etc.) to create or edit a document such as a paper or letter 26.2 41.8 32.0 1613 b) Use databases (e.g., Access®, Oracle®, etc.) to add data, sort information or create reports 46.3 42.7 11.0 1593 c) Use a spreadsheet (e.g., Excel®, Works®, etc.) to enter and calculate numbers 19.2 40.6 40.2 1598 d) Use a word processor, spreadsheet, or other program to create graphs, charts, or tables 18.5 43.0 38.5 1602 e) Create macros, de-bug computer programs, or write computer programs 64.8 27.4 7.8 1604 f) Diagnose and fix problems with a computer 59.1 30.6 10.3 1607 MULTIMEDIA TOOLS a) Create a slide show presentation (e.g., PowerPoint®) 60.4 26.2 13.4 1590 b) Use image-editing software (PhotoShop®, etc.) to alter or enhance photographs or images 39.5 39.4 21.1 1590 c) Create a multi-media presentation (that might include text, sound, pictures or video) (e.g., using Hyperstudio® or HyperCard®) 63.0 28.0 9.0 1583 d) Use web authoring software (e.g., Dreamweaver, Netscape Composer®, Frontpage, or other software) to create web pages 58.3 31.7 10.0 1574 e) Upload images to a computer from a scanner, camera, or other source 40.0 33.4 26.6 1582 f) Use MIDI software to create or alter music 48.9 32.5 18.6 1575 g) Use Computer Aided Design software or Computer Aided Manufacturing software 74.9 20.0 5.1 1573 51 I don’t know how to do this I can do this but sometimes need help I can do this completely by myself n RESEARCH AND COMMUNICATION TOOLS a) Attach files to and send email messages 29.9 30.2 39.9 1594 b) Find information on the Internet using a search engine 18.2 35.2 46.6 1585 c) Chat, use instant message, or subscribe to newsgroups 35.6 35.3 29.1 1579 d) Download files from the Internet and access them from you computer 37.8 30.1 32.1 1575 e) Maintain a website (use programming software, upload materials, etc.) 53.5 32.6 13.9 1579 f) Use probes (e.g., temperature sensor, motion detector) and probe software to collect and analyze information 71.1 22.3 6.6 1584 g) Help set up or troubleshoot problems with computer networks 64.0 28.5 7.5 1587 52 Appendix C. English translations of student and teacher surveys Teacher Survey This survey is designed to help the United States Agency for International Development (USAID) learn how effective its programs are in Macedonia’s secondary schools. Please answer the questions as honestly as possible so that we can better meet the needs of Macedonia’s secondary schools now and in the future. There are no “correct” answers. Your responses will be kept STRICTLY CONFIDENTIAL and will NOT be shared with other Teachers, directors or school officials in Macedonia. You will answer the questions in this survey by marking with the pencil provided the oval for the response that most accurately represents your situation. If you mark the wrong oval and need to change your response, please erase thoroughly and then mark the correct oval. Please do not use a pen. It is important that you mark within the oval like this: Not like this: e e e e e Most questions ask you to choose only one from among a series of choices arranged in either a column or a row. A few questions will ask you to choose as many choices as apply to you. Those questions indicate specifically to “Mark all that apply.” Otherwise choose only one. There are various types of questions. For instance, some questions will ask you to mark how strongly you agree with a statement. An example is provided below. If you agree that a quiet classroom generally is needed for effective learning, you should mark the oval under “Agree”, as shown below. If you more strongly agree, then you should mark the oval under “Strongly Agree”. Likewise, if you disagree or strongly disagree, you would fill in the oval under the appropriate column. Fill in only one oval for each row for these questions. Please indicate the degree to which you agree with each statement by checking the appropriate oval. Strongly Agree Agree Disagree Strongly Disagree a) A quiet classroom generally is needed for effective learning. e e e There are many other question types that include from two to five answer options. Fill in only one oval for each row for these questions unless it indicates “Mark all that apply.” Place your completed survey in the sealed envelope provided, place the Secondary Education Activities project (SEA) sticker across the fold, and return it to the box in the office of your school pedagogue or the location marked on the outside of the envelope. School administration and staff have been instructed not to open the envelopes. These envelopes will be sent directly to the American Institutes for Research (AIR) in the United States for analysis. Honest, independent responses are needed by USAID to make changes to its projects to help Macedonia reform its schools. Please do not answer in the way you believe someone else might want you to answer or discuss the survey with colleagues to help answer questions. A Macedonian staff member is available to help answer any questions. Please contact Zoran Stojanov at 023- 218-050 or 070-337-761, and he will call you back to answer any questions. We can repay the cost of your phone call if required. 53 ANONYMOUS AND CONFIDENTIAL: DO NOT SHOW SURVEY OR ANSWERS TO OTHER TEACHERS OR DIRECTOR. DO NOT PUT YOUR NAME ON THIS FORM. 1. What is your age? e 30 years or younger e 31 – 40 years e 41 – 50 years e 51 – 60 years e 61+ years 2. What kind of students do you teach? (Mark all that apply.) e Gymnasium e Vocational 3. What kind of subjects do you teach? (Mark all that apply.) e Natural sciences or mathematics e Humanities, Social sciences, or Languages (e.g., psychology, history, literature, arts) e Information technology / Computers e Vocational subjects e Practical training 4. What is the highest level of schooling that you have completed? e Secondary degree e Some higher education e University degree e Some post-baccalaureate education or completed degree 5. How many years have you been a classroom teacher? e 0- 5 years e 6-10 years e 11-20 years e 21 years or more 6. What one language do you speak the most often at home? e Macedonian e Albanian e Turkish e Roma e Serbian e Vlach e Other 54 7. What languages do you teach in? (Mark all that apply.) e Macedonian e Albanian e Turkish e English e Other 8. What is the size of most of your classes? e 15 or fewer students e 16-24 students e 25-29 students e 30-34 students e 35-39 students e 40 or more students 9. What is your gender? e Male e Female 10. Please indicate the degree to which you agree with each statement. Strongly Agree Agree Disagree Strongly Disagree a) It is best when students work on assignments alone to show how much they know. e e e e b) All students should be encouraged to participate in class discussions even if they don’t know the answer e e e e c) Teachers know more than students and should just explain the facts directly. e e e e d) Teachers should provide feedback to students on assignments to show them how to improve their work. e e e e e) Classes should be focused on problems with specific, correct answers and ideas that students can grasp quickly. e e e e f) Student projects often result in students learning “incorrect knowledge." e e e e g) All real learning from school happens in the classroom. e e e e 55 11. Teachers may have one or more goals when asking students questions. When you ask students questions, how often do you try to accomplish the following goals? Never Sometimes Often Very often or always a) See if students know the correct answer e e e e b) Elicit/Draw out students’ ideas and opinions e e e e c) Get students to justify and explain their reasoning e e e e d) Have students relate what they are working on to their own experiences e e e e 12 Not including the matura and final exams, how important are the following in determining your students’ grades in the subjects you teach? . Not important Minor importance Moderate importance Very important a) Questioning e e e e b) Performance on experiments, projects, or other hands-on experiences e e e e c) Written reports or essays e e e e d) Oral presentations other than questioning e e e e 13 Please indicate if your students do the following in some of your classes: (Mark all that apply.) . a) Students conduct projects that require them to interact with community members outside of school e b) Students conduct projects that require them to interact with businesses outside of school e c) Students conduct projects where they must use libraries or community resources e d) Students conduct projects that last a week or more e e) Students review and revise their own work e f) Students have opportunities to set their own goals for learning or criteria for grades e g) Students learn how to understand job-related documents such as manuals, diagrams or memoranda e h) Students learn how to assess their own performance in a job e 56 14. Please indicate how often, if at all, you include the following instructional practices in some of your classes. Never Less than once a month 1-3 times a month At least once a week a) Class work includes problems or issues with no immediately obvious solution or answer. e e e e b) Assignments require students to troubleshoot why something does not work properly or diagnose why a situation does not go as planned. e e e e c) Students discuss what they already know about a topic before the lesson begins. e e e e d) Students write questions they would like answered before I start a presentation. e e e e e) Students create their own questions based on their lessons. e e e e f) Students work on assignments in groups. e e e e g) Students break into groups to discuss among themselves. e e e e h) Students debate ideas or discuss their reasoning with other students. e e e e i) Students compare their thoughts to those of authors or others. e e e e j) I give feedback on student assignments on how to improve. e e e e 15. What is your level of interest in the instructional practices mentioned in question 13 and 14? e High – I am interested in many of these instructional practices e Medium – I have some interest in some of these instructional practices e Low – I am not interested in most of these instructional practices 16 Indicate whether any of the following potential obstacles in using the instructional practices described in questions 13 and 14 apply to you. . This has been an obstacle for me to use these instructional practices a) I have not been trained in most of these instructional practices e b) These instructional practices do not apply to most of my courses e c) I don’t have time to practice these instructional practices e d) Inadequate teacher training support on how to use these instructional practices in the classroom e e) There is not enough flexibility in curriculum to make room for these instructional practices e f) I do not have the necessary equipment or materials e g) Too many students in classroom or insufficient classroom size e h) Class time is too short e 57 17. How often did you have the following types of interactions this year? Never or does not apply Less than once a month One to three times a month At least once a week a) Discussions with other teachers about how to teach a particular concept to a class e e e e b) Discussions with other teachers about ideas for student projects or sharing examples of student projects e e e e c) Discussions with teachers at other schools through professional associations e e e e d) Discussions with teachers at other schools through associations of schools with the same vocations e e e e e) Discussions with representatives of local companies or industry e e e e 18. Please indicate the degree to which you agree or disagree with each statement about your school. Strongly Agree Agree Disagree Strongly Disagree a) Teachers at my school have the pedagogical training they need to improve their teaching. e e e e b) I enjoy being a teacher e e e e c) Students’ training in my school is better now than it was five years ago. e e e e d) Students are more likely to get jobs when they graduate because of what they have learned in school. e e e e e) Students are more likely to get into university when they graduate because of what they have learned in school. e e e e 19. Do you have a computer in you classroom or the annex to your classroom? e Yes e No 20. How often did you have access to the following devices this year? Never tried to get access or not interested Never Less than once a month One to four times a month More than once a week a) Computer reserved for teachers e e e e e b) Computer in the school computer lab e e e e e c) Computer printers at school e e e e e d) Internet at school e e e e e 58 21. In how many of your courses this year did you ask students to produce the following using computers? Not relevant to my classes None One or two Three or four Five or more a) Research using the Internet e e e e e b) Reports and term papers e e e e e c) Multimedia projects or music e e e e e d) Web pages, web sites, etc. e e e e e e) Artwork, photography or video products e e e e e f) Graphs or charts e e e e e g) Calculations or information using a spreadsheet e e e e e h) Reports or information derived from databases e e e e e i) Slide show presentations e e e e e 22. How often do you do the following? Never Once or twice a year Up to once a month At least once a month a) Use a computer or the Internet as part of your instruction e e e e b) Make handouts for students using a computer e e e e c) Create a test or assignment using a computer e e e e d) Email for professional use e e e e 23. What is your level of interest in using computers for your teaching or preparation? e High – Very interested e Medium – Moderate interest e Low – Not interested 24. Indicate whether any of the following potential obstacles to using computers in teaching apply to you. This has been an obstacle for me to use computers in teaching a) Too little access to computer labs e b) Too few computers for my students e c) Not comfortable enough with computers to use them for teaching e d) Not sure how to make technology relevant to my subject e e) Insufficient training on how to use technology for classes e f) Computers are too unpredictable or outdated, or the software is inadequate or does not work right e g) No access to printer or the printer frequently is not working or out of paper or ink e h) There is not enough flexibility in curriculum to make room for technology e 59 25. Who in your school makes decisions about the following issues (not according to law but in everyday experience). (Mark all that apply for each row.) Do not know MOES or BDE School Board School Director The Mayor Teacher Council Individual Teachers a) Suggesting which school director to hire e e e e e e e b) Appointing the director e e e e e e e c) Deciding which profiles, smer, or nasoka are offered e e e e e e e d) Determining course content e e e e e e e 26. Please mark which of the following responsibilities is exercised by the director of your school, if any (Mark all the apply) e Paying teacher salaries e Hiring teachers e Formulating the total school budget e Deciding on budget allocations within the school 26. Mark which of the following two groups, if either, are informed about school budgets and spending choices by the school director or school board? e Teachers e Parents 27. Which of the following actually help teachers at your school in their professional development? (Mark all that apply.) e MOES inspectors e BDE advisors e School director e School pedagogue e Teacher Council e Parent Council e Other teachers within school e Union of Employees in Education, Culture, and Science e School associations e Foreign organizations 60 28. Which of the following does your Teacher Council or Councils do? (Mark all that apply.) a) We do not have a Teacher Council or it is inactive e b) I do not know much about the Teacher Council e c) It makes recommendations to school board or school director e d) It negotiates or makes decisions together with school board or school director e e) It fights with school board or school director e f) It does whatever the school board or school director tells it to do e g) It recommends school budget priorities e h) It coordinates with other school associations or bodies (e.g., parent council or other teacher councils) e i) It coordinates with Union of Employees in Education, Culture, and Science e 29 What changes do you think are needed to improve the interaction between school directors and Teacher Councils? (Mark all that apply) . a) No changes are needed e b) Changes in the laws to better define the role of teacher councils e c) Changes in the regulations to better define the role of teacher councils e d) Training for Teacher Councils about how to assert their roles e e) Training for school directors about the proper roles of teacher councils e f) Other (please describe): e 30 Please check all of the professional development programs in which you have participated in the last six years: (Mark all that apply.) . e I have not participated in professional development courses or workshops in the last 6 years e Phare (VET I) e GOPA (VET II) e GTZ e Junior Achievement e ECO NET e SOROS (Reading and Writing for Critical Thinking) e Secondary Education Activity (SEA) e E-scola (Computer-IT) e World Bank Education Modernization e CRS (Civic Education) e CRS (Computer-IT) e Other (Please name or describe): 61 31. Have you received the Secondary Education Activity (SEA) training for teacher trainers? e Yes e No 32. For those who have participated in SOROS or Secondary Education Activity (SEA) professional development programs, please indicate the degree to which you agree with each statement. Strongly Agree Agree Disagree Strongly Disagree a) Techniques I learned through SOROS or SEA have helped me improve my teaching. e e e e b) Techniques I learned through SOROS or SEA improved my pupils’ learning. e e e e c) Individual pupil involvement during lessons has improved due to techniques I learned through SOROS or SEA. e e e e d) My pupils’ retention of information has improved when I use techniques I learned through SOROS or SEA. e e e e Thank you for completing the survey. Please place the survey in the envelope provided, seal it with the SEA label, and put it in the sealed box in the director’s office. To maintain confidentiality and to make sure each teacher responds based on their own experiences, please do not discuss your answers with others. 62 MACEDONIA SECONDARY SCHOOL STUDENT SURVEY Your test administrator will tell you how to fill out this section. 1. School 2. Year 3. Class 4. Student 5. Label number e 0 e 0 e 3 e 0 e 0 e 0 e 0 e 0 e 0 e 0 e 0 e 1 e 1 e 4 e 1 e 1 e 1 e 1 e 1 e 1 e 1 e 1 e 2 e 2 e 2 e 2 e 2 e 2 e 2 e 2 e 2 e 2 e 3 e 3 e 3 e 3 e 3 e 3 e 3 e 3 e 3 e 3 e 4 e 4 e 4 e 4 e 4 e 4 e 4 e 4 e 4 e 4 e 5 e 5 e 5 e 5 e 5 e 5 e 5 e 5 e 5 e 5 e 6 e 6 e 6 e 6 e 6 e 6 e 6 e 6 e 6 e 6 e 7 e 7 e 7 e 7 e 7 e 7 e 7 e 7 e 7 e 7 e 8 e 8 e 8 e 8 e 8 e 8 e 8 e 8 e 8 e 8 e 9 e 9 e 9 e 9 e 9 e 9 e 9 e 9 e 9 e 9 You will answer the questions in this survey by filling the block marked by the same letter as the answer that you consider correct. Fill the oval like this: Not like this: e e e e e If you fill in the wrong box and need to change your response, make sure to erase your old answer thoroughly first and then fill in the correct box. PRACTICE EXAMPLES SECTION The first few questions will ask you to choose from among a series of choices arranged in a column. You will fill in one box for each question. An example is presented below for a question about age. If you are 18 years old, you would fill in entirely the box next to 18, as shown below. If you are 17, you would instead fill in the box next to 17. What is your age? e 15 e 16 e 17 18 e 19 e 20 63 PRACTICE EXAMPLES, CONTINUED Some questions will ask you to answer either yes or no. You will fill in the box that answers the question best. You will fill in one box for each row. For row a, if you have attended classes at some point during school, you would go across the first row and fill in the box in the column under “yes,” as shown below. You will fill in only one box for each row. Fill in row b with the appropriate response based on whether you ever visited the school pedagogist about your classes or career. Have you ever participated in the following activities during school? Yes No a) Attending classes e b) Visiting the school pedagogist about your classes or career e e Other questions will ask you to select how often something happens. The options will be listed in the columns over the checkboxes. Fill in the box that answers the question best. You will fill in only one box for each row. In this example, suppose that all of your professors encourage students to talk about their previous knowledge on the topic before the lesson starts. The best column is the last column because this is true for more than 11 professors and the last column is the one to mark if the statement is correct for more then 11 teachers. In row a) you will fill the oval under the option “more than 11 teachers”. Next, fill in row b by yourself and wait for the test administrator to discuss it with the class. Throughout your years of high school, how many of your professors were using the following specific methods? 0 – 5 professors 6– 10 professors 11 or more professors a) Encouraging students to discuss what they already know about a topic before the lesson begins e e e b) Discussing ideas or presenting opinions to the class e e e 64 For this survey, there are no right or wrong answers. It is important that you answer all questions as truthfully as possible based on your own experiences. All of your answers are 100% confidential and will not be seen by any school staff or administration and will not affect your grades or graduation. Accurate, truthful responses are needed for the Secondary Education Activity project to understand the school system and help make changes. Do not answer in the way you believe someone else might want you to answer or the way your friend answers. There are no wrong answers, only the correct, honest answer given your experience at school. During this survey, you may ask the test administrator about the meaning of the questions or the columns. The test administrator will not tell you a “right” answer or how to answer, only how to understand the question so that you may answer it yourself. YOU NOW MAY TURN THE PAGE AND BEGIN THE SURVEY. 65 1. What is your age? (Mark only one box.) e 15 e 16 e 17 e 18 e 19 e 20 2. What is your gender? (Mark only one box.) e Male e Female 3. What is your ethnic group? (Mark only one box.) e Macedonian e Albanian e Turkish e Roma e Serbian e Vlach e Other 4. What language do you speak most often at home? (Mark only one box.) e Macedonian e Albanian e Turkish e Roma e Serbian e Vlach e Other 5. What is the size of the place where you live: (Mark only one box.) e Village or rural area (less than 10,000 people) e Town (10,000 – 50,000) e Small city (50,000 – 100,000) e Skopje 66 6. What is the highest educational level each of the following persons completed? If you are not sure, please give your best guess. (Mark only one for each row.) Primary education or less Secondary education Some higher education or university degree Do not have this family member or do not know a) Father or male guardian e e e e b) Mother or female guardian e e e e c) Oldest brother or sister e e e e 7. Which of the following does your family have at home? (Mark only one for each row.) Have Do not have a) Daily newspaper e e b) Dictionary e e c) Encyclopedia or other reference books (Either of these two) e e d) Magazines e e e) Puzzles, crossword puzzles, or games that involve strategy (any of these) e e f) A specific place to study e e g) Computer e e h) Cell phones for the older children e e i) CD player e e j) Cable television e e k) Satellite television e e l) High speed internet connection e e m) DVD player e e n) Automobile e e 8. In total, how many professors did you have during all years of high school? (Mark only one.) e 29 or less e 30-39 e 40-45 e 46 or more 67 9. How much do you disagree or agree with each of the following: (Mark only one for each row.) Strongly Disagree Disagree Agree Strongly Agree a) School has been a waste of time e e e e b) I am good at most school subjects. e e e e c) School has helped give me confidence to make decisions e e e e d) School has taught me things that could be useful in a job e e e e e) I learn things quickly in most school subjects. e e e e f) School has done little to prepare me for adult life when I leave school e e e e g) School has helped me learn how to solve problems and make decisions e e e e h) I do well in tests in all school subjects. e e e e 10. Throughout your years of high school, how many of your professors were using the following activities? (Answer all questions truthfully based on your actual experiences in class. Mark only one for each row.) 0 – 5 professors 6 – 10 professors 11 or more professors a) Having students work on assignments with others in small groups e e e b) Having students discuss among themselves what they read or observe e e e c) Some or all students help lead their group to complete the project e e e d) Having students work together to complete classroom assignments e e e e) Having students explain how to reach solutions or decisions in writing or orally e e e f) Having students debate ideas with or explain their reasoning to the class e e e g) Encouraging all students to participate in class discussions e e e h) Encouraging student presentations or interesting writing assignments e e e 68 11. During all your years of high school, how many of your professors were using the following specific strategies? (Mark only one for each row.) 0 – 5 professors 6 – 10 professors 11 or more professors a) Class work that includes problems or issues with no immediately obvious solution or answer e e e b) Assignments that require students to organize information (comparing, classifying, ordering, etc.) e e e c) Assignments that require students to figure out for themselves the best way to deal with complex issues or situations e e e d) Assignments that require troubleshooting why something does not work properly or diagnosing why a situation does not go as planned e e e e) Encouraging students to discuss what they already know about a topic before the lesson begins e e e f) Having students write questions they would like answered before the professor starts a presentation e e e g) Having students create their own questions based on their lessons e e e 12. During all your years of high school, for how many of your professors have the following approaches occurred in your classes? (Mark only one for each row.) 0 – 5 professors 6 – 10 professors 11 or more professors a) My professors show me what to do to improve my work e e e b) I review and revise my own work e e e c) Professors involve students in determining how well the students are doing in class e e e d) I set my own goals for learning e e e 69 13. This question relates ONLY to professors (if any) who you mentioned in questions 10, 11 and 12 as professors who use or encourage some of the activities covered in these questions. How often do they use or encourage these activities? (Mark only one for each row.) Less than once a week Between one and three times a week Almost every day I did not have this type of professor None of my professors in these subjects used these methods a) My general education or gymnasia professors (All students) e e e e b) My vocational subject professors (Vocational students only) e e e e e 14. To what extent are you taught to develop each of the following skills through school or classroom activities? (Mark only one for each row.) Not at all Somewhat A lot a) How to begin, conduct, and complete large projects (at least one week long) e e e b) How to speak in front of others e e e c) How to identify and use resources (human, financial, informational) for projects e e e d) How to write reports or technical documents e e e 15. Have you ever been involved in the following types of projects through school? (Mark only one for each row.) No Yes Does not apply to me a) Projects in my general education or gymnasia courses that last a week or more e e b) Projects in my vocational education courses that last a week or more e e e c) Projects where I interact with community members e e d) Projects where I interact with businesses e e e) Research using computers, libraries, or community resources e e 70 16. Did school help you become more informed about the job market in a potential future profession in the following locations? (Mark only one for each row.) No Yes a) In your local area or municipality (locally) e e b) Across Macedonia e e c) Internationally e e 17. Have you ever been taught the following skills through school? (Mark only one for each row.) No Yes a) How to interview for jobs e e b) How to write a resume e e c) How to search for and find jobs e e d) How to write a letter to apply for a job e e e) How to determine the skills needed for a future profession e e f) How to understand manuals, diagrams, memoranda, and other job￾related documents e e g) How to assess my performance in a job e e 18. Have you ever participated in the following activities during school? (Mark only one for each row.) No Yes Don’t know what this is a) School business or firm e e e b) Virtual school business or firm e e e c) Practical work outside the school e e e d) Participate in career guidance discussions/activities e e e e) Role playing about different job situations e e e f) Record on my learning process being related to the world of business. e e e 19. Have you ever participated in the following activities? (Mark only one for each row.) No Yes Don’t know what this is a) Develop a portfolio e e e b) Job shadow e e e c) Prepare a business plan e e e d) Career exploration e e e 71 20. How many times in the previous two full school weeks did you skip classes? (Mark only one box.) e 0 e 1-2 e 3-4 e 5 or more 21. How often do you use a computer in these places? (Mark only one for each row.) Less than once a month Between once a week and once a month At least once a week a) at home e e e b) at school e e e c) at other places e e e 22. During your years in high school, about how many times have you used computers either as part of the class or outside of the class period for the following subjects? (Mark only one for each row.) 0-5 times total 6-20 times total More than 20 times a) Science or Mathematics e e e b) Humanities, Social sciences, or Languages (e.g., psychology, history, literature) e e e c) Vocational courses (Vocational students only) e e e 23. How many of your professors in high school have required you to use a computer either during class or outside of class at least two times during the school year? (Mark only one for each row.) Number of Professors 0-2 3-10 11 or more Does not apply to me a) Gymnasia courses (Gymnasia students only) e e e e b) General education courses (Vocational students only) e e e e c) Vocational education courses (Vocational students only) e e e e 72 24. When you are at school, how difficult is it to get to use a computer when you want to? (Mark only one box.) e Easy to get access to school computers e Somewhat difficult e Very difficult or no access to school computers 25. Do you have access to the Internet in the following places? (Mark only one for each row.) No Yes a) at home e e b) at school e e c) at other places e e 26. Please indicate how well can you do each of the following tasks on a computer. (Mark only one for each row.) I don’t know how to do this I can do this but sometimes need help I can do this completely by myself General Tools a) Start and run programs or software e e e b) Delete a computer document or file and copy a file from one location to another e e e c) Printing documents e e e Documents, Databases, and Programming a) Use word processing software (e.g., Word®, Works®, WordPerfect ®, etc.) to create or edit a document such as a paper or letter e e e b) Use databases (e.g., Access®, Oracle®, etc.) to add data, sort information or create reports e e e c) Use a spreadsheet (e.g., Excel®, Works®, etc.) to enter and calculate numbers e e e d) Use a word processor, spreadsheet, or other program to create graphs, charts, or tables e e e e) Create macros, de-bug computer programs, or write computer programs e e e f) Diagnose and fix problems with a computer e e e 73 I don’t know how to do this I can do this but sometimes need help I can do this completely by myself Multimedia Tools a) Create a slide show presentation (e.g., PowerPoint®) e e e b) Use image-editing software (PhotoShop®, etc.) to alter or enhance photographs or images e e e c) Create a multi-media presentation (that might include text, sound, pictures or video) (e.g., using Hyperstudio® or HyperCard®) e e e d) Use web authoring software (e.g., Dreamweaver, Netscape Composer®, Frontpage, or other software) to create web pages e e e e) Upload images to a computer from a scanner, camera, or other source e e e f) Use MIDI software to create or alter music e e e g) Use Computer Aided Design software or Computer Aided Manufacturing software e e e Research and Communication Tools a) Attach files to and send email messages e e e b) Find information on the Internet using a search engine e e e c) Chat, use instant message, or subscribe to newsgroups e e e d) Download files from the Internet and access them from you computer e e e e) Maintain a website (use programming software, upload materials, etc.) e e e f) Use probes (e.g., temperature sensor, motion detector) and probe software to collect and analyze information e e e g) Help set up or troubleshoot problems with computer networks ..................................................................... e e e You have now completed the survey. Please close the booklet and wait for the test administrator to tell you when you may hand in your survey and return to your classroom. You do not need to re-check your answers. 74