UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT Contract # OUT-HNE-I-802-00-00075-00-00 COMPREHENSIVE EVALUATION OF QUALITY EDUCATION through SUPPORTING TEACHING (QUEST) Ministry of Education, Science and Technology Save the Children/United States (PROJECT NO. 612-0267) April 18, 2002 Comprehensive Evaluation Team The Mitchell Group, Inc. Paul Hebert Elizabeth Randolph Bertha Udedi ii TABLE OF CONTENTS Page ACRONYMS ………………………………………………………………… ii EXECUTIVE SUMMARY …………………………………………………. iv-viii I. BACKGROUND ……………………………………………………. 1 II. PURPOSE …………………………………………………………… 2 III. METHODOLOGY …………………………………………………. 3 IV. FINDINGS …………………………………………………………. 6 Impact and Effectiveness of QUEST in Relation to QUEST …….. 6 Objectives A. Enhancement of Access …………………………………….. 6 B. Enhancement of School Quality and Efficiency …………… 7 General Observations ……………………………….. 7 Impact of Direct Teacher Training …………………. 8 Impact of Supervision Training in Support of Teacher Professional Development……………… 13 Impact on Pupil Learning Outcome………………… 15 Impact on Pupil Dropout and Retention…………… 19 Impact on Integrated Curriculum Pilot Study……… 20 Impact of Community Training and Mobilization for School Improvement………………………… 22 Collaboration with Partners…………………………. 23 Impact of QUEST in Relation to USAID Results Framework……………………………………….. 26 Impact of QUEST in Relation to Policy Investment Framework for Education………………………… 28 V. SUMMARY FINDINGS ………………………………………….… 29 VI. LESSONS LEARNED ……………………………………………….. 31 VII. RECOMMENDATIONS …………………………………………….. 33 VIII. STAKEHOLDER WORKSHOP……………………………………… 35 iii Table of Contents (Continued) Page ANNEX A. List of School Visited ……………………………………………… 38 ANNEX B. List of Persons Consulted ………………………………………… 39 LIST OF TABLES Table 1. Status of Construction Work……………………………… 7 Table 2. Results from Teacher Survey……………………………… 12 Table 3. Performance Over Time: Chichewa, English Mathematics 17 Table 4. Percentage of Dropout …………………………………… 19 Table 5. Percentage of Repetition …………………………………. 20 LIST OF FIGURES Figure 1. Mean %age Chichewa Reading Scores: QUEST/non-QUEST.. 18 Figure 2. Mean %age English Reading Scores: QUEST/non-QUEST…. 18 Figure 3. Mean %age Mathematics Scores: QUEST/non-QUEST……... 18 Figure 4. Integrated Curriculum Pilot: Chichewa Passages…….……… 21 Figure 5. Integrated Curriculum Pilot: English Passages…………….... 21 Figure 6. Integrated Curriculum Pilot: Mathematics………..…………. 21 iv ACRONYMS CDA Community Development Assistant COPE Community Options Protection and Empowerment CRECCOM Creative Center for Community Mobilization DCDO District Community Development Officer DDC District Development Council DEO District Education Office/Officer DFID Department for International Development DDO District Development Officer DC District Commissioner FPE Free Primary Education GABLE PPC Girls’ Advancing Basic Literacy and Education, Policy Planning and Curriculum IEQ Improving Educational Quality JCE Junior Certificate of Education MIE Malawi Institute of Education MSCE Malawi School Certificate of Education MSSP Malawi School Support System Program MIITEP Malawi Integrated In-service Teacher Education Program MOEST Ministry of Education, Science, and Technology MOGYCS Ministry of Gender, Youth and Community Services NGO Non-government Organization P/ID Partnership and Institutional Development PIF Policy Investment Framework PEA Primary Education Advisors PTA Parent-Teacher Association QUEST Quality Education through Supporting Teaching SC/US Save the Children Federation/US UNICEF United Nations Children’s Fund USAID United States Agency for International Development VBS Village Based Schools v EXECUTIVE SUMMARY The Team conducted a comprehensive evaluation of the QUEST Project implemented by SCF/US in the Districts of Mangochi, Balaka, and Blantyre Rural. The purpose of the evaluation was twofold: 1) to examine the impacts and effectiveness of the project in relation to the objectives stated for the grant 2) to analyze implementation problems and draw out lessons learned about improving school effectiveness, specifically enhancing children’s participation and learning. The team’s evaluation strategy was formative and participatory in keeping with standards of appropriate comprehensive assessment. In addition to the study of project documentation, the review and analysis of data, the field visits to the target Districts proved a most valuable benefit. The visits to schools to interview and hold focus meetings with teachers, pupils, parents, and community leaders provided a highly meaningful opportunity to experience first-hand the impact of project activities. Interviews with MOEST officials and staff at the Central, Divisional and District levels also furnished excellent means of determining the influence of the project on the national system and its decentralized branch systems. The collaborative work of partner donors and institutions (MIE. IEQ, CRECCOM) involved with QUEST was also reviewed and assessed. Summary of Findings/Results: Overall SC/US has achieved its central aim to decentralize action to improve the quality of education at the zone, cluster and school level and has, for the most part, achieved its targets for enhancing the access, quality and efficiency of primary school education in the three target Districts. The results are summarized in the following: ÿ MOEST officials at all levels of the system are keenly aware of the activities of the QUEST Project and strongly support the program. At non-central levels, education officials are directly involved in implementation by advising and monitoring activities and by administering the QUEST Fund. ÿ Thirty-three new community schools have been established with enrolments of 11,387 pupils, 69% of the target of 16,500. Construction is near to completion with 64% of all construction completed. ÿ Teachers trained under QUEST practice what they learn in classrooms and classroom environments and teaching in QUEST schools is characteristically more interactive and more creative as compared to teaching in non-QUEST schools. Teachers value their QUEST training enormously, have more confidence as a result and would like to participate in more training in the future. ÿ The teaching of literacy and numeracy, though improved, does not maximize pupils’ opportunities for meaningful application of new skills and concepts and rarely provides opportunities for creative thinking, self-initiated learning, and problem solving. Exploration of novel and effective ways to enhance the teaching and learning of literacy and numeracy in Primary school classrooms is critical. ÿ It should not be assumed that teachers can develop adequate learning environments simply as a result of training. Some materials are required, even over and beyond those that can be produced locally, particularly for literacy vi enhancement. Avenues for supplying materials that can be used for teaching need to be explored further. ÿ Teachers are knowledgeable about continuous assessment but their ability to use such information to guide their day-to-day practices and to engage pupil responsibility in learning needs to be strengthened. ÿ The School Cluster System and Mentor Teacher Program are among the most successful components of QUEST and support to facilitate the enhancement and institutionalization of these local support systems is basic for sustained classroom reform. ÿ Community training has been successful, though community involvement in school improvement is variable across communities. More systematic support, facilitation, and monitoring of community practices in school involvement is required for this result to be sustained. ÿ The target of 20% gain in pupil performance was met in the first two years of the project for the majority of subjects taught for Std 1 to Std 4. This should be taken as, overall, successful achievement of the performance indicator for school quality. ÿ With the exception of Standard 4 Chichewa, after two years, pupils in QUEST schools outperformed those in non-QUEST schools in mathematics, Chichewa (words in passages), and English (words in passages). ÿ The rate of pupil dropout decreased in all QUEST districts for Standard 1 through Standard 4 over the first two years of the project with the targeted 10% reduction in pupil dropout approximated in Standard 1 and Standard 2 for Balaka and Blantyre rural. Results for the final year of QUEST will not be obtainable until 2003. ÿ Results from the integrated curriculum pilot study showed this method to effect a performance advantage for Chichewa and mathematics, but not English. ÿ QUEST’s partnership with MIE and IEQ/Malawi has been extremely productive and encouragement of these partnerships to continue to explore avenues for enhancing the literacy and numeracy teaching strategies in junior primary schools is advised. ÿ Though successful, the accomplishments made by QUEST are not considered to be sustainable at this time and it would be premature to discontinue the support to these districts until more sustainable reform is achieved. Two avenues for effecting sustained change include: 1) enhancing and institutionalizing the School Cluster System and Mentor Teacher Program and linking these system to Malawi’s current and future teacher education programs; 2) further enhance the facilitation, support and monitoring of community participation until a measure of sustained involvement is achieved. ÿ QUEST has positively contributed to the overall SO4 strategy and has had a positive impact on the two intermediate results of the USAID/MALAWI SO4 Results Framework. ÿ QUEST has contributed substantially to the successful accomplishment of the goals of MOEST revised sector strategy as presented in the Principle Investment Framework (PIF). vii The exercise of an evaluation process assists in identifying lessons learned from the activities carried out during the life of the project. Some twenty-eight 28) such lessons have been gleaned and categorized into five areas: 1) the sensitization and orientation of stakeholders; 2) the training of teachers, parents, and local leaders; 3) the supervision of teachers; 4) the QUEST Term Plan/Initiative program; and 5) the construction of village￾based schools and housing for teachers. These practical and concrete observations about what has been learned during the implementation of the project can serve as important contributions to the on-going development efforts of all the stakeholders. More specifically, such lessons can provide very helpful guidelines to a more effective on￾going QUEST project, should such be decided upon. Inevitably, a comprehensive evaluation involves the presentation of recommendations intended to provide bases for future development project activity of current and potential partners. A total of thirty-seven (34) recommendations has been formulated by the team in the particular context of the various stakeholders of QUEST. Among the more important of these are the following: 1) the teacher-training model utilized by QUEST should be continued and supported; 2) the Teacher Training Program should be extended to the teachers of Standards 5-8; 3) the Mentor Teacher Program should be replicated, being as it is the key to sound supervision of teachers; 4) the School Cluster System should be institutionalized into the national school system; 5) consideration should be given to expanding the QUEST Project to other Districts, especially in the rural areas; 6) the research program on continuous assessment and the integrated curriculum should be continued in close collaboration with the MOEST; 7) a national policy on community participation should be developed and implemented by the GOM/MOEST; 8) a more systematic method of monitoring and evaluating community participation in the school is highly desirable; 9) in the light of the serious national food crisis and its effect on the attendance of Primary school pupils, the establishment of a school feeding program should be explored by USAID in coordination with other partners; 10) in the light of its recognized impact on the national system of Primary Education, the QUEST Project should be extended for another three-year period. On April 16 a workshop involving the various partners of QUEST was held at the Malawi Institute of Management. The objectives of the workshop were twofold: 1) to have presentations of the Internal Review Report of QUEST by SC/US and a draft of the Final Report of the USAID External Evaluation Team. 2) provide QUEST partners an opportunity to discuss the findings and recommendations of these two reports and provide comments, suggestions, and observations and to propose recommendations for moving forward after QUEST. A number of such recommendations were developed by the participants and these are presented in summary form as part of the Final Report. In summary, the team’s assessment of QUEST demonstrates that the project has met the objectives established for the QUEST grant to SCF/US. The team further concluded that the results of the evaluation have contributed importantly to meeting the objectives of the USAID Mission’s Country Strategic Plan (CSP) and its Strategic Objectives, and more specifically Strategic Objective Number Four: to support the development of an education sector reform wherein schools are providing an environment in which the majority of children are learning. viii The Evaluation Team would like to express its sincere appreciation to the staff members of the USAID Mission/Malawi, the SC/US, the MOEST and other stakeholders for their generous assistance in making this evaluation exercise such an enriching and stimulating experience. Evaluation Team Dr. Paul A. Hebert, Team Leader and Education Specialist Dr. Elizabeth Randolph, Evaluation Specialist Ms. Bertha Udedi, MOEST, Division of Planning 1 I. BACKGROUND The acronym QUEST refers to Quality Education through Supporting Teaching. It is a joint activity with the Ministry of Education, Science and Technology (MOEST), the Ministry of Gender, Youth, and Community Services (MOGYCS), the Malawi Institute of Education, and communities funded by the USAID/Malawi Mission. The project functions district wide in the Districts of Mangochi, Balaka, and Blantyre Rural and targets decentralizing action down to the zone, cluster, and school levels. The aim of the QUEST project is to increase children’s access to a basic education of high quality in a school setting that is conducive to effective learning. The objectives of the project are to: 1. Increase Access to Basic Education: · creation of approximately 16,500 new places · establishment of 33 new community schools with 132 classrooms and 33 boreholes/wells for safe water · training of 33 school committees in school management 2. Enhance Quality: · support of creative teaching to obtain a 20% increase in pupil test scores from the start to the end of the school year 3. Increase Efficiency: · empower the school committees through training to manage, develop, implement school curricula and monitor pupil drop-out and repetition so as to achieve a reduction of 10% in the districts involved 4. Test the Impact of an Integrated Curriculum · test the impact of an integrated curriculum in the District of Balaka so as to influence education policy The project has four main indicators: · increased access for 16,500 pupils in 33 community/village based schools · enhanced pupil learning by 20% · decreased pupil repetition by 10%, and · decreased pupil drop-out by 10%. In its approximate three years of operation, the project has been favorably received in the three districts, and also by partner counterparts and other interested stakeholders. Among the operational features of QUEST are: · Improved education quality for 336, 111 pupils in 466 Primary schools · Benefits extended to 4,746 teachers/headteachers, 32 Primary Education Advisors (PEAs), 41 Community Development Assistants (CDAs), school committee members, key Parent-Teacher Association members, and local leaders · Per pupil costs of approximately $10.48 over a three-year period. 2 II. PURPOSE The purpose of the QUEST Evaluation, according to the USAID Task Order (OUT￾HNE-I-802-00-00078-00, pp. 9-10) was: v To examine the impacts and effectiveness of the QUEST project in relation to the objectives stated for the grant and in relation to the Results Framework; and v To analyze implementation problems and draw out lessons learned about improving school effectiveness, and specifically enhancing children’s participation and learning. According to the task order (above) the focus of activity addressed, but was not to be limited to: a) Review of established targets, including: · 20% gain in pupil test scores · 10% reduction in pupil retention · 10% reduction in pupil dropout · Creation of approximately 16,500 new primary school places through the construction of 33 new community schools b) Assessment of learning gains associated with integrated curriculum. c) Evaluation of the sustainability and replicability of QUEST initiatives. d) Assessment of the cost effectiveness of QUEST interventions. e) Impact of QUEST activities on national education policy. f) Perceived role of communities in improving school quality and impact of community training on participation on schools and policy. g) Effectiveness of teacher, community and district training. The QUEST Project has undergone two internal reviews. The first was conducted in September, 2001 and the second in March 2002. The following recommendations came out of the September Review: · Provide additional and more structured support of teacher-initiated continuous assessment. · Support districts in designing strategies for community involvement in school development and management. · Mobilize districts and communities to more effectively implement the school construction component of QUEST, including better community involvement and more effective SCF procurement processes (especially Blantyre Rural and Balaka). · Improve the effectiveness of Term Plans (e.g., reduce number and scope). · Conduct monthly team reviews. · Maximize stakeholder involvement in monitoring and evaluation processes. 3 III. METHODOLOGY Guiding Principles for the Evaluation Research informs that sustainable education reform requires change to take place within three inter-related systems in school learning: the classroom, school system, and community (see illustration below).1 According to this framework, the classrooms are made up of teachers, students, as well as parents. Even though parents aren’t present in the classroom on a day-to￾day basis, their influence in the performance of the classroom is ever present both directly and indirectly through their participation (or lack thereof) in their child’s learning. Schools and school systems endorse and support instructional change in the classroom, without which, long-term sustainability of new approaches is unlikely. Communities that recognize the benefit of high quality schooling not only for children but for the community itself will do much to assist in the upgrading of their schools. However, in most instances it takes a deliberate effort to sensitize communities to the intimate relationship between quality schools and quality communities and to empower its members to become active in school improvement. Nested systems at play in schooling reform (Senge, et.al., 2000, p.13) The QUEST program is aligned with this framework as it seeks to reach its goals of access, quality, and efficiency through: (i) the training of teachers and school management bodies; (ii) informing district, divisional, and central administration about good practices and encouraging national policy that supports such initiatives; and (iii) mobilizing and building the capacity of communities to participate productively in school development and school governance. Following this framework our research strategy considers the interplay of these three nested systems for school learning: the classroom, school system, and community. By considering the interdependency of these systems, we are better able to understand the inter-relationships among the activities of QUEST in the context of the overall goals of the program, the MOEST’s sector investment plan for education and USAID’s strategic framework for education. Under the nested systems framework we avoid looking at issues as isolated events perceived as road-blocks to pupil learning. 1 Peter Senge, et.al. (2000) Schools that learn. Doubleday, New York, p. 11-19. 4 Instead, we attempt to understand the precipitating conditions that may have lead to such constraints and consider courses of action for improving the situation in the context of the nested systems framework. In this way we avoid the tendency to recommend quick fix actions to complex problems and are in a better position to consider as a team (i.e., including participants from Save the Children Foundation-US (SCF-US), selected members of the teaching service, government officials and participating members of civil society) a set of recommendations that facilitates spread, sustainability, and further enhancement of primary school learning in all three inter-dependent areas: the classroom, school system, and community. Furthermore, the team takes special care in considering the diverse contexts in Malawi where learning takes place; that is, different regions, cultures, households, access to schools and so on, and we consider their differential impact within and across these connected systems. From this framework we are in a better position to consider differential approaches that can assure the greatest positive interplay of classrooms, school systems, and communities across the variable learning contexts that exist in Malawi. The formative assessment will be guided by the following2 : 1. The evaluation will be formative and participatory in nature. The evaluation team consists not only of the local and international consultants, but includes the larger body of stakeholders in the QUEST program, including SCF-US, selected members of the teaching service, government officials and participating members of civil society. The evaluators will view their role as a co-learner3 with other team participants rather than expert or judge. 2. Political and social representativenesss. All aspects of the education development context will be attended to and viewpoints from multiple sources will be taken into account throughout the evaluation. 3. Individual rights. The rights, welfare, dignity and worth of participating individuals will be respected throughout the evaluation process. 4. Side effects and unanticipated outcomes. New programs often have positive and/or negative unanticipated effects and the evaluation process itself can effect participants or other persons in positive or negative ways. Any suggested side effects will be brought to the attention of decision￾makers and other relevant audiences. 5. Meta-evaluation. The evaluation team members are encouraged to self￾reflect on their activities throughout the formative evaluation, seek to achieve a comprehensive and representative analysis of the current situation, and yet be transparent about relative strengths and weaknesses of the evaluation process. 2 Snyder, C. (2000)Evaluation Guidelines in Program Evaluation, pp. 4-8. In Snyder, W. and Welsh, T., Performance Monitoring and Evaluation Series, USAID and Government of Ghana; and a review of the following: The Joint Committee on Standards for Educational Evaluation (1981) Standards for Evaluations of Educational Programs, Projects, and Materials; Joint Committee (1994) The Program Evaluation Standards; Evaluation Research Society (1982) Standards for Program Evaluation. 3 Weiss, C.H. (1998) Evaluation, 2nd Ed., Prentice Hall, NJ., p.100 5 6. Evaluation impact. The evaluation team will involve a wide range of stakeholders both in the development, analysis, interpretation of results, and dissemination stages. One of the major outcomes of the evaluation will be to facilitate a deeper understanding of the program or project, with a special focus on the potential and feasibility of replication, spread and sustainability. General Approach The QUEST evaluation strategy was, by design, an iterative and collaborative process, principally formative in nature with the inclusion of summative findings, particularly regarding specific performance indicators. Members of the QUEST evaluation team included a core of three made up of one MOEST representative and two external evaluation consultants. The core evaluation team view themselves as facilitators and co-learners along with an extended participant team, including: · SC/US staff from Mangochi, Balaka, and Blantyre-rural. · Field participants from the MOEST including teaching service and administrative staff from the local, district, and Lilongwe education centers including teachers, headteachers, mentor teachers, PEAs, District Education Managers, Division Managers and Planners. · Field participants from the MOGYCS including District Community Development Officers and Community Development Assistants. · Education development partners including staff from CRECCOM, MIE/IEQ, and USAID. Outline of Activities General Planning and Logistics Organization Information Collection · Review of Documents · Field Visits - School visits in target districts (i.e., Mangochi, Balaka, Blantyre￾rural) – cf. List of Schools Visited in Annex A - School visits in control schools in Salima. - Partner visits, including MOEST, MIE/IEQ, CRECCOM, and selected donors. - Interviews with SC/US staff. - Review of existing quantitative and qualitative data. · Initial analysis and interpretation of data. · Compilation, presentation, and discussion of preliminary results. · Identification of data collection gaps and planning/organization of final data collection and analysis. · Preparation of draft evaluation report and planning/organization of stakeholder workshop. · Conduct stakeholder workshop. · Finalization of report. 6 Description of School visits. School visits were conducted in the three days preceding the Easter break, the 26th through the 28th of March. Eighteen (18) schools were visited, 6 schools in each of the three target Districts: Mangochi, Blantyre Rural, and Balaka. One core evaluation team member accompanied by a SC/US staff member and an MOEST representative visited each school. Data was collected from teacher surveys, classroom observations, and focus group discussions with parents, school committee members, community leaders, teachers, and pupils. In addition teacher observations were conducted in five QUEST control schools in Salima. External observers who were trained PEAs or other school officials (i.e., PEAs assigned to zones other than those of the schools they visited) conducted the teacher observations in the control schools. Teacher surveys were completed by 122 Standard 1 to Standard 4 teachers: 24 from Mangochi; 49 from Blantyre Rural; and 49 from Balaka. Classroom observations were made in 27 classrooms in schools within the QUEST Districts and 24 classrooms in the five control schools in Salima. Data from the QUEST monitoring and evaluation system were used to measure performance of QUEST on the set of performance indicators for the QUEST program. These include performance data from the longitudinal study, pupil dropout and repetition study, and the integrated curriculum study. IV. FINDINGS The following set of findings is presented according to the overall goals and specific objectives of the evaluation as stated in the USAID mandate Impact and effectiveness of the QUEST in relation to the QUEST objectives A. Enhancement of Access The objective of the construction component was to create approximately 16,500 new Primary school places through the construction of 33 Primary school, 11 in each of the three target Districts. Each of the schools was to have: · Two classroom blocks · A teacher’s house · Pit latrines · Safe water through provision of a borehole All 33 community schools have been established through construction. Most of the schools are completed while some are in their final stages (See Table 1). These latter schools are providing the intended maximum number of new Primary school places (16,500) of which 11,387 or 69% were filled, as follows: 7 District Enrolment Mangochi 4263 Balaka 3318 Blantyre Rural 3806 Total 11,387 Table 1. Status of construction work. District Number completed 1 st c/room block 2 nd c/room block Latrines Teachers houses Boreholes Sites with major components* Balaka 11 10 9 10 11 10 Mangochi 11 2 2 4 11 2 Blantyre Rural 11 9 7 9 10 9 Total 33 21 18 23 32 21 % Completed 100% 64% 55% 70% 97% 64% * Major components refer to 2 classroom blocks, 1 teacher’s house and a borehole The construction of community schools has helped in reducing late entry into the school system. This is mainly because of reduced walking distances to school hence students are able to enroll at the age of six. The schools have also attracted more girls since they are able to learn close to home. The construction of the schools has also assisted in developing the spirit of ownership to the community through their involvement during construction in providing bricks and sand. B. Enhancement of School Quality and Efficiency. QUEST initiatives for enhancing school quality and efficiency include both teacher professional development and community mobilization for school improvement. The latter program also includes a small school/community grant scheme (i.e., QUEST Term Fund), that provides funding to school/communities who submit and achieve approval for school improvement projects. In addition a focus on building system-wide capacity for supporting sustained quality improvements is provided for through the establishment of school cluster networks and a mentor teacher program and through the facilitation of dialogue at the District, Divisional, and Central levels about systemic reforms and policy development that promote and sustain quality reforms. General observations. The Primary schools visited were in predominantly rural areas of Malawi and were characterized by a number of serious constraints to the quality of schooling that are not unknown to education professionals in Malawi. These include, but are not limited to: 8 · Environment not conducive to learning (see discussion in section following: Impact of Teacher Training) · Absence of pupil benches and desks in the lower Primary grades. · With the exception of a few teacher-made local materials, there were scarce teaching · and learning materials in most schools. · Few textbooks4 · Predominance of conventional question/answer, imitation, drill, and choral response instructional strategies · Teacher shortages · Pupil and teacher absenteeism (See below) · High teacher mobility5 In all schools there was substantial pupil absenteeism because of the national food production crisis and children were not attending classes because of “hunger”. On the average only 25% to 33% of the pupils were attending school at the time of the visits. Reportedly this was typical for the first term of this school year. Teachers suggested that attendance would improve in Term 2 because food was beginning to be more available. Teacher absenteeism was also reported as a problem with some reported incidences of teacher absenteeism as high as 5 to 7 school days out of a month. Most reports indicated that teacher absenteeism was related to the limited numbers of teacher houses in the school community. Impact of direct teacher training (i) Overview The QUEST teacher training program focuses on practical skills, pupil participation, diverse methods, utilization of local materials for teaching and learning in the classroom, teacher awareness of pupil participation and performance through ongoing tracking of pupil attendance, continuous assessment and individual learner support, and effective teacher supervision (i.e., for PEAs, Mentor Teachers, and administrative personnel at school and district levels). The initial objectives of the teacher training program as reported in the QUEST for Learning Proposal6 include: ÿ Use creative and diverse teaching methods in the classroom. ÿ Employ local materials and resources in teaching lessons. ÿ Make his/her classroom attractive for learning. ÿ Provide greater individual attention to pupils of both sexes and interact more with pupils in the classroom with particular awareness of gender balance. 4 In a few schools new textbooks supplied by the Canadian International Development Agency (CIDA) had been supplied and in some instances the books were being distributed on the day of the school visits. 5 Teacher mobility out of classrooms from one year to another has been observed to be as high as 50% (See Malawi Institute of Education, IEQ Project, Inside Story, Vol.1, No.1) 6 Save the Children/US, QUEST for Learning Proposal, September 1998, p. 15. 9 ÿ Apply continuous assessment of pupil learning and use the data in making promotion decisions. ÿ Collect and use data such as absence rates and drop out to identify problems such as those related to girls’ persistence at the school/zone-level and address them. Both quantitative and qualitative data were used to evaluate the impact of the teacher training program. Qualitative data through classroom observations, teacher surveys and focus group discussions were collected from the school visits. Quantitative data were provided from the set of QUEST studies that are used to provide ongoing formative assessment of the key school and community initiatives. (ii) Classroom Observations ÿ Creative and diverse teaching methods. Teachers in both the QUEST and non-QUEST schools demonstrated adequate class preparation through lesson plans that reflected sufficient detail and advanced planning. Teachers who have been trained under the QUEST program were observed to practice more diverse teaching methods than teachers who have not been exposed to the QUEST training. All teachers observed in QUEST schools used a variety of teaching methods and demonstrated ease in applying grouping strategies to enhance pupil participation. Classroom observations in the non-QUEST schools showed that pupils were eager to participate; however, teachers did not apply grouping strategies or other techniques to maximize opportunities for pupils to participate in class. Teachers in QUEST schools used a variety of methods including grouping strategies, role-playing, pair work, and integration of songs into classroom instruction. In spite of the fact that teachers in both QUEST schools clearly demonstrated their learning and application of many of the techniques trained under the QUEST program, they demonstrated weaknesses in their knowledge and application of pedagogical methods for teaching specific subject areas, namely basic literacy and numeracy. Though teachers who have been trained under the QUEST program demonstrate their ability to apply the methods learned, much of the teaching observed in classrooms failed to maximize the child’s opportunities for learning. Teaching continues to be dominated by the conventional strategies of demonstration, question/answer techniques, and drill though different contexts for learning such as grouping strategies, increased feedback to pupils, and pupil interaction have greatly improved. It should be emphasized that teachers do exercise the participatory approaches taught under the QUEST program; however, participatory approaches, though critical, do not in and of themselves maximize a child’s opportunities for learning, particularly independent learning, generalization of concepts and skills beyond the context of classroom instruction, and creative thinking. But research and experience have shown that schooling alone does not necessarily produce literacy or numeracy or the kinds of literacy and numeracy that students need in modern society. Innovative approaches for enhancing basic literacy [and numeracy] have common 10 elements that can be addressed through a better understanding of the nature of literacy and literacy education programs7…[and numeracy as well]. The importance of teachers’ further development of more effective ways of teaching the critical subject areas, particularly literacy enhancement that focuses on the exchange of meaning in the oral and printed word and higher order thinking and problem solving in mathematics, cannot be overstated. Novel methods used elsewhere in developing countries8 , approaches that are inconsistent with the use of the conventional practices of demonstration, imitation, and drill, should be explored in the region and piloted to determine their feasibility and effectiveness for the Malawi’s Primary schools. The QUEST Program in collaboration with MIE/IEQ and the MOEST provides a unique opportunity for these kinds of explorations. Through such collaboration and study stakeholders across Malawi learn together about the most appropriate intervention choices for improving Primary school education quality. ÿ Employ local materials and resources in teaching lessons. Teachers that had received the QUEST training were advantaged in their ability to construct teaching and learning materials from local materials (e.g., banana leaves, sticks, molded clay, fruits and seeds, etc). The majority of teachers used locally made materials in the classes observed in the QUEST schools. Only 50% of the teachers in the set of control schools used any teaching and learning materials other than the chalkboard, texts (when available) and pupil exercise books and rarely were these materials produced from local materials. ÿ Make his/her classroom attractive for learning. The classroom environments observed were not particularly stimulating in either the QUEST or non-QUEST schools even though most of the QUEST classrooms observed did have learning centers consisting of a table of materials, usually from local materials. These learning centers were not observed in the non-QUEST classrooms. It should not be assumed that teachers can develop adequate learning environments simply as a result of training. That is, stimulating learning environments require more than what teachers can supply themselves. Some materials are required, even over and beyond those that can be produced locally, particularly for literacy enhancement. Literacy is enhanced when children are taught in a print-rich environment and none of the classrooms observed would meet this expectation, even minimally. The teaching and learning of basic literacy is enhanced where the exposure to reading materials at different performance levels are readily available and opportunities for pupils to read independently and with peers are provided on a regular basis. Other than locally produced materials, there is no budget from which head teachers can source paper, poster board, markers and so on to be used to make materials. The QUEST grant scheme is one avenue whereby schools may access funds for use in improving the classroom environment. 7 UNESCO: Literacy and International Development[online] http://www.literacyonline.org 8 Some examples include: reflective practice, balanced literacy approach to multi-lingual literacy instruction (Guyana), interactive radio for mathematics or English (Republic of South Africa, Bolivia, Haiti), bilingual approaches to multi-lingual litracy (Republic of South Africa), whole language approaches to multi-lingual literacy (Zambia). 11 Additionally, most schools had few, if any, desks and/or small tables and chairs for pupils to work at, especially for the Junior Primary classes. One community activity that could be considered that would improve the situation in the interim until furniture is either produced locally or procured is to have communities produce mats from local materials to allow the children to sit on something other than the floor when learning. ÿ Provide greater individual attention to pupils of both sexes and interact more with pupils in the classroom with particular awareness of gender balance. Though some individual attention was given to pupils in both the QUEST and non-QUEST classrooms, the grouping strategies used by the QUEST-trained teachers provide more opportunity for individual attention. Pupil-teacher interaction was much more prevalent in the QUEST as compared to the non-QUEST classrooms; however, in neither the QUEST nor the non-QUEST classrooms were teachers observed to overtly facilitate pupil-initiated interactions and/or challenge their thinking beyond the immediate instructional content. The need to provide additional training and support to teachers to assist them in facilitating pupil￾initiated responses has been recognized by the SC/US as well and this will be an area of training that will be enhanced in the future. No favoritism related to gender was observed in any of the QUEST classrooms. In three of the sixteen non-QUEST classrooms teachers clearly favored boys as opposed to girls and in these classrooms girls rarely participated. ÿ Continuous assessment. In both QUEST and non-QUEST classrooms teachers provided feedback to pupils and most teachers reported that they regularly assessed their pupils’ performance through weekly tests and through assessment of homework and daily exercises. However, teachers’ knowledge of the way to conduct formal and informal assessments of their pupils’ learning and their utilization of assessment information for guiding day-to-day classroom instruction was considered weak in both QUEST and non-QUEST classrooms. Informal and formal pupil performance monitoring in the classroom should be linked to research-based standards in literacy and numeracy that are known by teachers, pupils and parents. Teachers should learn how to and be given substantial support in utilizing this information to guide their day-to-day instruction and to communicate with pupils and parents about their learning. Pupils should be more involved in monitoring their own learning and teachers should facilitate their active involvement. Real and sustained teacher change occurs through a developmental process whereby teachers are taught to recognize their own needs and taught to define in their own context quality classroom interventions. Teachers can learn much about his/her own teaching effectiveness through information gained by continuous assessment processes and therefore enhanced training in these processes should include skills in how to use pupil information to facilitate his/her own self-evaluation. 12 ÿ Collect and use data such as absence and dropout rates to identify problems such as those related to girls’ persistence at the school/zone-level and address them. Teachers reported that they have learned ways to track pupil attendance and overall the teachers and head teachers interviewed reported that they followed up with parents (often through the head teacher) when absenteeism appeared to be chronic. This was concurred with by parents that were interviewed in the parent meetings. (iii) Teacher surveys. A simple survey was administered to teachers from the QUEST schools visited to ascertain their views about how QUEST has benefited them. One hundred twenty-two (122) teachers completed these surveys (22 in Mangochi, 49 in Balaka, and 49 in Blantyre Rural). Responses of the teachers regarding the benefits, challenges, replicability and way forward for QUEST can be seen in Table 2. The most frequent response to questions about the benefits of training was in the area of producing teaching aids from local materials. Seventy￾two percent (72%) of the teachers named this as assisting them the most in their day-to-day instruction. Approximately 20% of the teachers reported that the pedagogical approaches for stimulating pupil participation and techniques to handle large classrooms (e.g., grouping strategies) were the most useful. Lesson planning and the use of continuous assessment were also named but to a lesser extent (16% and 12% respectively). The most frequent response to the question about which techniques were the most difficult to implement were group work (49%), role play (21%) and handling large classrooms (18%). The most frequent responses to techniques that should be replicated across Malawi included the use of local materials and applications of continuous assessment (27% and 12% respectively). Sixty-five percent (65%) of the teachers felt that the extent and audience of the teacher training program should be increased in future QUEST projects. The provision of teaching and learning materials and the provision of training on how to teach specific subjects were also mentioned, especially the teaching of reading. Table 2. Results from Teacher Survey Response from Teacher Frequency What aspect of QUEST training has assisted you most in your day-to-day teaching? Local produced teaching and learning materials 88 (72%) Pedagogical approaches to facilitate pupil participation and interaction (particularly questioning techniques). 27 (22%) Handling of large classes 25 (20%) Lesson plan development 20 (16%) Continuous assessment 15 (12%) Integrated curriculum 12 (10%) Individual instruction for accommodating slow learners 3 (2%) Which of the methods you learned under QUEST have been the most difficult to implement? Why? Group work 49 (40%) Dramatizing/Role Play/Games 26 (21%) Handling large classrooms 22 (18%) Using resource persons from community to assist in classroom 9 (7%) Integration of curriculum 5 (4%) Pupil Assessment 4 4 (3%) Production of teaching and learning materials 2 (2%) 13 Table 2 Most difficult to implement (Continued) Field Trips 1 1 (1%) What should be replicated across Malawi? Use of locally produced TLMs/Learning Centers 33 (27%) Use of continuous assessment 15 (12%) Group work 9 (7%) Handling of large classrooms 9 (7%) Lesson Plans 7 (6%) Integrated Curriculum 7 (6%) Supervision 3 (2%) Pupil-teacher relationships 3 (2%) Clustering of all schools 3 (2%) Team planning and team teaching 3 (2%) PTAs and school committee training 2 (2%) Future QUEST Programs More teacher training workshops 65 (53%) Expand to include Standards Five to Eight 55 (45%) Provide Teaching and Learning Materials 22 (18%) More training on monitoring pupil performance/attendance 16 (13%) Pedagogical Training for Specific Subjects 17 (14%) Reading (12) Music (3) PE (1) Creative Arts (1) Team Expansion to Other Districts 9 (7%) Involvement of Local Leaders 3 (2%) More modern methods 2 (2%) Introducing Academic Training 2 (2%) Integration of teacher training into certification programs 2 (2%) Special education training 1 (1%) (iv) Focus Group Discussions with Teachers Focus group discussions were held with teachers from the QUEST schools visited. Overall, teachers felt the most valuable aspect of the QUEST Project was the supervision that was provided them through the mentor teachers and through the school cluster network. Trained and untrained teachers alike unanimously expressed the need for additional training. In some of these discussions trained teachers (or teachers who are currently enrolled in the MITTEP Program) were asked to discuss some of the unique qualities of the QUEST training program as compared to methodologies taught in the teacher training colleges and in the MIITTEP Program. Even though trained teachers reported that they had learned many of the pedagogical methods promoted under the QUEST training program in their previous teacher training programs, they consistently reported that the on-site support provided through the mentor teacher program and teacher networking provided through the school cluster system was unique, extremely valuable and helped them successfully implement these good practices. When teachers were asked if they would be able to meet at the cluster level on a regular basis (e.g., monthly) for formal training or professional dialogue with their peers, they all agreed this would be possible and the teachers were very interested in the possibility of developing this further in future QUEST projects. The desire to extend the training program to teachers in Standards 5-8 was also reported in all of the teacher focus group discussions. 14 Many teachers expressed the need for further training in teaching specific subjects, particularly the teaching of reading, and went on to say that there should be some credit given for participation in the SC/US training that could be applied toward a teacher training certificate. Finally, it should be added that community members have enormous confidence in the teachers in their schools, recognize the value of the teacher training program and want it to be continued and extended to include the training of Standard 5 to Standard 8 students. Impact of Supervision Training in Support of Teacher Professional Development (i) Overview. The QUEST Project strategy for enhancing the quality and efficiency of education encompasses a strong capacity development program that not only supports teacher professional development as discussed above but also provides for the establishment of a network of support for teachers through the supervisory training of PEAs and the identification and training of Mentor Teachers who work within a school cluster network to provide support to teachers on-site and through cluster training. (ii) Impact ÿ School Cluster Networks and the Mentor Teacher Program The establishment of school cluster networks and the mentor teacher program was considered the most successful area of the QUEST Project. Teachers, community members, and education officials at the District, Division and Central levels all agreed that before the establishment of the school cluster and mentor teacher systems by QUEST there had been a serious gap in Malawi’s teacher support programs. It is well known that improvements in the quality of teaching and learning require on-site, school-based support and the cluster network and mentor teacher program provide a solution for Malawi. Extensive international experience shows that when teachers work together and collaborate new approaches to teaching are more likely to be sustained. At this point in time the system of local support to teachers through school cluster networks and the mentor teacher program is not sustainable in the current QUEST Districts. In order to facilitate the sustainability of this system, future QUEST programs should enhance the school cluster network to better support regular professional networking of teachers and opportunities for ongoing training through trained mentor teachers operating at the cluster level. The system should be institutionalized under the MOEST and supported by national policy that allows for (if not mandates that) schools be closed early once per month for teachers to attend cluster meetings. In order to maximize the benefit and sustainability of the cluster network system and mentor teacher program they should be linked to other teacher education initiatives to provide peer support and responsive follow-on training at the cluster and school level (e.g., zonal training through MSSP, monitoring and follow-up support to teachers working on their certification through the MITTEP Program, practicum for new Bachelor of Education programs through UPIC). 15 The goals and assessment results of the supervisory training program for PEA’s and Mentor Teachers in relation to teacher professional development are presented according to the stated goals of the supervisory program (See QUEST Proposal, p. 6) as follows. ÿ To promote the five areas of teacher education development described under the teacher training section above (See Impact of Direct Teacher Training, p. 6) The results of classroom observation, teacher surveys, and focus group discussions with teachers provide evidence that teachers are successful at implementing the methods taught by the QUEST teacher training program. These are discussed in the preceding sections. ÿ Provide regular supervision of teachers at the school. The supervision and support of teachers are provided regularly to schools through the mentor teacher and school cluster program. The supervision by PEAs varies from regular and productive teacher support to scarce involvement of the PEA. In the majority of situations where PEA supervision is not effective, the PEA lives outside the zone and has limited access and/or sufficient fuel to visit schools/communities. When PEA involvement has been a problem over the course of the QUEST Project, SC/US staff work closely with the District Monitoring Team (DMT) and education officials to ameliorate the situation. In some instances QUEST responds to this problem by constructing PEA houses in the district (e.g., often near the newly constructed Teacher Resource Centers). The collaboration of the DMT and SC/US in making mid-course corrections where problems exist relative to supervision has been successful and appreciated by District officials. This collaboration and problem solving are to be commended. ÿ Track areas of need for training in each school, cluster, and zone and conduct in￾service training regularly in response to those needs to promote professionalism within the school cluster network. This is another one of the high points of the QUEST Program. According to teacher and head teacher reports, the mentor teachers are intimately and actively involved with teachers at the school level and do organize training in accordance with defined needs. Institutionalizing the mentor teacher program and linking the program to existing teacher professional development programs in the Primary sector (e.g., UPIC, MSSP) would enhance the impact of this process immensely. Impact on Pupil Learning Outcome. (i). Assessment of Methodologies. In the first year of QUEST a partnership with MIE through the MIE/IEQ2 program, was established to collaboratively design and conduct a longitudinal study of the impact of the QUEST Program, and of pupil learning in particular. SC/US should be commended for its attention to and investment in a rigorous monitoring and evaluation system for QUEST. Their extensive data collection and utilization of a repeated measures design provide a rich source of data from which to learn about factors related to Primary school performance in Malawi over and beyond their immediate need to address specific QUEST performance 16 indicators. The study tracks pupil performance in Mathematics, Chichewa reading, and English reading using a set of performance-based, curriculum-bound instruments. School level information including teacher performance and demographics and community demographics are also collected as a part of the longitudinal study. Data collectors include trained education professionals who are not associated directly with SC/US program and the schools to which they are assigned to collect data from. Data are entered and analyzed by the SC/US analyst; however, MIE/IEQ2 staff also analyze the data providing for a cross-check on results. Baseline data were collected in the three target Districts in February 1999 and follow-on data collection occasions included October 1999 and October 2000. The 2001 teacher strike precluded data collection in October 2001. In October 2000 data from five matched schools in the Salima were added to the longitudinal study as a control measure; however, due to the 2001 teacher strike only one year of data is available for cross sectional comparisons with the treatment schools and districts. The overall goal of improving the quality of schooling is to positively impact pupil learning. The performance indicator given for the school quality result is: To achieve a 20% gain in pupil test scores. The language of this indicator should be revised in order to more clearly specify the target learning areas and means by which the indicator is to be measured (e.g., mathematics, reading passages, reading comprehension) and over what period of time (e.g., is it 20% gain across the life of the project or 20% gain on the average per year?). Furthermore, there is no indication that a 20% gain is necessarily related to an impact of the QUEST interventions. Gain should be assessed from a comparative perspective (i.e., Is a 20% gain in pupil performance in QUEST schools higher than that which is achieved in comparable non￾QUEST schools?). SC/US demonstrated its awareness of this issue by introducing the Salima schools in the analysis in October 2000 as a matched comparison group. Collection of the comparative data from the beginning (i.e., Feb ’99) would have been ideal. Tracking the treatment and control schools from baseline provides an opportunity to compare the learning growth patterns of pupils in QUEST and non-QUEST schools across the life of the project. The performance-based instruments used in the study are directly linked to the curriculum via the incorporation of passages taken directly from pupil texts. It may also be valuable to know if pupils are able to generalize what they learn in the classroom (i.e., from the school text) to reading experiences outside the classroom. Therefore, in the future SC/US may want to include leveled passages9 or other printed materials that are not taken from the pupils’ own textbooks. Given the multi-lingual context of Malawi SC/US may also want to consider monitoring pupil development in oral as well as written language. Finally, in regard to analysis procedures, when the focus of intervention is directed toward school reform (e.g., teacher development, community involvement) with pupil performance as the principal outcome of interest hierarchical linear modeling procedures offer a more sensitive, valid, and meaningful interpretation of results. The team may want to consider these analysis options in the future. 9 Leveled text in this context is taken to mean text that is graded according to the existing curriculum. 17 (ii) Study Results. The following presents results from the longitudinal study on pupil performance given by the analyses conducted at SC/US. Average mean scores for pupils at baseline (Feb 1999) and two follow-up assessments at the end of the school year in October 1999 and October 2000 are presented in Table 3. It can be seen in Table 3 that with the exception of Standard 2 English Passages, Standard 2 and Standard 3 English Comprehension, and Standard 4 Mathematics the performance indicator of 20% gain was successfully achieved. These data underscore two other critical issues in Malawi’s Primary sector. First of all, the gap between boys and girls reading performance is notable throughout (favoring boys), particularly in the lower standards. Secondly, the English literacy performance of Malawi Primary school pupils, even those participating in the QUEST classrooms is low. This observation is noteworthy considering that the results given for the post-test (Oct ’00) are results for pupils just completing Std 3, 4, and 5 reading passages from the year preceding. For example, most pupils who entered the program in Std 2 (Feb ’99) are completing Std 3 by the time of the final posttest (Oct ’00). Considering the longitudinal design, pupils read the same passage over time. Therefore, the results presented at the time of the Oct ’00 posttest represent the results of pupils reading a passage from the preceding school year (i.e., pupils completing Std 3 read the same Std 2 passage they read at baseline). Furthermore, the gap between performance in comprehension and reading of words on the reading passages subtest speaks to the lack of meaningful exchange of ideas and the chronic practice of choral reading and rote memorization in English instruction, in spite of the fact that teaching practices have improved to be more pupil centered under the QUEST Project. Table 3. Performance over time: Chichawa, English, Mathematics Class 'Feb-99 'Oct-99 'Oct-00 %Gain Chichewa Reading Passages (Mean % wds correct) Std 2 boys 14.04 32.75 53.67 39.63 Std 2 girls 6.46 16.40 39.43 32.97 Std 3 boys 42.11 60.14 80.08 37.97 Std 3 girls 21.28 45.00 77.49 56.21 Std 4 boys 68.46 83.77 89.61 21.15 Std 4 girls 70.49 85.34 94.86 24.37 English Reading Passages (Mean % wds correct) Std 2 boys 6.12 13.80 25.73 19.61 Std 2 girls 3.18 7.21 17.83 14.65 Std 3 boys 13.91 34.72 57.81 43.90 Std 3 girls 4.95 21.37 42.19 37.24 Std 4 boys 44.97 70.04 80.04 35.07 Std 4 girls 41.65 54.81 75.51 33.86 English Reading Comprehension (Mean % correct) Std 2 3.45 Std 3 17.13 12.88 Std 4 12.50 25.00 33.16 20.66 Mathematics (Mean % correct) Std 2 40.41 51.58 67.79 27.38 Std 3 62.24 69.12 82.54 20.30 Std 4 72.01 79.04 89.05 17.04 For the Reading Passages studies, pupils in Std 2, for example, read a Std 2 passage at baseline (Feb’99) and read the same Std 2 passage at the end of that school year (Oct’99), then again in the following year when most had been promoted to Std3 (Oct’00). The same procedure was applied for pupils who were in Std 3 and Std 4 at the baseline assessment. The Reading Comprehension Questions were based on the passages read. 18 In October 2000, data were collected in five schools in Salima to provide a comparative view. A cross-sectional comparative study was conducted to investigate group differences between QUEST and non-QUEST schools. The results for Chichawa, English and mathematics are shown in Figure 1, Figure 2, and Figure 2, respectively (On the following page). 0 20 40 60 80 % Figure 2. MEAN PERCENTAGE ENGLISH SCORES BETWEENQUEST AND NON QUEST AREAS QUEST 21.78 50 77.76 NON-QUEST 4.93 43.66 71.04 standard 3 standard 4 standard 5 0 20 40 60 80 100 % Figure 1. MEAN PERCENTAGE CHICHEWA READING SCORES IN QUEST AND NON QUEST AREAS QUEST 46.55 72.13 92.24 NON-QUEST 11.5 79.95 88.73 standard 3 standard 4 standard 5 0 20 40 60 80 100 % Figure 3. MEAN PERCENTAGE MATHEMATICS SCORE IN QUEST AND NON QUEST AREAS QUEST 67.79 82.54 89.05 NON-QUEST 48.46 65.41 73.39 standard 3 standard 4 standard 5 These results represent post-test comparisons of the performance of pupils who were enrolled in QUEST schools for two years compared to the controls. That is, Pupils in Std 3, Std 4, and Std 5 began the program as Std 2, Std 3, and Std 4 pupils, respectively. It can be seen in Figure 1 and Figure 2 that the advantage in performance is notably greater for the QUEST Schools for Std 3 Chichewa and Std 3 English as compared to the performance advantage shown for Std 4 and Std 5 pupils. It will be important to see if the early learning advantage seen for pupils in QUEST Schools is generalized to the subsequent grades in the following years. For example, the Std 3 For Mathematics, pupils in the QUEST Schools consistently performed, on the average, 17.4 percentage points higher than pupils in non￾QUEST schools. 19 Impact on Pupil Dropout and Repetition. (i) Assessment of Methodologies. Due to the delay and reported unreliability of the Primary School Census data, SC/US conducted a rigorous pupil tracking system to assess the level of pupil dropout and repetition in the target schools. All Std 1 – 4 pupils in the three target Districts were entered into a database in June-July 2000. These pupils were then followed in the subsequent school year, March 2001. Pupils who had not transferred to another school and who did not register for school in the following year were counted as a drop out. Pupils who attended school, but were held back were counted as a repeater. The numbers of dropouts and repeaters were divided by the original number of pupils in the database to ascertain the rate of dropout and repetition for the study. These rates were then compared to the National Figures given by the Primary School Census data from 1999. Although the methods applied by SC/US were somewhat different than those used to calculate pupil dropout and repetition in the National Census, the end result of these two calculations are, theoretically, similar and deemed comparable. (ii) Results of study on pupil dropout and repetition. Table 4 presents the percentage of pupils who dropped out of school between the 2000 and 2001 school years given by the SC/US study described above as well as the number of pupils from the 2000 cohort who were retained in the same grade in 2001. These figures are compared to the percentages of pupils who dropped out and who were held back (i.e., repeaters) between the 1998 and 1999 school years immediately preceding the beginning of the QUEST Project given by the 1999 MOEST Education Basic Statistics. In Table 4 it can be seen that the rates of pupil dropout has decreased in every District and in every Standard since the outset of the program in 1999 (Percentage of pupils who dropped out between the 1998 and 1999 school years). The reduction in dropout rate is greatest for the Balaka and Blantyre Rural Districts, approximating the target of 10% reduction in these Districts for Standards 1 and Standards 2. Less of an impact is seen for Mangochi overall and for Standard 3 and Standard 4 in Balaka and Blantyre Rural. These results should be considered with caution, because there are no available statistics for Districts outside of those where QUEST is being implemented. It could be that these results simply reflect national trends in dropout. On the other hand such trends are rarely seen as dramatic as these results and therefore it is likely that the decreases in dropout shown here are a demonstration of impact, improved efficiency in the QUEST target Districts. Table 4. Percentage of Dropout in QUEST District: Performance of QUEST Schools Districts after 2 Years of Intervention compared to Baseline Performance given by MOEST Data*. District Standard 1 Standard 2 Standard 3 Standard 4 Boys Girls Boys Girls Boys Girls Boys Girls Mangochi ’98-‘99 16.5 16.6 15.1 13.9 11.6 11.5 10.5 10.7 Mangochi ’00-‘01 11.4 11.0 8.5 7.9 7.7 7.9 8.6 10.1 Balaka ’98-‘99 13.0 12.3 10.3 9.3 9.0 8.8 7.8 8.5 Balaka ’00-01 3.0 2.3 1.7 1.3 3.7 3.7 2.6 5.1 Blantyre Rural ’98-‘99 11.7 12.1 10.5 10.4 8.9 9.3 9.0 11.0 Blantyre Rural ’00-‘01 3.9 3.4 3.4 2.7 5.4 4.9 5.0 7.0 *Residual dropout from MOEST 1999 statistics were used as baseline comparison. 20 Table 5 shows the dramatic increases in the percentage of pupils who are being held back a grade in comparison with national statistics given at baseline and reflects the trade-offs that take place when increased attention is given to pupil performance and readiness for promotion. Under the QUEST Project teachers have learned ways to monitor pupil progress and make decisions about promotion through formal assessment at the end of the year. When teachers are aware of and care about pupils’ performance they are less likely to promote them to a class where they are assured to fail. This attention to the individual performance levels of pupils is recognized as a very positive outcome of the QUEST Project and should be viewed as such in spite of increases in pupil repetition. Until the quality of schooling is improved sufficiently enough that pupils attain basic literacy and numeracy at a rate consistent with grade expectations (and these curriculum standards are appropriately aligned with research-based developmental standards) substantial advances in schooling efficiency will not be realized. Table 5. Percentage of Pupil Repetition in QUEST District: Performance of QUEST Schools Districts after 2 Years of Intervention compared to Baseline Performance given by MOEST Data*. District Standard 1 Standard 2 Standard 3 Standard 4 Boys Girls Boys Girls Boys Girls Boys Girls Mangochi ’98-‘99 16.5 16.8 14.8 15.4 14.8 14.7 10.0 10.8 Mangochi ’00-‘01 38.9 41.0 28.8 28.5 26.8 35.5 22.9 22.4 Machinga ’97-’98 (Totals Only)** Total % 18.0 Total % 14.7 Total % 14.2 Total % 10.8 Balaka ’00-01 (Average across sex) Ave % 33.5 Ave% 21.7 Ave% 30.3 Ave % 16.9 Blantyre Rural ’98-‘99 16.7 18.3 16.3 15.6 17.1 18.5 11.3 10.7 Blantyre Rural ’00-‘01 41.8 41.5 30.3 28.3 38.0 34.7 23.4 22.1 *MOEST 1999 statistics were used as baseline comparison. **Repetition Rate for Balaka could not be calculated because of District change between 1998 and1999. Therefore, the overall repetition rate for Machinga was used as a comparison. Impact of Integrated Curriculum Pilot Study (i) Background. SC/US developed an integrated approach to implement the curriculum in the Junior Primary Standards during the VBS pilot from 1994 to 1998. Essentially, this approach integrated the music, physical education, and creative arts into the core subjects of mathematics, Chichewa, English, and general studies. In partnership with MIE and the MOEST, SC/US incorporated a pilot study of the integrated curriculum into the QUEST Project. Standard 1-4 teachers in the QUEST schools in the Balaka District were trained to implement the integrated curriculum and post-test results pupil performance results were compared to three sets of comparison schools: QUEST schools in Balaka, Control Schools in Mwanza and Machinga. (ii) Results of study. The results of the study are shown in Figures 4 – 6. The impact of the integrated curriculum is demonstrated by higher pupil scores, on the average, for mathematics and Chichewa in the schools where the integrated curriculum was being implemented. English literacy on the other hand was not enhanced as a result of the integrated curriculum. 21 Standard 3 pupil performance in Chichewa, English, and Mathematics for pupils in integrated curriculum classrooms (QUEST, Balaka) are compared to three controls. In addition to the QUEST-Mangochi both Machinga and Mwanze were used as a control. Machinga is more similar to Balaka in that Yao is spoken by the children in addition to Chichewa. The low scores from Mwanza reflect the constraint that a different home language (Yao in this case) can present to literacy learning in a second language (in this instance, Chichewa). It is also important to note the large performance gap between girls and boys in Chichewa and Mathematics. 58 60 62 64 66 68 70 72 Figure 4. STANDARD 3 CHICHEWA PASSAGES Boys 70.68 65.23 64.03 69.25 Girls 69.7 68.07 63.25 66.47 QUEST Integrated QUEST Only Machinga Mwanza 60 62 64 66 68 70 72 Figure 6. STANDARD 3 MATHEMATICS Boys 70.33 68.73 67.98 67.39 Girls 67.24 65.78 66.3 64.49 QUEST Integrated QUEST Only Machinga Mwanza 0 20 40 60 Figure 5. PERCENT ENGLISH SCORES BY GROUP AND SEX Boys 30.87 29 33.38 25.22 Girls 30.42 30 35.72 27.44 QUEST Integrated QUEST Only Machinga Mwanza 22 Impact of Community Training and Mobilization for School Improvement (i) Background The community support program is supported by the training of PTA members, school committee members, and community leaders in the following areas: · Participation in school management. · Participation in implementing the school curriculum. · Following up on pupil absenteeism and dropout, with particular focus on girl participation. · Holding regular meetings to discuss the quality of schooling for pupils. · Providing storage of distributed learning materials as needed. · Identifying and prioritizing school improvement priorities through the development of the QUEST Term Plan/Initiative and develop proposals to the QUEST Fund where plans cannot be met by local resources. The PEAs and CDAs are the primary trainers of school committees and PTAs and are key in the development of sustained mobilization of community involvement in the school. One critical factor in the ongoing support program for communities is the residency of the PEA within the zone. As discussed above, this constraint has been alleviated to some degree by the construction of PEA houses within the zone. The impact of the community development program was assessed through two means: 1) focus group discussions with community members, including parents, PTA officials, and school committee members, community leaders, and pupils; 2) review of analyses conducted by SC/US relative to community involvement. (ii) Community Focus Group Discussions In the majority of the schools/communities visited a large number of community members attended the meetings held and enthusiastically contributed to the discussions. The two most frequent responses from community members regarding the benefit of the training was their learning about the roles and responsibilities of the community in improving school quality and their learning about ways to encourage pupil attendance and discourage dropout. Though communities had varied levels of participation in the school, all community members reported that they value the training immensely and would like to have more training, particularly in the area of planning for school improvement (e.g., developing a term plan/initiative) and developing proposals to request financial support from the District. Community members also expressed the need for SC/US training to include traditional and/or other community leaders. Community school relationships were in general viewed as very good and community members had substantial confidence in their teachers and the methods they employed to enhance pupil learning. Community members unanimously indicated that they would like to see the teacher training extended to Standards 5-8 (i.e., where schools included these 23 standards)10. Teacher absenteeism, viewed as a problem by some, was attributed to teachers living outside of the District and teacher housing was expressed as a high priority to resolve teacher absenteeism. School committees all reported that they met on a regular basis. It was noted that when a traditional leader (e.g., Community Headman or Headwoman) was involved there seemed to be substantially more enthusiasm, involvement, and unity about community goals and objectives relative to school improvement. Clearly, school committees, PTAs, and community leaders are eager to have the involvement of the QUEST Program, yet variable involvement in school improvement exists across the communities served and sustained involvement, though not well defined, appears to be lacking in many communities. Therefore, more regular community facilitation and support may be necessary from the PEAs and/or CDAs to establish sustained community involvement in school improvement. The CDA seems to be the individual who has the most potential as a community facilitator as community mobilization is their area of professional expertise.11 The CDAs work very closely with PEAs in their community facilitation work but have expressed a desire to work more closely with the SC/US staff. The CDA is also in a position to link education development activities in the community with other sector development projects. This latter asset may be instrumental in providing linkages between the SC/US education program and SC/US programs designed to mitigate the HIV/AIDS problem such as COPE. (iii) Focus Group Discussions with Pupils Pupils interviewed were not necessarily knowledgeable about the QUEST Project per se; however, the pupils had very favorable comments about their schools and their teachers. Their sense of ownership and pride toward their schools was demonstrated in their expressed willingness to participate in planting flowers and keeping the grounds neat as well as their attentiveness and eagerness to participate in the classroom. The pupils demonstrated a very positive opinion of and much respect for their teachers and reported that they are able to discuss problems openly with their teachers. In particularly, the pupils were cognizant and happy that corporal punishment is not used by their teachers. Teachers as well showed respect for pupils in the classroom. Pupils were aware of the scarce numbers of textbooks and teaching and learning materials and wait patiently for the day they will have textbooks. Collaboration with Partners The specific aim of the QUEST Project is to decentralize action to improve the quality of education at the zone, cluster and school level. To this end SC/US works together with a number of education development partners in Malawi including the Malawi Institute of Education (MIE), Ministry of Education Science and Technology (MOEST), and Ministry of Gender, Youth, and Community Development (MOGYC), Creative Center for Community Mobilization (CRECCOM), and finally various donor organizations active in the education 10 Communities members where the school was a Junior Primary community schools (Std 1-4) all expressed the desire to expand the number of school blocks to accommodate Std 5-8 classrooms. 11 CDAs are certified in community development and have had special training in community mobilization. 24 sector (e.g., DFID, GTZ, and CIDA). In this section we discuss the nature and implications of these relationships in relation to the success of the QUEST Project. (i) Interviews with MOEST Officials at Central and non-Central levels. ÿ Education Headquarters, Lilongwe The Director of Basic Education, Director of Planning, and the Director of Methods and Advisory Services were interviewed to ascertain their views and levels of involvement with the QUEST Project. These central education officials, though not as actively involved as education administration officials at Divisional and Central levels, are keenly aware of all of the QUEST activities and extremely supportive of the work conducted by the project. ÿ Divisional Managers and Planners Both Divisional Managers felt that QUEST has had a positive and significant impact on the school system. “QUEST is the best project in my Division”, said one of the Managers. The managers felt that training of teachers and mentoring has helped in improving teaching practices while on the other hand the training of school committee/PTA changed their perspective from spectators to partners. The Managers also said that they were personally involved and were members of the District Monitoring Team. As monitors, they were going out into the concerned Districts to check on the performance of the project. Monitoring helps in identifying problems and gaps in the project hence have timely interventions. The Planner of Southwest Division was involved in the evaluation of term plans/initiatives that emanated from QUEST District schools. There was, however, a concern that some initiation ceremonies are still overlapping with the school calendar, especially in Mangochi District despite the Ministry’s effort to extend the holiday between first and second term from two to four weeks. The Manager therefore expressed the need to have more sensitization to community leaders in areas where the initiation ceremonies are practiced so that appropriate compromise could be reached. The Managers indicated the need to continue QUEST in the three Districts, and particular emphasis was mentioned on Mangochi district, which still has outstanding problems. The Managers also indicated the need to replicate the project to other Districts. Priority was given to Machinga which shares boarders with Mangochi hence having common problems. ÿ District Education Managers All District Managers felt that QUEST has had a positive and important influence on the District’s education system because of the training of teachers, strengthening of supervision through the mentors and PEAs and also through training of school committee members, PTA members and community leaders. QUEST also involved the District Managers a lot. All the managers have participated actively as key members of district monitoring teams. Due to their involvement, the Managers have been able to support the PEAs strongly in their supervisory functions. 25 To make sure that the involvement of the community is sustained, a separate budget line item was created in the National and District Budget. The line item has been introduced into the new national budget, which will be considered shortly by the National Assembly. There is, however, a common concern that the term plans/initiatives presented by the school committees are not prepared properly. Most often, the term plans are not addressing issues relevant to quality. Further training in preparation of term plans was therefore considered necessary so that the term plans should be kept as a vital asset to school quality improvement. The training would also help to minimize delays in implementation of term plans since turning down of plans would be minimized. The need for reviewing the process of approving and funding term plans was also highlighted. In order to ease the burden on the community, the process needs to be streamlined. The streamlining could also help in solving the problem of lack of feedback to communities especially on proposals that are turned down. (ii) Interviews with MOGYCS officials. ÿ District Community Development Officer (DCDO) All the DCDOs mentioned that activities QUEST have had a sound influence in making the communities’ relationship with the schools more meaningful. They further mentioned that the direct involvement of the officers has provided a critical leadership model to the community leaders. The officers also appreciated the QUEST mentorship program as it has assisted in improving teaching practices in the schools. Two concerns were, however, mentioned. One was on the weak relationship between Community Development Assistants (CDA) and SC/US Trainers and the second was on delays on delivery of construction of materials to the community. The need to strengthen the relationship of the CDA and SC/US Trainer was considered important. On construction materials to the community, measures to minimize delays in delivery need to be put in place in future projects. (iii) Creative Center for Community Mobilization (CRECCOM) The Malawian NGO CRECCOM has been conducting a national campaign for improving education through community involvement under the USAID-funded Social Mobilization for Education Quality (SMCEQ) Project since 1998. Although CRECCOM works in the same Districts where QUEST is implemented, the NGO conducts its social mobilization program throughout Malawi and provides a more general action-oriented social mobilization campaign for education quality improvement rather than providing specific training approaches taken by the QUEST Project. In this way the two programs complement each other well. Indeed, QUEST and CRECCOM began working collaboratively on their training schemes for communities in 1999 in Mangochi; however, because of the demands and differences in schedules this collaboration was not continued after that. There is enormous potential for these two community development (i.e., for education) programs to work side by side and in complement. If schedules were coordinated and training programs were drawn up to insure that they built on each other there could be much value added to the community development programs for education improvement. Without this coordination the risk of duplication and confusion on the part of communities is great. For example, both CRECCOM and QUEST 26 support the development of community action plans for education improvement. This is counter productive and should be discouraged. Schools/communities should work together to develop a single action plan focusing on school quality improvement and the action planning process should be integrated into existing District and Division programs that support community/school action for education development. Secondly, both CRECCOM and QUEST train PEAs and CDAs in techniques for mobilizing PTAs, School Committees, and Community Leaders for school improvement and rely on them to provide follow-up to communities and feedback to the organization. As the goal is a common one the involvement of PEAs and CDAs in community action would seem to be much more productive if there were continuity in training and a single agency to interface with for follow-on support to communities and for monitoring progress. (iv) Malawi Institute for Education (MIE) and Improving Education Quality 2 (IEQ2) The partnership between QUEST and MIE in general, and their research activities supported through the IEQ2 Project in particular, was established to collaborate on a longitudinal study to be used in part to monitor the progress of the QUEST Project. However, the study also provides a rich source of data from which to learn more about the various factors that influence school quality improvement. The outcomes of this study have been used in collaboration with IEQ/Malawi to provide a context of discussion about education quality in National and District Forums and to inform policy. A second collaboration is between MIE and QUEST, the partnership to conduct the Integrated Curriculum Pilot Study. According to the Director of MIE, the relationship between MIE and SC/US has been very positive and he would be pleased if the MIE and SC/US would continue to work closely together in the future. One suggestion made by the Director was that the roles and responsibilities of SC/US and IEQ/Malawi should be more clearly defined. The first step to this end would be to formalize the partnership relative to the work in continuous assessment through a revision of the current Memorandum of Understanding between SC/US and IEQ/Malawi. In our discussion we outlined a three-step process by which MIE and SC/US could collaborate on future applied education research projects to support the quality improvements in Primary schooling. In this process education research priorities would first be explored through a small controlled feasibility study (e.g., the feasibility of a new instructional approaches for literacy enhancement). In the first phase MIE would take the lead, yet SC/US would be involved in all aspects of the study. If the result of the feasibility study were positive and the MOEST supported the continuation of the study, SC/US would then take the lead in piloting the program in a few selected Districts in the country under, for example, the QUEST Project with the involvement of MIE. This latter study would provide the MOEST the information it needs to ascertain the approaches’ viability for full integration into the Nation’s curriculum. The ongoing feasibility study on applications of continuous assessment in Malawi’s Primary schools is, in some sense, a model of this process; that is, in the first phase. A continuation of QUEST in collaboration with MIE/IEQ may provide the opportunity to take this feasibility study to the second and third phases as described above. 27 Impact of QUEST in Relation to the USAID Results Framework The overall strategy for USAID/Malawi’s Strategic Objective 4 (SO4) is to support the development of an education sector reform wherein schools are providing an environment in which the majority of children are learning. The QUEST Project addresses this overall goal directly as it has targeted and been successful in achieving improved pupil performance among Junior Primary pupils in the three target Districts. Impact will be addressed according to the two immediate results under the SO4, as follows: (i) IR4.1. Effective schools developed in targeted areas. Effectiveness in education is achieved when teachers are able to translate educational policies and new methodologies into improved student learning in the classroom. It is determined as well when the whole school environment is improved to support teachers. The QUEST Project, by working simultaneously to build capacity among teachers and education support systems (e.g., school cluster networks and communities) at the local level and use information gained from interventions to inform and support policy development that will lead to sustained quality improvement, has done much to support this intermediate result. The most valuable impacts relative to IR4.1 in the success that QUEST are: ÿ Facilitation of substantial non-Central involvement in Primary school quality improvement. The importance of support to teachers and schools by education authority and policy cannot be overstated and is often hard to come by. QUEST has been enormously successful in focusing the attentions of the education administrative community at all levels of the system on quality improvement issues and solutions in the Primary sub-sector. This support is paramount for obtaining sustainable education reform. Continuation of SC/US involvement in furthering the capacity of the education authority to support Primary education is encouraged. ÿ Establishing systems for enhancing teacher professionalism and development through school cluster networks and a mentor teacher program. For teachers to be able to sustain classroom reforms (achieved through any teacher professional development program, not just QUEST) it is essential that they have the opportunity to dialogue with their peers and to seek council from a master teacher who is readily available locally. The QUEST Project has established for the first time in Malawi a system for providing this support to Primary school teachers through its cluster networks and mentor teacher program. The further enhancement and institutionalization of these systems would be invaluable for sustained classroom reforms. ÿ Building capacity of communities to participate in education quality improvement. QUEST has successfully engaged communities in education development and management of schools and provides, through the QUEST Fund, an avenue for schools to source additional funds to meet school improvement priorities. Community involvement is essential for furthering the effectiveness of schooling given the financial constraints faced by the MOEST today. The success of this component varies with each community and further enhancement of the facilitation, 28 support, and monitoring systems for community development will be necessary to achieve sustained community involvement in education. ÿ Teacher professional development. The teacher training program of QUEST has resulted in improved pupil performance, particularly in Chichewa literacy and mathematics. Therefore, it can be said that teachers in these classrooms are more effective. However, the constraints to effective teacher preparation and training in Malawi are great and much work is needed to further enhance the quality of classroom instruction. QUEST in collaboration with MIE/IEQ has proved successful in conducting applied education research that can be used to inform education practices and policy relative to Primary schooling. The continuation of this effort, particularly with regard to identifying and piloting literacy and numeracy enhancement strategies that can be applied nationally are highly recommended. (ii) IR4.2. Policy Reforms and Investment Strategy Formulated. A central aim of the QUEST Project is to decentralize the quality improvement initiatives for Primary education. Decentralization of responsibilities and services is a primary aim of the MOEST sector strategy. To this end QUEST deliberately works toward supporting non￾Central and Central decision-makers through provision of opportunities for open dialogue and debate regarding education quality improvement with a common goal to inform policy development with information gained from non-Central sources and from results of ongoing applied research. There is considerable enthusiasm for the advocacy that QUEST provides for quality Primary schooling, particularly in the success the program has had in linking lower levels of the system to higher levels of education authority. QUEST has also worked with Central and non-Central authority to promote policy reforms in the following areas: 1) Collision of initiation ceremonies with school calendar; and 2) Increasing term of office for school committee members to 3 as opposed to 1 year. Impact of QUEST in Relation to Policy Investment Framework for Education The QUEST Project serves to support the sector strategy by directly addressing many of the education priorities given by the MOEST Policy Investment Framework (2001). The following lists components of QUEST that directly support the MOEST in achieving its objectives. ÿ Decentralization. Support the decentralization of education through capacity development activities at the District and Division levels, particularly with regard to the capacity for non-central involvement in coordinating, monitoring, and supporting education quality improvement initiatives at the local level. ÿ Increased access. One of the strategies set in the Policy and Investment Framework (PIF) is to build more schools and classrooms based on clearly stated needs and priorities especially in areas where there is under-provision. QUEST has supported the strategy by reaching the most disadvantaged areas. 29 ÿ Gender equity. QUEST has been successful in creating a conducive environment for both boys and girls in its target Districts. Through teacher orientation, teachers have better managed to motivate and support boys and girls equally in schools. ÿ Increased efficiency. QUEST has been successful in reducing pupil dropout and addressing chronic pupil absenteeism through collaborative strategies between the school and the community and has assisted in the provision of school health and sanitation through provision of toilets and boreholes. ÿ Involvement of parents and communities. QUEST has directly responded to the sector strategy to involve parents and communities in school management through its training and support program for school committees, PTAs and community leaders. V. SUMMARY OF FINDINGS Overall SC/US has achieved its central aim to decentralize action to improve the quality of education at the zone, cluster and school level and has, for the most part, achieved its targets for enhancing the access, quality and efficiency of primary school education in the three target Districts. The results are summarized in the following: ÿ MOEST officials at all levels of the system are keenly aware of the activities of the QUEST Project and strongly support the program. At non-central levels, education officials are directly involved in implementation by advising and monitoring activities and by administering the QUEST Fund. ÿ Thirty-three new community schools have been established with enrolments of 11,387 pupils, 69% of the target of 16,500. Construction is near to completion with 64% of all construction completed. ÿ Teachers trained under QUEST practice what they learn in classrooms and classroom environments and teaching in QUEST schools is characteristically more interactive and more creative as compared to teaching in non-QUEST schools. Teachers value their QUEST training enormously, have more confidence as a result and would like to participate in more training in the future. ÿ The teaching of literacy and numeracy, though improved, does not maximize pupils’ opportunities for meaningful application of new skills and concepts and rarely provides opportunities for creative thinking, self-initiated learning, and problem solving. Exploration of novel and effective ways to enhance the teaching and learning of literacy and numeracy in Primary school classrooms is critical. ÿ It should not be assumed that teachers can develop adequate learning environments simply as a result of training. Some materials are required, even over and beyond those that can be produced locally, particularly for literacy enhancement. Avenues for supplying materials that can be used for teaching need to be explored further. 30 ÿ Teachers are knowledgeable about continuous assessment but their ability to use such information to guide their day-to-day practices and to engage pupil responsibility in learning needs to be strengthened. ÿ The School Cluster System and Mentor Teacher Program are among the most successful components of QUEST and support to facilitate the enhancement and institutionalization of these local support systems is basic for sustained classroom reform. ÿ Community training has been successful, though community involvement in school improvement is variable across communities. More systematic support, facilitation, and monitoring of community practices in school involvement is required for this result to be sustained. ÿ The target of 20% gain in pupil performance was met in the first two years of the project for the majority of subjects taught for Std 1 to Std 4. This should be taken as, overall, successful achievement of the performance indicator for school quality. ÿ With the exception of Standard 4 Chichewa, after two years, pupils in QUEST schools outperformed those in non-QUEST schools in mathematics, Chichewa (words in passages), and English (words in passages). ÿ The rate of pupil dropout decreased in all QUEST districts for Standard 1 through Standard 4 over the first two years of the project with the targeted 10% reduction in pupil dropout approximated in Standard 1 and Standard 2 for Balaka and Blantyre rural. Results for the final year of QUEST will not be obtainable until 2003. ÿ Results from the integrated curriculum pilot study showed this method to affect a performance advantage for Chichewa and mathematics, but not English. ÿ QUEST’s partnership with MIE and IEQ/Malawi has been extremely productive and encouragement of these partnerships to continue to explore avenues for enhancing the literacy and numeracy teaching strategies in junior primary schools is advised. ÿ Though successful, the accomplishments made by QUEST are not considered to be sustainable at this time and it would be premature to discontinue the support to current districts until more sustainable reform is achieved. Two avenues for effecting sustained change are: 1) enhance and institutionalize the School Cluster System and Mentor Teacher Program and link these system to Malawi’s current and future teacher education programs; 2) further enhance the facilitation, support and monitoring of community participation until a measure of sustained involvement is achieved. ÿ QUEST has positively contributed to the overall SO4 strategy and has had a positive impact on the two intermediate results of the USAID/MALAWI SO4 Results Framework. ÿ QUEST has contributed substantially to the successful accomplishment of the revised sector strategy as presented in the Policy Investment Framework (PIF). 31 VI. LESSONS LEARNED A process of evaluation assists in the identification of lessons learned in the implementation of the different activities of the project. Herein follow the lessons that were drawn from a study of the project work. These concrete and practical observations can serve as important contributions to the on-going developmental efforts of all QUEST stakeholders. In particular, such lessons can provide very useful guidelines to a more effective on-going QUEST project. The lessons have been categorized into five areas. 1. Sensitization and Orientation of Members of Staff and Partners a) Community mobilization is an effective way to sensitize communities in the identification and solution of problems. b) A program of specific activities has to be developed to maintain the active participation of community stakeholders in the improvement of the school. c) Community participation needs constant nurturing and follow-up in the face of numerous obstacles. d) The community school is a valuable asset/resource to the community – community center, literacy program site, cultural/social venue. e) The direct involvement of MOEST officials is a valuable asset to a sound community/school partnership. f) Project involvement with a national professional institution (MIE) provides appropriate professional/technical support for project activities, e.g., research. 2) Construction -- Activities related to village-based schools and houses for Primary Education Advisors (PEAs) a) A key person in providing motivation to the villages is the Clerks of Works, but he must be sensitized to the priority needs of the communities. b) Many communities did not demonstrate sufficient persistence when difficult times arose; continued involvement is difficult in such cases. c) Building a school for their children served as a strong motivation for community participation. d) The leadership provided by the Traditional Leaders is a key factor to the involvement of the community. 32 3. Teacher Training on Creative Methods of Teaching a) The production of instructional materials (for teaching and learning) is still inadequate. b) Teachers should be assisted to support remedial learner because many feel that they are unprepared to work with slow learners. c) Teachers do not utilize information on pupil performance to guide their practices and communication with the parents and the pupils. d) The learning of pupils in class is frequently affected by the mobility of teachers, and the training of the latter is thus neutralized. e) Effective teacher in-service training is critical to on-going efforts at quality education in a school. f) It should not be assumed that teachers can develop adequate learning environments simply as a result of training. 4. School Committee and Local Leaders’ Training a) Training in itself will not assist communities develop sustained involvement in school improvement. b) The annual turn over rate in the membership of the School Committees influences the on-going involvement of many members. 5. Teacher Supervision a) The use of Mentor Teachers has increased the frequency of on-site support to teachers. b) In-service training in facilitation skills is necessary for the Education Trainers. c) Teachers embrace the idea of monthly meetings at the Cluster level for training and professional exchange. d) The choice of a capable Mentor Teacher is essential since she/he be able to support the teachers in her/his Cluster. e) Teacher supervision is critical for effective in-service teacher training over the long￾term. 33 6. QUEST Term Plan/Initiative a) The identification of needs and priorities can be done effectively by the communities themselves. b) There is a frequent misunderstanding in communities concerning the plan/initiative having a direct influence on the teaching and learning in the classroom. c) The various phases of the plan/initiative process – planning, appraisal, and approval are very long, resulting in a delay of disbursement of funds and the execution of the plan/initiative. d) Partners frequently forget that the Term Plan/Initiative is intended to be an exercise in decision-making and not only a matter of providing a service to the school. e) Action plans for school improvement proposed and supported by different groups are frequently counter productive. VII. RECOMMENDATIONS Inextricably, a comprehensive evaluation looks to the formulation of recommendations intended to provide bases for future developmental project activity of current and future partners. A total of thirty-seven recommendations have been developed by the team in the context of the various stakeholders of QUEST. Save the Children Federation/US A. Project Management 1. The materials management for the Term Plan/Initiative Program should be strengthened to minimize delays in the delivery of approved items B. Project Programs 1. The Term Plan/Initiative Program should be improved: · more training of PTAs and School Committees on the preparation of plans · stronger collaboration of teachers in the preparation of the plans · a review of procedures and process to streamline steps involved. 2. The Training of Teachers (TOT) Program should be expanded to Standards 5- 8 so that its pedagogical benefits can be provided to all teachers in a school. 3. The integration of the QUEST TOT Program into the MOEST Teacher Certification Scheme should be explored, so as to improve teacher career planning, professional growth, and motivation to improve oneself. 4. In order to provide for appropriate continuity in the supervision of the Term/Initiative plans, the term of office of the School Committee officers should be extended to three (3) years. 34 5. The innovative Mentor Teacher Program should be replicated, being as it is the key to sound supervision of teachers. 6. Psycho-social issues should be introduced into the training programs for School Committees, PTAs and local leaders. 7. A re-evaluation and restructure of the Term Plan/Initiative Program should be undertaken with all the stakeholders involved to make the process more empowering, meaningful, and sustainable. 8. The training programs for the School Committees should be held more frequently, while the number of PTA who attend training programs should be raised. 9. Enhance the support to communities and schools by developing more systematic and stronger community facilitation and monitoring processes. Ministry of Education, Science and Technology 1) In the light of its strong impact on the national system of Primary education, the USAID QUEST Project should be continued for another three-year period 2) Consideration should be given to expanding the QUEST Project to other Districts, and especially to the rural areas 3) Further discussions with traditional leaders should be initiated to minimize interference of the annual initiation ceremonies on the school term calendar 4) A national policy on community participation should be developed and implemented 5) The integration of the QUEST TOT Program into the MOEST Teacher Certification Scheme should be explored, so as to improve teacher career planning, professional growth and the motivation to improve oneself. 6) There should be a single agreed upon action plan process for school improvement that is recognized by the District Education Office. 7) The School Cluster System and Mentor Teacher Program should be institutionalized into the national school system. Partners 1) The research study on continuous assessment should be continued in collaboration with the Improving Quality Education/Malawi (IEQ) Project 2) The partnership with CRECCOM should be promoted in order to encourage sound complementarities and to minimize redundancy. 3) The School Cluster System and the Mentor Teacher Program should be linked to other teacher training initiatives to provide peer support and responsive follow-on training. USAID 1) In the light of the serious national food crisis and its effect on the attendance of Primary school pupils, consideration of the establishment of a school feeding program should be explored 2) As a major innovation in the national school system, the School Cluster System should be further developed 35 3) The strong research activity of IEQ has greatly influenced discussion on key national education policy issues, and continued funding of its efforts should be continued Community 1) A more systematic method for monitoring and evaluating community participation is highly desirable. 2) Additional training responsive to specific community needs should be considered. 3) Greater involvement of the community in the development of teacher housing should be encouraged, e.g., the provision of building materials. 4) The role of the community in the provision of school benches/desks should be encouraged: · SC-US can assist by providing appropriate design plans · provision of materials from available local wood resources · provision of artisan skills of local resident · collaboration with School Committees to include fabrication of school benches/desks in Term Plan/Initiative proposals. 5) To foster a closer relationship between the community and the school, membership of a traditional/local leader on the School Committee should be established Teacher Training 1) The Program should be expanded be expanded to Standards 5-8 so that its pedagogical benefits can be provided to all teachers in a school. 2) The enrichment of the academic capacity of the teachers should be explored in order to provide them with knowledge of the trends and developments in their subject areas. 3) The teacher-training model utilized by QUEST should be continued and supported. 4) The teaching methods favoring the participatory approach should be maintained. 5) Informal and formal pupil performance monitoring in the classroom should be linked to research-based standards in literacy and numeracy known by teachers, pupils, and parents. 6) It is recommended that QUEST, in collaboration with MIE, support the MOEST in identifying and piloting novel approaches to strengthen literacy and numeracy instruction, approaches which have been proven successful in the region or elsewhere in similar contexts. VIII. STAKEHOLDER WORKSHOP On April 16 a workshop involving the various partners of QUEST was held at the Malawi Institute of Management. The objectives of the workshop were twofold: 1) to have presentations of the Internal Review Report of QUEST by SC/US and a draft of the Final Report of the USAID External Evaluation Team. 2) provide QUEST partners an opportunity to discuss the findings and recommendations of these two 36 reports and provide comments, suggestions, and observations and to propose recommendations for moving forward after QUEST. A number of such recommendations were developed by participants and these are presented in summary form on the following page. The recommendations presented below are organized according to the newly established USAID/Malawi priority areas. 1. Teacher Professional Skills Improved (a) The program should be expanded to Standards 5 to 8 so that its pedagogical benefits can be provided to all teachers. (b) The training model being utilized by QUEST should be continued and supported covering some of the content of the instructional materials being used in schools. (c) The participatory training methods of QUEST should be sustained and replicated. (d) The strategic supervision function of QUEST should be intensified and sustained. (e) The School Cluster System and the Mentor Teacher Program should be institutionalized into the national education system. (f) The newly recruited teachers should be given basic teaching skills and elements of child psychology by experienced trainers before they are sent to the schools. 2. Establishment of Effective Schools (a) Implementation of the national policy on community participation in school management should be encouraged. (b) The system for production and distribution and proper utilization of instructional materials should be strengthened. (c) The frequency of school committee training should be increased and the number of PTA executive members who attend the training should also be augmented. (d) Topics for training school committees, PTAs and local leaders should include content on psycho-social issues. (e) Training of District Education Managers, PEAs, headteachers and deputy headteachers on effective school management should be provided. (f) Establishment of a sustainable school feeding program that is integrated into school health and nutrition should be explored. (g) Enhance the utilization of continuous assessment results to facilitate parent involvement in pupil learning. (h) A more systematic method for monitoring and evaluation of community participation should be developed. 37 3. Policy Reform (a) As a major innovation in the national system, the School Cluster System and Mentor Teacher Program should be adopted. (b) The three-year term of office for members of School Committee be formalized and included in the Education Act, Chapter 30. (c) The Integrated Curriculum Pilot study should extended to include Standards 5 to 8. (d) Campaigns for reading in all schools should be launched and encouraged. (e) Professional standards for Primary school teaching should be established. (f) Set criteria for teacher recruitment should be established. 4. Effects of HIV/AIDS (a) HIV/AIDS, as a crosscutting issue, should be integrated into the curriculum design/review, classroom teaching, and training of teachers and community members. (b) Interventions that address orphan drop out and absenteeism should be developed and implemented. (c) Pupil, teachers and community members should be trained on the needs for behavioral change relative to HIV/AIDS prevention. The Evaluation Team would like to express its sincere appreciation to the staff members of the USAID Mission/Malawi, the SC/US, the MOEST and other stakeholders for their generous assistance in making this evaluation exercise such an enriching and stimulating experience. Evaluation Team Dr. Paul A. Hebert, Team Leader and Education Specialist Dr. Elizabeth Randolph, Evaluation Specialist Ms. Bertha Udedi, MOEST, Division of Planning 38 ANNEX A LIST OF SCHOOLS VISITED DISTRICT OF MANGOCHI (Dr. Elizabeth Randolph) Mchoka Kasolo Tukululu Sungo Bay Malenga Nikisi DISTRICT OF BALAKA (Mrs. Bertha Udedi) Ndungunde Burke School Chembera Kachere Balaka Chiyendausiku DISTRICT OF BLANTYRE RURAL (Dr. Paul Hebert) Nkhalango Chikuli Kachanga Lipunga Mdeka Lirangwe 39 ANNEX B LIST OF PERSONS CONSULTED USAID Dr. Sarah WRIGHT, Education SO Team Leader Mr. William MVALO, Cognizant Technical Officer, QUEST Ministry of Education, Science and Technology Mr. M. KALANDA, Deputy Director, Office of Basic Education Mrs. M. KABUYE, Director, Office of Methods Advisory Services Mrs. MAGALETA, Divisional Manager, South East Mrs. Grace BANDA, Divisional Planner, South West Mr. John CHAMBA, District Education Manager, Blantyre Rural Mr. NAMANGALE, District Education Manager, Balaka Mr. G.S. MAFUTA, District Education Manager, Mangochi Mr. M TRIGU, Office of Methods Advisory Services, South West Mr. Anisa KAMBWIRI, Program Education Advisor, Blantyre Ministry of Gender, Youth, and Community Services Mr. ZENENGEYA, District Community Development Officer, Balaka Mrs. BONDWE, District Community Development Officer, Blantyre Mr. N.C. KAPAPAMPHA, District Community Development Officer, Mangochi Mrs. Margaret DIMBA, Senior Community Development Officer, Blantyre Mr. KALELE, Community Development Assistant, Balaka Save the Children/US Mr. Justin OPAKU, Director, SC/US/Malawi Mr. Thomas McCORMACK, Deputy Director, SC/US/Malawi Mr. Lester NAMATHAKA, Senior Program Manager for Education Mrs. Sophie MHONI, District Education Coordinator, Blantyre Rural Mr. Andrew MALIWASA, District Education Coordidnator, Mangochi Mr. Master KALULU, District Education Coordinator, Balaka Ms. Alisa PHILIPS, Program Officer Mr. Francis MABETI, Data Officer Mr. M. IPHANI, Integrated Curriculum Coordinator 40 Malawi Institute of Education Mr. S. HAU, Director AIR/Improving Education Quality/IEQ Mr. Docks JERE, Coordinator Center for Creative Community Mobilization Mr. Simon MAWINDO Department for International Development/Malawi Mr. Keith GRISTOCK, Coordinator