A LOOK AT LEARNING IN GHANA: The Final Evaluation of USAID/Ghana’s Quality Improvement in Primary Schools (QUIPS) Program DR Prepared by: The QUIPS Program Evaluation Team: The Mitchell Group, Washington DC and the Educational Assessment and Research Centre, Accra October 2005 A Look at Learning in Ghana: The Final Evaluation of USAID/Ghana’s Quality Improvement in Primary Schools (QUIPS) Program Prepared by: The QUIPS Program Evaluation Team: The Mitchell Group, Washington DC and the Educational Assessment and Research Centre, Accra The authors’ views expressed in this evaluation do not necessarily reflect the views of the United States Agency for International Development or the Ghana Ministry of Education and Sports October 2005 FOREWORD AND ACKNOWLEDGEMENTS Larry Dolan and his team at USAID/Ghana/EDU (William Osafo, Elsie Menorkpor, Wendy Aboadi, Patrick Fosu-Siaw, Nicole Tobin, and Jennifer Koranteng) have been the champions of this study. Knowing there was a wealth of experience and data from seven years of work on improving primary education all across Ghana, they provided the opportunity for many to learn from this national effort. The QUIPS Final Evaluation was a way to understand how children's learning and project management, in Ghana and elsewhere, may be improved. Sharon Cromer, Director of USAID/Ghana, gave the study special attention; her questioning has much improved the analysis presented here. The Final Evaluation was implemented under a contract (No. AEP-I-00-00-00025-00, Order No.6) between the United States Agency for International Development and The Mitchell Group (Washington, D.C.), along with the Educational Assessment and Research Centre (EARC, Accra) and the Centre for Education Development and Education Management (CEDEM, Accra). The research team was made up of a diverse group of analysts from government, universities, and research firms as well as individual consultants from several countries. The insider-outsider diversity contributed significantly to the quality of data collection and analysis. Thanks to the core team of Elizabeth Randolph, Leslie Casley-Hayford, Rosemary O'Shaughnessy, Wes Snyder, Emma Gyamera, Lawrence Owusu Ansah, Felicia Kafui Etsey, Kenneth Asamoah, Timothy Andol, Kofi Marfo, Bert Bagbin, Felix Gyamerah, Hillary Ampoh, Augustine Kyeremeh, Charles Sagoe, Rosemond Owusu, Richard Vormawor, Emmanuel Havi. Your efforts were whole-hearted. It was inspiring how everyone collaborated. Thank you Lloyd Mitchell, Jenkins Cooper, Abby Fasosin, Lans Kumalah, Anne Grant, Winifred Asare, Maxwell Owusu Kodua, Kafui Etsey, Chris Hammond, Steve Manu, Molly Onyina Mensah, Francis Oduro, Simon Harvey, Richard Adu-Poku, Herbert Gorman, Charity Owusu, Alex Afari, Mustapha Hamidu, and Korantema Adi-Dako. The research work, however demanding, was a joy to conduct. Government, donors, schools, communities, parents, and ultimately the children themselves may find A Look at Learning in Ghana to be useful for their decisions to improve education. The findings and opinions expressed in this report are those of the authors and not necessarily those of USAID/Ghana. Wilfred Owen, Jr., Team Leader, Final QUIPS Evaluation i Comments and Acknowledgements from USAID/Ghana In an effort to ensure the accuracy and objectivity of this study, the draft was shared with the QUIPS implementing partners, including the Ghana Education Service. The feedback and comments provided offer additional valuable insights into the evaluation findings. Accordingly, they have been inserted into the evaluation at the points where they are relevant. In every case the comments are clearly identified as coming from the QUIPS partners. USAID/Ghana/EDU wishes to take this opportunity to thank the entire QUIPS Evaluation Team, the Ghana Education Service, the Ministry of Education and Sports, the participating school and community officials and the QUIPS partners in the production of this evaluation. It is our hope that the findings of this evaluation will not only benefit the young children of Ghana, but will also contribute the development of more effective basic education programs elsewhere. The USAID/Ghana Education Team ii TABLE OF CONTENTS Foreword and Acknowledgements i Comments and Acknowledgements from USAID/Ghana ii Acronyms vii List of Tables x List of Figures xii Executive Summary xiii CHAPTER 1: PURPOSE AND METHODOLOGY OF THE EVALUATION 1.1 Background 1 1.2 Purpose of the Final Evaluation 2 1.3 The Methodological Approach to the Evaluation 3 1.4 Selection of Sites for Evaluation 6 1.5 The Field Team Approach 7 1.6 Limitations of the Study 8 1.7 Context and Impact of Other Donor Activities 9 CHAPTER 2: THE CONTEXT OF LEARNING IN GHANA 2.1 Access and Participation 11 2.2 The Effect of Poverty 12 2.3 Participation of Girls in Ghana’s Education 13 2.4 The Socio-Cultural Context 14 2.5 Effectiveness of the Primary School Program 14 2.6 Ghana’s Response to Challenges 15 2.7 External Aid 16 2.8 Additional National Findings on Primary Schooling in Ghana 16 2.9 Conclusions 20 CHAPTER 3: LITERATURE REVIEW: SCHOOL EFFECTIVENESS AND QUALITY EDUCATION 3.1 International Research 21 3.2 Educational Quality in Ghana 25 3.3 Issues Prevalent in Ghanaian Schools 29 3.4 Conclusions 30 iii iv CHAPTER 4: OVERVIEW OF THE QUIPS INPUTS, RESULTS FRAMEWORK AND RELATED PERFORMANCE 4.1 Core Inputs of the Program 31 4.2 General Approach 33 4.3 Overview of SO2 Results Framework 33 4.4 Comments on the SO2 Performance Indicators 37 4.5 Performance of the Strategic Objective 38 4.6 Performance of Intermediate Results 39 CHAPTER 5: LEARNING OUTCOMES 5.1 Introduction 40 5.2 Results of the Analyses of Pupil Learning Growth 44 5.3 Analysis of Achievement after the Withdrawal of QUIPS 57 5.4 Conclusions 61 CHAPTER 6 : SCHOOL BASED OUTCOMES 6.1 Key Inputs and Intervention Strategies of QUIPS 63 6.2 Influence of QUIPS on School Enrollment 63 6.3 Teachers as Input s 65 6.4 Classroom Teaching Outcomes of QUIPS Interventions 66 6.5 Results from Qualitative Field Work 71 6.6 Infrastructure 71 6.7 Textbooks 73 6.8 Availability of Teaching Staff 76 6.9 Teacher In-Service Training 78 6.10 Teacher Practices and the Professional Commitment of Teachers 79 6.11 Head Teachers and District Staff Support to Schools 80 6.12 Teacher Performance 81 6.13 Challenges to Sustainability 84 6.14 Conclusions 85 CHAPTER 7: COMMUNITY-LEVEL OUTCOMES 7.1 Key Inputs and Intervention Strategies of QUIPS 89 7.2 Community Level Outcomes on the Community “Best Practice Instrument” 89 7.3 Key Outcomes of Community Support by QUIPS 97 7.4 Increased Parental and Community Awareness, Interest and Commitment 98 7.5 Heightened Awareness of the Role of Parents in a Child’s Education 99 7.6 Increased Enrollment of QUIPS Schools 100 7.7 Improved Community-School Relationship 101 7.8 Strengthened Leadership Capacity 102 v 7.9 Power of Community Awareness and Involvement 102 7.10 Increased Community Contributions to School Improvement and School Performance Improvement Plans 103 7.11 Community Level Challenges to Sustainability 104 7.12 Challenges, Sustainability and Conclusions 105 CHAPTER 8: OUTCOMES AT THE DISTRICT AND NATIONAL LEVEL 8.1 Outcomes and Impact at the District Level 107 8.2 District Wide Outcomes 113 8.3 National and International Perspectives on QUIPS Outcomes 120 CHAPTER 9 : KEY FACTORS WHICH AFFECT CHILDREN’S LEARNING 9.1 Introduction 124 9.2 Quantitative Results 124 9.3 Qualitative Findings 131 9.4 Conclusions 140 CHAPTER 10: LESSONS LEARNED 10.1 Child-related Lessons Learned 142 10.2 Home and Community-Related Lessons Learned 142 10.3 School-Related Lessons Learned 144 10.4 District-Related Lessons Learned 145 10.5 Institutional Lessons Learned 146 10.6 Project Management Lessons Learned 147 CHAPTER 11: KEY CONCLUSIONS AND RECOMMENDATIONS 11.1 Impact of QUIPS on Pupil Learning 150 11.2 Better Understanding the Factors Related to Pupil Learning 151 11.3 What Worked and What Did Not Work 155 11.4 The Spread and Sustainability and Spread of QUIPS Good Practices 157 11.5 Recommendations 158 vi ANNEXES: References: 161 Annex 1: Interview Schedule of National, District & Community Levels 166 Annex 2: Field Work Plan, April 2005 173 Annex 3: QUIPS and Control Schools included in Field Work 175 Annex 4: Two-Year Intervention Cycle in the Schools 176 Annex 5: Two-Year Intervention Cycle in Communities 177 Annex 6: School-Community Categorized by Teacher Performance and Community Participation 179 Annex 7: School Facilities in Sampled QUIPS and Control Schools 180 Annex 8: Teacher Profile, Sampled Schools, North, Middle and South 182 Annex 9: Independent Review of the QUIPS Achievement Tests 183 Annex 10: Scope of Work for the QUIPS Final Evaluation 194 Annex 11: Summary of the Twenty Questions Addressed in the Final Evaluation 200 vii ACRONYMS: AD Assistant Director ADB African Development Bank AED Academy for Educational Development AI Appreciative Inquiry ANCOVA Analysis of Covariance ANOVA Analysis of Variance BECE Basic Education Certificate Examination BED Basic Education Directorate, Ghana Education Service CBPAI Community Best Practices Assessment Instruments CBPI Community Best Practices Instruments Cedi Unit of Currency in Ghana: 9100 Cedis = $1.00 COI Classroom Observation Instrument CPC Community Participation Coordinator CRDD Curriculum, Research and Development Division, GES CRIQPEG Centre for Research on Improving the Quality of Primary Education in Ghana, University of Cape Coast, Ghana CRS Catholic Relief Services CRT Criterion Reference Test CS Circuit Supervisor, District Education Office CSA Community Schools Alliance CSIP Community School Improvement Plans DA District Assembly DAO District Administrative Officer DCD District Coordinating Director, District Assembly DDE District Director of Education, Ghana Education Service DEOC District Education Oversight Committee DEO District Education Office, Ghana Education Service DEPT District Education Planning Team, Ghana Education Service DEST District Education Support Team DfID Department for International Development (British International Development Fund) DGM District Grant Mechanism DMIT District Management Implementation Team DSA Daily Subsistence Allowance DTST District Teacher Support Team EARC Educational Assessment & Research Centre EMIS Education Management Information Systems ESL English-as-a-Second-Language ESP Education Strategic Plan fCUBE Free Compulsory Universal Basic Education GES Ghana Education Service GETFund Ghana Education Trust Fund GLSS Ghana Living Standards Survey viii GNAT Ghana National Association of Teachers GoG Government of Ghana GPRS Ghana Poverty Reduction Strategy GSSS Ghana Social Standards Survey GTZ German Development Agency HLM Hierarchical Linear Modeling IFESH International Foundation for Education and Self-Help ILP Improving Learning through Partnerships INSET In-service Training IP Implementing Partner IR Intermediate Result KG Kindergarten JICA Japanese International Cooperation Agency JSS Junior Secondary School M&E Monitoring and Evaluation MOE Ministry of Education MOEYS Ministry of Education Youth and Sports MOES Ministry of Education and Sports MTEF Medium Term Expenditure Framework NPA Non-project Assistance P Primary, Equivalent to Grade PA Project Assistance PL480 Public Law 480 or the Food for Peace Program PBME Planning Budgeting and Monitoring and Evaluation Department (MOE unit) PLA Participatory Learning Appraisal PME Performance Monitoring and Evaluation PMT Performance Monitoring Tests PRA Participatory Rural Appraisal PREP Primary Education Program PTA Parent Teacher Association QUIPS Quality Improvement in Primary Schools R4 Results Report and Resource Request SCHID School Identification Number SCMI Supervisory and Classroom Management Instrument SD Standard Deviation SIP School Improvement Plans SMC School Management Committee SO Strategic Objective SOW Scope of Work SPIP School Performance Improvement Plan TED Teacher Education Division, Ghana Education Service TLM’s Teaching-learning Materials TMG The Mitchell Group TOT Training of Trainers ix TTC Teacher Training College UCC University of Cape Coast UCEW University College of Education at Winneba USAID United States Agency for International Development WSD Whole School Development x LIST OF TABLES: Table 1.1 School Sampling: QUIPS and Control Schools 7 Table 2.1 Pupil Demographics across Regions 17 Table 2.2 Pupil Demographics by Rural and Urban Locations 17 Table 2.3 Availability of Textbooks and Children’s Books at Home by Region 18 Table 2.4 Availability of Textbooks and Children’s Books at Home by Urban and Rural Locations 18 Table 2.5 Home Literacy and Exposure to English in the Home by Region 18 Table 2.6 Home Literacy and Exposure to English in the Home by Urban and Rural Locations 18 Table 2.7 English Spoken to Children at Home by Region 19 Table 5.1 Baseline School Mean Scores for QUIPS and Control Schools 43 Table 5.2 Age of Pupils at Baseline: QUIPS and Control Schools 43 Table 5.3 Sex of Pupils at Baseline: QUIPS and Control Schools 43 Table 5.4 Pupils’ Exposure to English: QUIPS and Control Schools 44 Table 5.5 Home Literacy: QUIPS and Control Schools 44 Table 5.6 Slope Coefficients for the QUIPS and Control Schools 45 Table 5.7 Summary of Grade 3 Class Means over Time: QUIPS and Control Schools 46 Table 5.8 Summary of Grade 5 Class Means over Time: QUIPS and Control Schools 48 Table 5.9 Summary of Performance Level and Skill Clusters for Mathematics 52 Table 5.10 Summary of Performance Level and Skill Clusters for English Reading 53 Table 5.11 Aggregated Slope Coefficients across Cohorts: QUIPS and Control Schools 54 Table 6.1 Enrollments in QUIPS and Control Schools by Region 64 Table 6.2 Enrollments in QUIPS and Control Schools by Location 64 Table 6.3 Teacher Staffing across Regional Divisions 65 Table 6.4 Teacher Staffing in QUIPS and Control Schools: Urban and Rural Settings 65 Table 6.5 Distribution of Pupil Learning Groups across Regions 69 Table 6.6 For the North, “Encouraging Girls to Participate” by Learning Group 69 Table 6.7 For the South and Middle Belt, “Encouraging Girls to Participate” by Learning Group 69 Table 6.8 QUIPS Infrastructure Completion Level 74 Table 6.9 English and Mathematics Textbooks in Sampled Schools 74 Table 6.10 Range of Mathematics and English Exercises Completed By Pupils in their Exercise Books 82 Table 7.1 Shift in Performance on the Community Best Practice Assessment Instrument from Baseline to QUIPS Termination 91 xi Table 7.2 Trust in the School System 93 Table 7.3 Support for Quality Education 94 Table 7.4 Empower Local People to Act 95 Table 7.5 Strengthen School Management 95 Table 7.6 Use Participatory Planning 96 Table 7.7 Outcomes in Relation to Each Community Involved in the Site Visits 97 Table 8.1 District Sustainability and Spread of QUIPS Activities 117 Table 9.1 Pupil Factors Related to Achievement 126 Table 9.2 School Factors Related to Achievement 126 Table 9.3 Classroom Factors Associated with Pupil Learning 129 Table 9.4 Community Best Practices Associated with Primary School Learning 130 Table 9.5 Child Factors that Influence Learning 131 Table 9.6 Home and Community Factors that Influence Child Learning 133 Table 9.7 Teacher and School Factors that Influence Child Learning 135 Table 9.8 GES: Factors that Influence Child Learning 138 xii LIST OF FIGURES: Figure 1.1 Nested Systems at Play in School Reform 4 Figure 2.1 Net Enrollments for Primary Schools 12 Figure 2.2 Poverty Incidence and Attendance Rates by Region 13 Figure 2.3 Participation Rates by Gender and Educational Level 14 Figure 5.1 Baseline School Mean Scores: QUIPS and Control Schools 43 Figure 5.2 Learning Growth for QUIPS and Control Schools 45 Figure 5.3 Means Plots: Math from Grade 3 to Grade 4 47 Figure 5.4 Means Plots: English Reading from Grade 3 to Grade 4 47 Figure 5.5 Means Plots: Spoken English Narrative from Grade 3 to Grade 4 47 Figure 5.6 Means Plots: Math from Grade 5 to Grade 6 48 Figure 5.7 Means Plots: English Reading Grade 5 to Grade 6 49 Figure 5.8 Means Plots: Spoken English Narrative Grade 5 to Grade 6 49 Figure 5.9 Differences in Final Achievement Results: QUIPS and Control Schools 50 Figure 5.10 Hypothetical Group of Grade 1 Pupils, Predicted Means Plots for Six Years of Primary School 51 Figure 5.11 Gap in Pupil Learning for Grade 3 Mathematics by Cohort 54 Figure 5.12 Gap in Pupil Learning for Grade 5 Mathematics by Cohort 55 Figure 5.13 Gap in Pupil Learning for Grade 3 English Reading by Cohort 55 Figure 5.14 Gap in Pupil Learning for Grade 5 English Reading by Cohort 55 Figure 5.15 Gap in Pupil Learning for Grade 3 Spoken English Narrative by Cohort 56 Figure 5.16 Gap in Pupil Learning for Grade 5 Spoken English Narrative by Cohort 56 Figure 5.17 Adjusted Class Means for Grade 3 English from Mid-cycle Through the Second Out-year 58 Figure 5.18 Adjusted Class Means for Grade 3 Mathematics from Mid-cycle through the Second Out-year 58 Figure 5.19 Adjusted Class Means for Grade 5 English from Mid-cycle Through the Second Out-year 59 Figure 5.20 Adjusted Class Means for Grade 5 Mathematics from Mid-cycle through the Second Out-year 59 Figure 6.1 For the North, “Encouraging Girls to Participate” 70 Figure 6.2 For the South and Middle Belt, “Encouraging Girls to Participate” 70 Figure 7.1 Trust in the School System 94 Figure 7.2 Support for Quality Education 94 Figure 7.3 Empower Local People to Act 95 Figure 7.4 Strengthen School Management 96 Figure 7.5 Use Participatory Planning 96 EXECUTIVE SUMMARY A Look at Learning in Ghana: The Final Evaluation of USAID/Ghana’s Quality Improvement in Primary Schools (QUIPS) Program was a departure from usual practice. This evaluation is somewhat unusual in that USAID is going back to stakeholders well after the end of an intervention to learn from the people concerned what benefits have accrued to them and what they have been able to sustain: what worked, what did not work, and what else should be considered as USAID moves forward in its commitment to support quality primary education in Ghana. In addition to this final technical report, a reader-friendly booklet called “Helping Children Learn in Ghana: Lessons Learned from QUIPS” has been prepared, describing project factors that help children learn, to give feedback to stakeholders in communities, schools, and districts. The booklet not only shows people they have been heard, it also pulls together the major findings about factors that affect children’s learning in Ghana. It offers ideas and reminders to help children, parents, teachers and managers maintain the good practices they experienced during the course of QUIPS. THE PROCESS Listening to the voices of children, the ultimate and most important beneficiaries, was an innovative aspect of the evaluation. It was clear that primary school children in Ghana want to “be somebody” and have a good idea of what they need to reach that goal: good teachers, enough books, and enough food. Their voices, along with those of their parents, traditional and elected community leaders, members of School Management Committees (SMCs) and Parent Teacher Associations (PTAs), teachers, head teachers, and district officials all informed the final evaluation. The grassroots information was complemented by the views of former directors of the implementing agencies and officials of the Ghana Education Service (GES). With the factors that affect child learning as a guiding framework, respondents engaged in frank discussions about their experiences with QUIPS and their hopes and fears for primary education in Ghana in the future. In addition to this qualitative information, the research team had access to an integrated database collected during the course of the QUIPS project, especially from post￾intervention learning assessments. The exceptionally rich database included facts on pupil achievement, teacher instructional practices, and community participation in education. The longitudinal nature of these data was unique: Children in every QUIPS school and matched control school across the country participated in pre- and post￾testing in mathematics, spoken English, and English literacy. xiii xiv In tracking the learning growth of individual children, exciting data emerged on changes in learning from baseline to post-intervention. The qualitative and quantitative information together enabled the research team to investigate the results and challenges of QUIPS, identify lessons learned, and offer suggestions for how to increase the effectiveness of future interventions. MAJOR FINDINGS Major findings of the Final Evaluation of the QUIPS Program follow. The findings are presented in more detail in Chapters 5-9. The findings are brought together in summary form (along with recommendations) in Chapters 10-11. • During each two-year intervention cycle, the QUIPS program achieved the planned results identified by the SO2 Results Framework for each cohort. By the second year of the QUIPS intervention, pupils in QUIPS schools were able to read with meaning more than control schools. Further, as a consequence of the perceived reputation of the QUIPS schools as being “good,” enrollments, particularly of girls, were increased in these schools. Nonetheless, in many schools these gains have not been sustained, in large part because the basic conditions that existed during QUIPS, especially staffing and supervision, have not been maintained. • QUIPS-fostered community involvement in school management has had a lasting impact on the quality of education and resultant learning. Strengthening the role of SMCs/PTAs in primary education was the broadest QUIPS intervention, and because communities are far more stable than teachers or district personnel, the impact is more likely to last. QUIPS also strengthened the internal leadership structures in communities and encouraged communities to contribute and plan towards school improvement. The community focused interventions had a positive correlation with high performing schools and enhanced learning outcomes.1 1 In its review, the Academy of Educational Development stated commented: “The statement that “QUIPS fostered community involvement in school management has had a lasting impact on quality of education and resultant learning” is quite definitive, and as stated seems to suggest that this aspect of the QUIPS intervention package alone has had this distinctive and lasting impact on resultant learning. Certainly there is a strong correlation in between community involvement and high performance (as noted on pages 92-93), but [we] do not recall a separate statistical analysis that specifically assesses the unique contribution of community involvement as a separate, distinctive, contributing factor. It could be argued, for example, that the interaction effect of both community involvement and high performing teachers/ head teachers in combination were the key to differentiating low and high performing schools. Thus, some caution might be exercised in drawing the conclusion stated in this bullet, and this caution is in fact recognized on pages 202-203 where it is noted that: “it would be unfair and inaccurate to point to any one or more approaches being more responsible for increase in learning. Rather, it appears to be the result of a combined affect. […] A separate bullet might be proposed that points to a combination of good instructional practices, good school management and community involvement as providing a strong, consistent explanation, or at least demonstration of a strong correlation with higher pupil achievement.” xv • Building national awareness about the responsibilities of communities to support schooling was one of the major accomplishments of QUIPS. The importance of empowered parents and local authorities working with school personnel to improve instructional quality was emphasized by stakeholders throughout the evaluative fieldwork. Mainstreaming of SMC/PTA training across the nation has built awareness of the need to involve communities and share responsibility for primary education. • While there was no direct evidence that QUIPS-financed infrastructure improvements were tied to school performance and pupil learning, QUIPS infrastructure was successful in increasing communities’ sense of ownership of their schools by involving them in the construction process. Teachers’ work conditions in the QUIPS schools improved, schools stayed open during the rainy season and positive relationships were developed between schools and communities. Findings from the evaluation suggest that infrastructure programs should first define their goals. If the main goal is to improve pupil learning, an infrastructure project might consider building teacher housing to encourage retention, keep teachers punctual, and build their commitment and sense of accountability to the communities where they teach. On the other hand, if the goal is to facilitate community sense of ownership in the schools, the building of classroom blocks and acquisition of furniture may be more useful. In both cases, the responsibility for part of the construction or procurement process needs to be shared with the community leadership. • The QUIPS Program achieved an immediate return on training in that teachers were responsive and implemented most of the reforms in QUIPS classrooms. Specific targeted QUIPS teacher training and support activities were associated with high pupil learning during the two-year intervention cycle, particularly encouragement of pupils, especially girls, to participate; teacher questioning and feedback to pupils; and facilitation of pupil interaction and creative thinking. However, there is little evidence that these classroom reforms were sustained. The evaluation identified a number of factors impeding sustainability; the most serious was teacher mobility. Given the diffuse distribution of QUIPS schools across the nation, and an in-service training program targeting teachers in these schools, there was insufficient support for teachers to carry on the new practices over the long run. • The positive impact of effective head teachers on learning was clear in high-performing schools, whether the school received QUIPS assistance or whether it was a “control” school. Enhancing school leadership pays off by increasing school effectiveness. Further, the scaling up and spread of QUIPS influenced the practices of head teachers across districts observed by the evaluation teams and revealed that QUIPS was successful in spreading good xvi practices in non-QUIPS schools. However, better head teachers must have better material support if they are to do effective training in their schools on their own. • Education reforms targeting improvements in teacher instructional practices that fail to address systemic issues related to teaching (recruitment, training, remuneration, conditions of service) cannot be sustained. The QUIPS assumptions that desire for professional growth, improvement in the performance of duties, or commitment to the teaching profession would be sufficient to motivate teachers were misplaced. • Under QUIPS, district support and attention to school demands and supervision was found to be a major contributor to a school's success in teaching. However, the amount of district support provided to the QUIPS schools served to marginalize other schools in the district. Further, after the QUIPS intervention ended, district supervision of the QUIPS schools deteriorated. • The relevance in the learning context is supremely important. The team’s studies of what pupils were and were not able to do showed that mathematics story problems using common experiences of children were completed at higher grade levels (even to a smaller extent, problems at the same grade level of the pupil) than basic mathematics operations. • The M&E training provided through the QUIPS district grants led to a shift in thinking about district responsibility and accountability. The skills developed in collecting and, most important, using data to inform management decisions are remarkable, although their sustainability is challenged by shifting district priorities and funding limitations. • The training provided by QUIPS for DEO personnel enhanced their capacity to operate more effectively in management, supervision, planning, and M&E. Including district officers in training activities for teachers and SMC/PTA executives gave them learning opportunities and modeled effective ways to operate. Many of the activities that the district officers undertook as part of the QUIPS experience are no longer being done because funds to support them are not yet forthcoming. • Pupils can not be expected to make appreciable gains within a two-year intervention period in the absence of accelerated programs in literacy and numeracy. Pupils throughout Ghana were performing far below development expectations in English reading and mathematics. Two years of QUIPS interventions was simply not enough time to remedy this deficiency. xvii • Finally, the evaluation teams found that many children started the school day hungry and tired, thus undermining their ability to learn. All the groups with whom the evaluation teams interacted spoke of how children’s readiness to learn was affected by hunger. In QUIPS low-performing communities, parents often were neglecting the basic needs of their children. Interviews with children revealed that the vast majority of parents were not ensuring that their children’s basic food needs were met before and during school. Findings from the field suggested that USAID's school feeding program (PL480) was an excellent complement to the QUIPS program because it ensured that most children were able to obtain a basic meal that gave them enough energy to learn effectively, thereby ensuring that QUIPS interventions were supported. THE CONTENT OF THE REPORT QUIPS was designed to demonstrate the conditions required for effective and sustainable primary education in model “partnership schools” across Ghana. Taking an integrated approach, the program tried to act simultaneously on both national policy reform and school and community development. Chapter 1 introduces the QUIPS Program and the goals and methods of the evaluation. Chapter 2 describes the context of learning in Ghana. It outlines the diversity of schooling throughout the country, highlighting the well-known disparity between northern and southern Ghana and between rural and urban communities. School inputs—teachers, textbooks, and classrooms—are more readily available in southern and in urban schools than in northern Ghana and in rural communities. Still, pupils throughout Ghana have limited exposure to printed material; in some regions as few as 8% of the children reported having a textbook or any other reading material at home. Chapter 3 reviews international and Ghanaian literature on school effectiveness and the factors that promote learning by children, particularly in remote and deprived areas. It highlights five core dimensions related to quality schooling: learner characteristics and readiness to learn; social context; education inputs (teachers, textbooks); the teaching and learning process; and monitoring of the outcomes of efforts to educate. The challenges reported in studies from the 1990s still exist, including those related to policy on the language of instruction, limitations in literacy development, reading difficulties among children, pedagogy for teaching English as a second language, the limited availability of textbooks, and low teacher time on task. Finally, the chapter reports the growing trend of parents enrolling their children in private primary schools. The QUIPS Program was a complex and ambitious undertaking. Chapter 4 summarizes inputs and strategies and reviews the SO2 results framework. QUIPS used $51.8 million of project assistance (PA) and $6 million of non-project assistance (NPA) over a seven￾year period. During the two-year intervention cycle, QUIPS trained teaching staff and community members in 367 school-communities across Ghana as well as the executives of all the SMC/PTAs in the country. QUIPS also delivered management xviii support to education officials in all 110 districts in the country and to officials from the Ministry of Education and Sports and from the GES headquarters. QUIPS NPA was used to promote formulation of education policy and systems in support of quality education, including improved personnel management, national pupil assessment, and reinforcement for decentralizing education management. At the midterm review, a large proportion of the NPA was reallocated to new initiatives to promote the spread of QUIPS to more schools in each district. Chapters 5 to 8 describe QUIPS outcomes in detail and discuss the sustainability of each. Project outcomes are described at several levels, starting with children’s learning and moving on to school-based, community, district, and national outcomes. Learning outcomes were based on pre- and post-test results, which established that children in QUIPS schools had an advantage over their peers in terms of the academic benefits that accrued not only during the course of the intervention but also up to two years later. Chapter 6 describes school-based outcomes. It analyzes infrastructure, textbooks, supplementary readers, availability of teachers, teacher in-service training, teacher instructional practices, and head teacher and District Education Office (DEO) support. These factors are discussed in terms of the northern, southern, and middle sections of the country, QUIPS and non-QUIPS schools, high- and low-performing schools, and rural and urban schools. Chapter 7 summarizes the community outcomes of the Program, among them increased parent and community commitment to the process of education and schooling; strengthened relationships between teachers and community; greater involvement of community leaders in improving the school through SMC/PTAs and traditional leadership structures; and improved educational outcomes when the community supports the school through enrollment drives, community contributions, food support, books, volunteer teacher support, and maintenance of school buildings. Chapter 8 presents the district and national outcomes. Funds provided by QUIPS enabled district education officers to do stand-up training of teachers and community members. The availability and regular flow of resources and technical assistance to the District Education Office was a relief to often severely under-resourced locations, but did create imbalances within districts. The District Grant Mechanism (DGM) and the QUIPS interventions required the officers to give more resources and investment to a very few schools, leaving little attention and support available for the majority of schools in a district. All districts visited felt the impact of QUIPS: Its management training was seen as relevant to daily work and the needs of the district officers. A gradual conversion from the hierarchical to a horizontal pattern of management helped the district offices introduce more participatory approaches to their work. The ability of the districts to xix monitor and evaluate their educational programs was underscored as one of the more positive outcomes of QUIPS. The critical question addressed by the final QUIPS evaluation is the extent to which program interventions affected children’s learning. Chapter 9 reviews factors identified by stakeholders that both enable and constrain learning. All the groups with whom the evaluation teams interacted had clear ideas about what promoted and diminished effective learning. They spoke of how children’s readiness to learn was affected by hunger and by support and encouragement from parents and the community, how well teachers and schools provided opportunities for them to learn, and how GES at the district and national (HQ) levels support school and community in delivering quality education to the children. The opinions of stakeholders are supported by the statistical information, which described among other things factors like teacher feedback and encouragement, especially for girls. Chapter 10 looks at what was learned from the project. Although most of the lessons are not new, they are forceful reminders about what is important and about what does or does not work and why. They are also reminders of how complex the issue of effective schooling is, especially in a world of limited resources and interdependence of critical components. While an attempt was made to isolate outstanding lessons, it is clear that none stood alone. Because children across Ghana have high expectations, helping them realize their dreams must be the very basis of all school improvement efforts. It was clear also that once parents understood the value of education, they became powerful advocates for quality schools, particularly if their community had harnessed the collective energies of its members to influence how the school operates. Children in QUIPS schools demonstrated higher learning growth than their peers in the control schools. In schools where teachers sustained the effective practices they developed from QUIPS training, pupils continued to learn effectively. However, in far too many schools, this was not the case. When staffing was reduced to the pre-QUIPS level, QUIPS-trained teachers left the schools and active supervision diminished. So did school performance. But it was clear also that the continuing GES supervision and support necessary to maintain gains was beyond what was possible, given Ghana’s current education management and funding. There also emerged lessons about project design and implementation and the relationships between implementing partners. (See Chapter 10.) Modifications to the original design during the negotiations between USAID and the Ministry of Education diminished the chances of success. The drain on district resources required to implement QUIPS in a small number of schools negatively affected other schools. Moreover, there were clear differences in understanding between implementers and the GES on a number of dimensions. Setting requirements for reforms at the school level xx that could not be maintained given the country’s means threatened sustainability from the start. The final chapter of A Look at Learning in Ghana brings together conclusions and recommendations about the impact and sustainability of QUIPS. The chapter also summarizes the detailed process of how children all over Ghana learn. With an improved understanding of the factors contributing to learning, the recommendations the report makes are expected to contribute directly to improved pupil performance and thus help more children to “be somebody” in the future. CHAPTER 1: PURPOSE AND METHODOLOGY OF THE EVALUATION The United States Agency for International Development (USAID) helped the Ministry of Education (MOE)2 and the Ghana Education Service (GES) to implement the Free Compulsory Universal Basic Education (fCUBE) Program through the USAID-funded Quality Improvement in Primary Schools (QUIPS) program. QUIPS had two implementing partners, CSA and ILP, in the seven southern regions of the country and one, CRS, in the three northern regions: • Community School Alliances (CSA) worked to increase community awareness and responsibility and advocacy for education, to strengthen School Management Committees (SMCs) and Parent Teacher Associations (PTAs), and to enhance community participation in the design, implementation, and monitoring of school improvement efforts. • Improving Learning through Partnerships (ILP) worked directly with district and school personnel to help improve teaching and learning practices in primary schools (Grades 1–6) and to strengthen school-community relations and management support to schools by the District Education Offices (DEOs) and the District Assemblies. • Catholic Relief Services (CRS) supported both local communities and schools in the three northern regions. It provided training and technical assistance similar to what was being delivered by CSA and ILP in the south. USAID used the best practices emerging from QUIPS interventions to engage in policy dialogue with the Ministry and GES to support formulation of policy that would support quality education in a decentralized environment. 1.1 BACKGROUND From 1990 to 1997 USAID/Ghana worked with MOE through its Primary Education Program (PREP) to support the nation in its efforts to rebuild an education system that had deteriorated substantially in the 1980s. The agenda of PREP, which was designed to reinvigorate primary education programs across the country, was essentially to improve the supply and quality of education inputs, including textbooks, teacher training, and national assessment. 1 2 The name of the Ministry of Education (MOE) was changed to the Ministry of Education Youth and Sports (MOEYS) in 2003. It was changed again in 2005 to the Ministry of Education and Sports (MOES). The Youth Ministry has been merged with the Manpower Ministry and is now the Ministry of Youth, Manpower and Development (MYMD). MOE will be used in this report to denote not only the original ministry but also its successors. 2 In 1997 USAID/Ghana began a multilevel program of assistance to help improve schooling effectiveness. Like PREP, USAID’s Basic Education Strategy from 1997 to 2004 was to support the Government of Ghana’s continuing reform, known as Free Compulsory Universal Basic Education (fCUBE). The essence of the 1997–2004 Basic Education Strategy was to demonstrate the conditions required for effective and sustainable primary education and to replicate the successful elements in a national Model Schools Program that came to be known as Quality Improvement in Primary Schools (QUIPS). The QUIPS model dealt simultaneously with policy reform and with school and community development, emphasizing the following objectives: (1) improving the quality of teaching and learning; (2) building capacity for decentralized school management; (3) increasing community involvement in schools; and (4) improving the physical learning environment. Project assistance (PA) in the amount of US$51.8 million provided extensive technical assistance and training for teaching staff and community members in 367 school communities (three in each of 110 districts) with a two-year cycle of intervention for each school-community. The 367 schools were grouped into six cohorts that entered the QUIPS program each year for a 24-month period of service. Management technical assistance and training were provided to district education officials in all 110 districts, to officials from the Planning, Budgeting, and Monitoring and Evaluation (PBME) Department of the MOE and to GES headquarters officials. Non-project assistance (NPA) in the amount of US $6 million was given to promote education policy reform and systems for quality education, including improved personnel management, national pupil assessment, and policy to reinforce decentralization of education management. 1.2 PURPOSE OF THE FINAL EVALUATION The purpose of the QUIPS final evaluation was to: • Assess the impact of the QUIPS Program on pupil learning and investigate factors that explain: (1) differences between high- and low-performing schools, districts, and regions; (2) relatively modest learning achievement gains by pupils in QUIPS schools as compared to matched controls, especially in English literacy; and (3) the narrowing of the performance gap between QUIPS and control schools during the last three years of the program. • Identify factors (e.g., teaching methods, community practices) associated with improved pupil learning and challenges that need to be addressed in programs to improve pupil achievement. • Inform USAID and other implementing partners about what worked and did not work under the QUIPS program. • Establish whether approaches QUIPS used are being maintained at the school, community, and district levels. 3 A set of twenty questions identified in the Scope of Work (SOW) served as a framework for the evaluation. Annex 11 summarizes the findings for each question. The questions are particularly relevant to the new USAID education Strategic Objective SO8, which seeks to improve pupil achievement. Lessons learned from what did and did not work in QUIPS will inform current and future primary education programs to enhance their probability of success. Besides addressing the performance of QUIPS interventions, USAID asked the evaluation team to investigate factors that contribute to children’s learning in Ghana. Thus the inquiry moved from programs and treatments to the ultimate client, the child. The evaluation started in March 2005, six months after the final QUIPS program interventions drew to a close. 1.3 THE METHODOLOGICAL APPROACH TO THE EVALUATION Guiding Principles. Sustainable education reform requires that change take place within three related systems in school learning: classroom, school, and community. (See Figure 1.1.3) In this framework, the classrooms are made up of teachers, pupils, and parents. Even though parents are not present daily in the classroom, their influence on classroom performance is ever present, both directly and indirectly through their participation (or lack thereof) in their child’s learning. Communities that recognize the benefit of high quality schooling, not only for children but for the community itself, will do much for their schools, but usually it takes a deliberate effort to sensitize communities to the intimate relationship between quality schools and quality communities and to empower members to become active in school improvement. Lastly, unless schools and school systems endorse and support instructional change in the classroom, the long￾term sustainability of new approaches is unlikely. The QUIPS program was aligned with this framework; it sought to reach its goal of increased effectiveness of the primary education system through (1) training teachers and school managers; (2) informing district, regional, and central administration about good practices and encouraging national policy to support such initiatives; and (3) mobilizing and building the capacity of communities to participate productively in school development and governance. By considering the interdependence of these three nested systems for school learning, the evaluation team was better able to understand the interrelationships among QUIPS activities in the context of the goals of the program and pupil learning. We avoided looking at issues as isolated events perceived as road-blocks to learning. Instead, we attempted to understand the conditions that may lead to such constraints and considered recommendations for improving the situation in the context of classroom, school system, and community. 3 Peter Senge, et al. (2000) Schools that Learn. (New York: Doubleday), pp. 11-19. Figure 1.1: Nested Systems at Play in School Reform The team took special care to take into account the diverse contexts in Ghana where learning takes place—different regions, cultures, households, access to schools, and so on—and the resulting differences in how they operate within and across these connected systems. This placed us in a better position to consider alternative approaches to promoting the positive interplay of classrooms, school systems, and communities across all the learning contexts in Ghana. 1.3.2 Specific Methodologies. The evaluation was a countrywide assessment of QUIPS. Qualitative data were collected in a sampling of 16 representative districts as well as at the national level. Quantitative data from project M&E4 and results of the post-intervention pupil testing provided a second database and permitted a more comprehensive analysis than would have been the case with only one type of data. Structured interviews with a wide range of stakeholders solicited information about factors that support, enhance, and consolidate child learning. The conceptual framework for exploring the elements of the QUIPS model was rooted in international best practices, which suggest that factors related to child learning can be broken down into the following categories:5 (1) school or teaching-related factors; (2) home-related factors; (3) community-related factors; and (4) systemic factors outside the direct control of the school and community. This study made explicit a fifth category of factors influencing child learning that the team considered critical: child-related factors. 4 4 The QUIPS integrated database, archived at USAID/Ghana/EDU, consists of raw data from all implementing partners since the beginning of the project plus data from such national sources as the Criterion Reference Test (CRT) and Primary School Census data from the Ghana Education Service. The data were used not only for annual updating of R4 performance data tables, but also for special studies, such as this final evaluation. 5 Heneveld and Craig, 1995. 5 The evaluation took the following four complementary approaches: • Pupil achievement was analyzed in relation to quantified measures of QUIPS inputs and other factors associated with learning by identifying high- and low￾performing schools with respect to pupils’ learning and by studying potential predictors of learning. Quantitative analyses were based on an extensive pool of data collected and archived in the QUIPS integrated database throughout the program. In addition to descriptive techniques, a multilevel modeling procedure (HLM: “hierarchical linear modeling”) and analyses of covariance were used in this analysis. • Content of documents generated by the program was analyzed in terms of three questions: (1) In the context of the international best practice literature, what does the QUIPS model tell us? (2) Where did the QUIPS model succeed or not succeed in promoting pupil learning? and (3) What core factors contributed to the success or failure of the QUIPS model? • Qualitative investigation of trends emerging from the quantitative data (pupils' achievement on tests, pupils’ learning, changes in teacher and community behaviors) was achieved by visits to schools and communities in three different areas of Ghana. The findings allowed the field teams to probe interrelationships and validate trends identified by the quantitative analysis and document review. • A micro-study ensured that the rich data sources linking contextual and demographic data with child learning performance were thoroughly analyzed to elicit potential trends. Each of the four field teams used a team-generated field guide to manage focus group discussions with school, community, and district stakeholders and implementers: • Children who were in school during QUIPS. • Stakeholders in the schools: head teachers and teachers. • Community stakeholders: Members of the PTA and SMC, chiefs, pastors, and women's groups. • District stakeholders: DEOs, Circuit Supervisors (CSs), Assistant Directors and members of the District Teacher Support Teams (DTSTs), and District Management Implementation Teams (DMITs). • District Assembly Officers, District Coordinating Directors (DCDs), and Chairmen of District Education Oversight Committees (DEOCs). Each four-member field team used other techniques to help identify factors related to pupils’ learning: 6 • Classroom observation to identify key teaching factors and verify the quantitative information emerging from the QUIPS Classroom Observation Instrument (COI). • Review of pupil exercise books to evaluate both child and teacher output and performance. • Home visits to pupils and parents to confirm through observation the information received from the Grade 6 and Grade 4 pupil focus groups. 1.4 SELECTION OF SITES FOR EVALUATION The final selection of schools for the field study was based on a team decision to sample schools and communities in the same area that represent: (1) high and low pupil learning;6 (2) QUIPS and control schools; and (3) urban and rural locations.7 The evaluation team selected schools from Cohorts 4 and 6 to provide contrasting information about the differential impact before and after shifts in treatment made at mid-term in 2001.8 (See Table 1.1.) The team chose 16 schools in eight districts for the field studies.9 Three of the 16 were urban. Four schools were selected in each district: high-performance QUIPS, low QUIPS, high-performance control and low control. Four research teams were sent to the field: two to the north, one to the middle belt, and one to the south. Each team studied two school-communities in two districts over an eleven-day period. Eighteen different interview protocols and observation schedules guided the collection of information from pupils, teachers, head teachers, SMC/PTA members, community leaders, chiefs, women’s groups, pastors, NGO workers, and government officials. Two "low-learning" urban QUIPS schools were added to the base sample in an effort to explore the influence of urban location on learning factors. When the evaluation team reached the second urban school, they found a remote community of less than a thousand people. The community had been mislabeled as urban. 6 High-performing schools were those where pupil learning fell in the upper 33% group in three of four subject areas: Grade 3 and Grade 4 math, Grade 3 and Grade 4 English reading, Grade 5 and Grade 6 math, Grade 5 and Grade 6 English reading. 7 Urban was defined as village populations of more than 5,000 people and district capitals, even where the populations were fewer than 5,000. 8 Cohort 4 received a QUIPS treatment from Sept 2000 to July 2002 that was conducted exclusively by consultants from the implementing partners. At that time there was a concentration of effort at the school and community. In contrast, Cohort 6 entered the QUIPS program from July 2002 to Sept 2004 when the implementing partners were working increasingly through the District Education Office and district level training and the grant mechanism competed for the attention of program personnel. At the same time a portion of the in-service training at schools was delivered by mixed teams of consultants and district officers or by the district officers alone. Consequently, some complained that QUIPS for Cohort 6 was a watered down version of the earlier QUIPS efforts. 9 Two additional communities were added in the urban setting in order to compare more fully the rural environments. In total, the evaluation teams visited 18 communities. 7 Table 1.1: School Sampling: QUIPS and Control Schools Across Southern, Middle and Northern Ghana SCHOOLS South Middle North HIGH LOW HIGH LOW HIGH LOW QUIPS Watreso Dunkwa Presby Kwaku Pamfo Ntiribuoho (urban) Adukrom Tuobodum Nuriya Islamic (urban) Lonto Presby Kpanlori Nakpale Kworle Nawuni CONTROL Nyame￾bekyere Achiase Ntonso SDA (urban) Kunsu Datiem Ekumpe Gbangu￾Bangbini Tusundo Kpalgun Zion 1.5 THE FIELD TEAM APPROACH The field teams were made up of educators and researchers who had not been involved in the QUIPS program. Each team had three or four members—a number considered ideal to develop optimum rapport with the community. Teams in the north used translators, most of who were from the target communities. Each team spent one day talking to district officials and two days in each school and community,10 so as to better understand the dynamics of relationships among pupils, school, community, and district factors that contributed to pupil learning. In two cases, the field teams returned to the community for a third day to explore in greater depth why a school in a remote area school was high-performing. In one case a team visited a community at night to learn about pupil-managed study groups operating in each compound house. Interviews at district and regional capitals required more time. Given constraints on group interviews in offices, several teams organized informal, out-of-office sessions with district officers on the weekends. The teams made an effort to vary the context in which information was gathered so as to improve the validity of the information they collected. 10 See Annex 1, Interview Schedule of National, District and Community Levels. 8 In a number of cases, the evaluation teams returned to district and regional offices to debrief officials on initial findings from the visits to their schools and communities. Some officers asked that copies of the final evaluation report be mailed to them directly. 1.6 LIMITATIONS OF THE STUDY There is considerable movement of teachers and district personnel within the GES. Only 17 teachers with QUIPS training were still teaching in the 18 schools that were visited, compared to a total of 88 teaching in those schools during QUIPS treatment—a retention rate of 19%.11 The number of QUIPS-trained teachers remaining at QUIPS schools ranged from 0 to 5. Of the ten QUIPS schools visited, three continued to have QUIPS-trained head teachers; these schools were relatively well managed.12 There was also high staff turnover among education officers at district headquarters due to transfer and retirement, and a number of the officers who had been involved directly with QUIPS were no longer there. Moreover, the work schedule of officers often required them to be out of the office, which limited their availability for interviewing. The evaluation teams arrived in the districts on Tuesday, April 5, 2005, the first day of the third term of the academic year. Some schools were not prepared to have classes the first week of the term and in several cases not all teachers were present. Untrained teachers in the northern region were attending a GES training course. These facts reduced the number of teachers and pupils available and thus the amount of information collected. The teams addressed the limitations of the field research in several ways. For instance, the schedule was adjusted to interview communities before meeting with teachers at the school. This allowed a day for the head teachers and chiefs to call in teachers, especially QUIPS-trained teachers. District officers who were not present for structured interviews in the office were invited to weekend luncheons in two districts in the north. In fact, the working lunches helped verify information, because community members as well as GES officers attended and the setting allowed for open discussion and debate. In addition, some of the interview data during the field research was verified through a third day of visits to high scoring rural schools in the north, one community being visited at night. The visit verified and added detail to the system of pupil-led study groups that the community members and pupils had described earlier. 11 See Annex 8: Teacher Profile, Sampled Schools, North, Middle, and South. 12 See Annex 6: School-Community Categorized by Teacher Performance and Community Participation. Note that three of the high-scoring primary schools (Adukrom, Dunkwa Presby and Kpanlori) had QUIPS￾trained head teachers still at post. 9 The Evaluation Team is confident that the steps taken by each of the field teams were adequate to address the limitations to the field studies. Overall, the team trusts that the information brought back from the field is of a quality that ensures valid findings. 1.7 CONTEXT AND IMPACT OF OTHER DONOR ACTIVITIES At the suggestion of one of the QUIPS partners, the Academy for Educational Development, the following additional contextual information has been provided by them: “QUIPS implementation began just around the time that the DFID funded Whole School Development Program (WSD) got underway throughout Ghana. The WSD program [ ] operated in ways that sometimes appeared to create a competition for attention of key GES staff who were critical to ensuring the sustainability of QUIPS school reforms. The impact of the parallel implementation of the WSD program on QUIPS implementation is difficult to measure in quantitative terms, especially 1-2 years after each program came to a close, but from an implementation point of view, it should not be ignored. A few ways in which the WSD program impacted QUIPS are illustrated below: • Circuit supervisors and other DEO staff were frequently assigned to carry out WSD funded work which prevented them from participating in QUIPS supported activities at the district/school level. For example, when the QUIPS master teacher trainers came to a district for school based training and expected that circuit supervisors and DEO staff would participate as co-trainers, such individuals frequently simply weren’t available, even though they had previously committed to participate and had been given advance notice of the planned visit. • Financial incentives provided through WSD (e.g. per diem to attend workshops, special data collection initiatives, etc) made it more attractive to attend to WSD work than QUIPS and other general responsibilities. • Initially there was some confusion at the district to school level as to how to harmonize the resources and school based training approaches provided through both programs. (This issue was addressed and largely worked out about halfway through the life of the project when the key stakeholders formalized a process for harmonizing of approaches and materials developed by QUIPS and WSD.) Related to the above point about other donor initiatives taking place at the time QUIPS was being implemented, it could be argued that there is reason to be cautious about what appears to be a quiet assumption of the QUIPS evaluation strategy, namely that control schools did not receive other non-QUIPS treatments that were similar to the QUIPS school based in-service program (WSD in particular), or did not benefit over time from possible spread effects from QUIPS schools during the life of the project. In reality, this assumption did not always [apply]. That is, there were some instances where the control schools (or selected teachers and headteachers) did in fact benefit from non-QUIPS in-service interventions that were similar to the QUIPS interventions 10 during the life of the project, particularly in-service teacher training provided by WSD. In addition, the circuit supervisors that participated in QUIPS were encouraged to carry QUIPS approaches to other schools under their supervision during supervisory visits, and at least some of the circuit supervisors did indeed report doing so in other schools in their circuits, which included control schools. The QUIPS/ILP Project Office did request that during the annual data collection for the PMP reports that information be collected from control schools on the degree to which teachers and headteachers at these schools might have received other training that could impact their instructional behavior. However, for the most part, no additional analysis factoring this into account was incorporate into the annual PMP reports submitted to USAID, particularly in the final two years of the project, when spillover to control schools would have been more likely. No mention is made in the current evaluation study whether such possible confounding effects were explored during visits to control schools by evaluation team members or whether this issue was considered when carrying out the quantitative analysis - although at one point in the report it is mentioned that there was the possibility that the small differences in pupil achievement gains reported between control and QUIPS schools could be due to the possibility of QUIPS interventions having spread to non￾QUIPS schools. The likelihood of this having happened in the later years of the project (when QUIPS and WSD initiatives were harmonized and spread to non-QUIPS schools) could be more explicitly mentioned in the report [ ] as a possible complication in interpreting the data and findings related to achievement. CHAPTER 2: THE CONTEXT OF LEARNING IN GHANA 2.1 SCHOOL ACCESS AND FEMALE PARTICIPATION The developmental priorities documented in the Coordinated Program for Economic and Social Development of Ghana (2003-2012)—"Vision 2012"—and the Ghana Poverty Reduction Strategy (GPRS) 2003-2005 include strengthening democracy, invigorating economic growth, and improving the lives of Ghanaians by enhancing the quality of education and health services delivery. To benefit from the growing number of opportunities for earning a livelihood in an increasingly modern society, Ghana’s citizens must be able to read with understanding, write, and do simple mathematical calculations. The goals of equitable economic growth, poverty reduction, and sound democratic governance for Ghana, in other words, depend on equitable access to quality basic education. Although there have been modest increases in primary school enrollments in the past decade, from 72.8% in 1998 to 79.9% in 2002,13 these rates of growth are inadequate for achieving the 2015 Education for All goal stated in the MOE Education Strategic Plan (ESP), 2003-2015. Furthermore, the national figure fails to reflect the disparity of access in different areas and between girls and boys. Enrollments for the three northern regions of Ghana are estimated at 65%, compared to a national average of 79.9%. Girls’ enrollment in these regions are only 58% compared to the national 75%, and only 40% of girls in the north complete the full primary school cycle through Grade 6, compared to 68% of girls nationwide. Figure 2.1 below presents the net enrollment rates (percentages) for primary schools disaggregated by region; it underscores the problems of access in the three northern regions. A variety of known factors impede access to basic education in Ghana; all are directly related to poverty and socio-cultural factors. Distances to schools in some communities are too far for a young child to travel. Teacher shortages and absenteeism are high in hard-to-reach areas, deterring parents from sending their children to school. Due to the depressed economy, many families are not able to meet the costs of education (e.g., uniforms, materials, and school levies). They also keep children, especially girls, at home to help with domestic chores or to work elsewhere to contribute to household income. Socio-cultural factors, including early marriage and the fostering of girls to distant relatives further deter girls’ enrollment, especially in the north. 11 13 Latest statistics released by MOE. Figure 2.1: Percent Net Enrollments for Primary Schools14 69.9 74.9 72.6 80.9 64.7 75.6 78.9 69.3 49.9 56.1 51.1 0 10 20 30 40 50 60 70 80 90 National Western Central Gr Accra Volta Eastern Ashanti Brong Ah Northern Upper E Upper W 2.2 THE EFFECT OF POVERTY Figure 2.2 shows the relationship between family income and attendance in primary school. The data underscore the challenges faced in all three regions in northern Ghana (followed by the Central, Eastern, and Volta regions). The Ashanti, Western, and Greater Accra regions show attendance rates for both girls and boys of above 90%. This can be attributed to their association with Ghana’s three largest urban centers: Kumasi, Takoradi, and Accra. Poverty not only interferes with access and attendance, it is directly related to a child’s readiness and ability to learn. With 39% of the population in Ghana living below the poverty line, 27% in extreme poverty,15 it is not surprising that children arrive at school hungry and are often taken from school to help with domestic chores or augment the family income by working. Malnourishment in the north is high: over 37% of children less than five experience severe stunting compared to about 20% in the south. The relationship between chronic malnutrition and learning is well documented. For Ghana, this translates into a significant number of children who are at risk of academic failure because of malnutrition alone. Schools in poor rural communities are also less likely to have such basic amenities as latrines, water, and enough classrooms for the number of pupils. There, it is rare to find a school with a full complement of teachers, trained or untrained. Poverty, particularly rural poverty, correlates with reduced adult literacy. The rate of literacy among women in rural Ghana has been estimated at 28.2%, with the situation 12 14 MOES Primary School Statistics, 2004. Ghana Social Standards Survey (GSS). 15 The majority of the poor live in the north, where 80% of the population live in poverty (World Bank, 2002). worse (13.7%) for women considered to be “rural poor.”16 The emerging literature on literacy points to exposure to printed material and adults or siblings who read as important factors in a child’s academic development. The literacy rate data suggest that the majority of children in rural communities are deprived of this exposure. Figure 2.2: Poverty Incidence and Attendance Rates by Region17 POVERTY INCIDENCE AND PRIMARY SCHOOL ATTENDANCE RATES BY REGION 0 100 10 20 30 40 50 60 70 80 90 PERCENTAGE INCIDENCE OF POVERTY 88 84 69 48 44 38 36 28 27 5 SCHOOL ATTENDANCE: MALE 64.9 60.3 67.2 84.7 89 81.4 91.6 90.8 93.6 91.7 SCHOOL ATTENDANCE:FEMALE 54.6 73.2 57.3 82.6 90.4 83.6 83.9 87.8 90.8 85.9 Upper East Upper West Northern Central Eastern Volta Brong Ahafo Ashanti Western Greater Accra 2.3 PARTICIPATION OF GIRLS IN GHANA’S EDUCATION PROGRAMS Although for oys, especially in rural areas of the country and the north, it is alarming that the isparity between boys and girls persists all the way through secondary and tertiary ducation programs. These poor transition rates for girls again relate to both socio￾ultural and economic factors. For instance, many parents prefer to support boys, often the belief that a boy's education is an investment for a lifetime. Boys are expected to ssist their parents in old age while girls will belong to the husband’s family. Ghana, girls tend to drop out in upper primary and junior secondary school (JSS). ee Figure 2.3.) They are unlikely to be supported at higher levels of education such s senior secondary school (SSS). 18 In some regions, girls are perceived as too ulnerable to attend SSS because there is little supervision and they may be at risk of regnancy. Transactional sex among girls and boys is rising at the higher levels of ducation—a growing challenge to protecting Ghana’s youth from HIV/AIDS.19 the challenges of access to primary schooling are greater for girls than b d e c in a In (S a v p e 13 16 Ghana Social Standards Survey (2004). 17 Casely-Hayford, 2004; Ghana Social Standards Survey data, 2004. 18 Casely-Hayford, 2001. 19 Casely-Hayford, 2001. Girls at SSS and tertiary levels of education find it a financial challenge unless they are supported by their parents; they use sex as a way of paying for their basic needs. 14 Figure 2.3: Participation Rates by Gender and Educational Level .4 THE SOCIO-CULTURAL CONTEXT ocio-cultural factors that constrain access to basic education and impair its quality glect, all the labor, youth , imply that 9 of 10 pupils who complete six years of primary schooling do not master the English curriculum, and 19 out of 20 do not master mathematics: Pupils who complete the full primary school cycle still are unable to read with meaning or perform basic mathematics operations. Participation Rates by Gender at Various Educational Levels (2000/01) 80 30 40 50 60 70 Percentage 20 0 10 Girls Boys Girls 47.2 45.43 43.5 39 26.83 Boys 52.8 54.57 56.5 61 73.17 Primary JSS/Middle SSS Teacher Training University 2 S include the low priority parents place on education, distance to school, parental ne and attitudes and practices that impede girls’ education.20 In the last category f emphasis on marriage and motherhood, competing demands on a girl’s time for internal or external labor, and stereotyping of a woman’s role in society as being to care for children and a husband. These attitudes are exacerbated by persistent and growing levels of child unemployment, and youth migration. The growing incidence of streetism (“street children”) and prostitution, especially among girls from the north, are signs of urban unrest. The growing alienation among rural youth has roots in the education system which fails to meet their expectations and those of their parents. 2.5 EFFECTIVENESS OF THE PRIMARY SCHOOL PROGRAM In the last decade, although Ghana has seen improvement in the performance of primary school pupils, levels of mastery of English and mathematics21 are still alarmingly low. In 2002, 9.6% of Grade 6 pupils achieved mastery levels in English and 4.4% in mathematics. The figures would 20 Casely-Hayford, 2001. 21 Mastery level is being defined as a score of 60% for English and 55% for mathematics on the Criterion Reference Test (CRT) administered to Grade 6 pupils in Ghana. 15 uch has been learned about literacy since the United Nations declared it to be a basic o address economic, social, and political sues. When schooling is effective, children attain basic literacy at an early age, e tability for quality schooling. he management system is also weak. rograms, such as flexible schedules aligned with local needs and accelerated learning ecial t in school, the llion cedis (approximately $9.6 million) were earmarked r girls’ scholarships at the primary and JSS levels. The commitment to support needy children and to improving the quality of primary education demonstrated at the district level as well. In district development planning reports M human right. It is now seen as a major tool t 22 is around the third or fourth year of formal primary education.23 The majority of Ghanaian children attending public primary schools do not acquire basic literacy at all. The reasons are well-documented, among them inadequate resources allocated to basic education; ineffective, centralized management; and developmentally inappropriate teaching methods. More specific impediments to learning are low teacher attendanc and tardiness. Moreover, when teachers are in class, there is not enough time on task, inadequate resources in the classroom, and limited accoun T 2.6 GHANA’S RESPONSE TO CHALLENGES The Government of Ghana (GoG), through the Education Strategic Plan (ESP) and the Ghana Poverty Reduction Strategy (GPRS), has demonstrated a commitment to supporting the poor, particularly in remote rural areas. The ESP and GPRS recognize that a large number of children remain out of school (over 500,000) and suggest that children living in hard-to-reach areas should be supported through more innovative p programs for pupils who enter the education system late. The ESP also recognizes the need to target vulnerable groups, like children with sp needs, children living in areas of extreme poverty,24 and children who are no particularly girls. The GoG has initiated policies and programs targeting girls; in 2003-04 school year over 8.7 bi fo education for girls and is across the nation, there is increased in-service and cluster-based teacher training. Nevertheless, certain continuing systemic issues serve to maintain constraints to access and to compromise primary education quality in disadvantaged regions. Among them are district and school levies on parents of primary school pupils, central and district management and planning that is not information-driven, and limited accountability and transparency due to a lack of effective systems for tracking allocations and expenditures. 22 UNESCO: Literacy and International Development [online] http://www.literacyonline.org 23 LeFevre, J., Research on the Development of Academic Skills: Introduction to the Special Issue on Early Literacy and Numeracy. Page 3. Canadian Journal of Experimental Psychology Abstracts. June, 2000 Retrieved January 7, 2002 from http://www.cpa.ca/cjep/edito_eng.html. 24 The Ghana Living Standards Survey defines families living in “extreme poverty” as those earning less than 700,000 cedis ($78) per year. 16 the taining girls in schools in the north of Ghana and also contributes to their ability to provide hildren in selected districts with their most basic need, nutrition, while helping the ol. teacher support program. The World Bank supports education by providing frastructure and by building management and technical capacity within the MOE. DfID 2.8.1 Enrollment and Female Participation. Demographic data for 1999 to 2002 were collected at baseline from pupils throughout Ghana for Cohorts 3 through 6. Tables 2.1 and 2.2 summarize these findings from approximately 9,000 children. They underscore well-known disparities related to socioeconomic differences, from the most advantaged in the south to the least advantaged in the north. In Ghana, as elsewhere, urban communities spect to the vailability of books, exposure to English as a second language, and literacy in the home. 2 ge of G an ou is a crea a age of hi th to , th s reve G de 3 en in rth ar vera ut a y unger th ade 3 children in the south, but G child the n re abo ar older disparities are similar comparing rural and urban children: Rural Grade 3 children are 2.7 EXTERNAL AID A number of donor agencies are working within the education sector in Ghana. Over last three years, the GoG has moved toward a sector-wide approach that attempts to bring external funding under one comprehensive education plan, the ESP. Several donors have been supporting improved access and participation. Studies by the World Food Program suggest that providing food is an important factor in attracting and re learn. The USAID-funded school feeding program (PL480) was found to c children to access education and be cognitively ready to learn and to remain in scho The QUIPS program, along with the Whole School Development program run by the British Department for International Development (DfID), attempted to move beyond access and participation to improving quality within the classroom. Other programs with similar goals at the national and district levels are the ASTEP program run by the German Development Agency (GTZ) and the Japanese International Cooperation Agency in has provided budgetary support to increase district investment in in-service teacher training and supervision. QUIPS also made attempts late in the program to harmonize methodologies in the classroom in collaboration with GES and other programs. 2.8 ADDITIONAL NATIONAL FINDINGS ON PRIMARY SCHOOLING IN GHANA are advantaged over rural ones, particularly with re a .8.2 A Pupils in G c rades 3 ld m d 5. Alth e h gh there th rth slight de is i se in the verage rade 3 ren fro sout e no trend r sed in rade 5. Gra childr the no e on a r ge abo ear yo an Gr rade 5 en in orth a ut a ye . Age younger on average than urban, but the reverse is true for Grade 5. 17 le 2 l De hic s R of de 3 pils pled f e 5 ils pled e e 3 e 5 tage les e 3 tage les e 5 Tab ber .1: Pupi er o mograp s acros egions Region Num Gra Pu Sam Numb Grad Pup Sam Averag Age Grad Averag Age Grade Percen Fema Grad Percen Fema Grad South 3,019 2.668 10.42 11.44 37.06 54.91 Middle Belt 3,115 2,846 10.00 11.75 39.42 43.47 North 2,828 2,321 9.65 12.85 43.94 40.10 NATIONAL 8,962 7,835 9.95 12.15 40.86 44.82 *All ANOVA comparisons across regional sub-divisions were significant (ANOVA): p < 0.001. Age Females e Females Table 2.2: Pupil Demographics by Rural and Urban Locations Population Number Grade 3 pupils Number Grade 5 pupils Average Age Average Percentage Percentag Division sampled sampled Grade 3 Grade 5 Grade 3 Grade 5 Rural 5823 4817 9.78 12.27 41.48 44.06 Urban 3139 3018 10.33 11.74 39.42 46.59 NATIONAL 8962 7835 9.95 12.15 40.86 44.82 *All ANOVA comparisons across urban and rural locations were significant (ANOVA): p < 0.001. 2.8.3 Percentage of Females. For girls, the disparities are similar: Slightly more r in the It i in rural areas, particularly in the nort er, which would help explain the older ages of northern Grade 5 children in the sample. The lower percentage of girls in Grade 5 reinforces what has been well–documented: fewer northern girls persist th he prim e d he 2.8.4 Availability r Children’s Reading Materials at Home. ’ ure to prin ome (inc the avai of textbooks for their use at home), home literacy, and exposure to English in the home was collected also from pupils in Grades 3 and 5 in the QUIPS and Control schools and at baseline before the QUIPS inte ns were uced. Th describ state of affairs in Ghana from 1999 to 2002 al emerging literacy f girls in the south are enrolled in Grade 3 classes than in the north and slightly fewe the south are enrolled in Grade 5. There may be a tendency for families in rural areas to send young children, often pre￾primary aged, to school, as a substitute for child care while parents work, perhaps in fields; by Grade 5, enrollments more accurately represent the children who are attending for school’s sake and not for day care. s likely that, except for very young children in school for child care purposes, children h, start school lat rough t ary cycle du to competing emands for t ir time. of Textbooks and Othe Information about pupils expos t at h luding lability rventio introd e data e the with regard to critic actors. The results are summarized in Tables 2.3 to 2.7 that follow: 18 Table 2.3: Availability of Textbooks and Children’s Books at Home by Region rcentages) n Grade 3 English Text e 5 English Text e Grade 3 Math Tex ome Grade 5 Math Tex me Oth ks at Home (Pe Regio at Hom Grade at Hom t at H t at Ho er Boo South 11.83 13.93 5.82 32.07 47.35 Middle Belt 14.92 29.22 12.68 29.15 45.39 North 13.42 15.79 7.41 10.67 21.64 NATIONAL 13.47 19.35 8.59 21.57 35.19 Table 2.4: Availability o oks an ildren’s at Hom rban and l Locatio ercenta Locat rade 3 English Text rade 5 English Text rade 3 Math Grade 5 Math Other Books f o Textb d Ch Books e by U Rura ns (P ges) ion at Home at Home Text at Home Text at Home at Home G G G Rural 9.96 19.61 8.60 15.49 31.57 Urban 21.66 18.75 8.58 35.75 43.64 NATIONAL 13.47 19.35 8.59 21.57 35.19 Table 2.5: Home L nd to e n of Children tage ers ead P ge of Mothers who P ge Spoken in iteracy a Exposure English in th Home by Regio Region Number Sampled Percen of Fath who R ercenta Read ercenta English Home South 5,934 63.77 44.48 71.45 Middle Belt 6,150 63.16 31.59 54.79 North 5,528 26.87 16.75 52.19 NATIONAL 17,612 46.98 28.13 57.78 Table 2.6: Home Literacy and Exposure to English in the Home by Urban and Rural Locations cation ber of ildren mpled ntage thers Read P tage of s who ad entage Spok Lo ome Num Ch Sa Perce of Fa who ercen Mother Re Perc English en in H Rural 13,564 45.34 1 .07 28.5 56 Urban 4,066 50.81 26 .78 27. 61 NATIONAL 17,630 46.98 3 .78 28.1 57 19 Table 2.7: English Spoken to Children at Home by Region Does anyone speak to you in English at home? Region No English Spoken in Home English Spoken, but not to Me English Sometimes Spoken to Me English Always Spoken to me Total South 304 5.1% 2,092 35.4% 1,104 18.7% 2,416 40.8% 5,916 100.0% Middle Belt 188 3.1% 45.2% 20.8% 30.9% 100.0% 2,764 1,276 1,892 6,120 North 108 2.0% 2,892 52.3% 1,120 20.3% 1,410 25.5% 5,530 100.0% Total 600 3.4% 7,748 44.1% 3,500 19.9% 5,718 32.6% 17,566 100.0% For the nation as a whole, 9% to 22% of the pupils sampled had textbooks to use at home; results varied by type of textbook (math or English) and grade. As expe ewer Grade 3 pupils reported having textbooks for use at home than did G cted, rade 5 for ild’s me is highlighted more and more in the research tal cy and Exposure to Spoken English at Home. Early exposure nglish north. (See Tables 2.6 nd 2.7.) f pupils; math texts were particularly scarce. Equally predictable is the advantage of more populous and urban areas; there was a profound paucity of reading materials for northern children. Note that only about 35% of primary school pupils nationwide reported having supplementary reading materials at home; in the north, the percentage dropped to about 22%. Holistic approaches to effective schooling and related research emphasize the need parental involvement in primary schooling. The role of parents in supporting their ch learning through encouragement at ho as a factor underpinning academic growth in developing countries.25 However, paren efforts to encourage their children’s home study are precluded if there is no textbook or supplementary reader the child can use at home. This obstacle to learning is pronounced in Ghana, as evidenced by the data presented in Table 2.3. 2.8.5 Home Litera to print and to parents and siblings who read facilitates literacy development in the young child. Table 2.5 demonstrates the paucity of home reading material and exposure to English at home, particularly in the north. During the QUIPS baseline data collection, children were asked: ”Does your mother (father) read at home? “Does anyone speak English in the home?” and “Is E spoken to you by a family member"? The results suggest that children have limited exposure to reading role models and there is little support for second language–literacy outside the school environment. The situation is exacerbated again in rural communities, and in the a 25 Education Sector Review, 2001. 20 the last n years, although there have been some improvements, such as achieving gender he QUIPS evaluation and the GLSS have demonstrated significant challenges to romoting equitable educational investment where there are inequities, as in rural areas nd in the north. Targeted approaches to investing in education are needed to reach the oor and raise the quality of education in deprived contexts. onors whose programs complement educational investments (such as feeding rograms) are directly helping poor children attain a better learning outcome. They need to be encouraged to expand these programs, particularly in the poorest areas. The lack of learning tools, such as textbooks and reading materials both in the home and at school, remains a barrier to an educational investment like QUIPS. It needs to be tackled by all stakeholders. GoG should be held accountable for providing the tools that are basic to realistic learning achievement in resource-poor areas. 2.9 CONCLUSION Access and participation indicators have not moved at the speed expected over te parity within certain regions and increasing girls’ enrollment. The challenge for Ghana remains ensuring that all children receive better-quality education in the public system and that they remain long enough to achieve basic literacy and numeracy. T p a p D p CHAPTER 3: LITERATURE REVIEW: SCHOOL EFFECTIVENESS AND QUALITY EDUCATION School effectiveness and improving the quality of education worldwide have attracted international attention. For the past decade, the World Bank, USAID, DfID, and other multilateral and bilateral agencies have invested large sums of money in improving education in developing countries. The World Declaration on Education for All (1990) emphasized the provision of basic education to all children, youth, and adults. However, merely providing education is not enough. A high quality of education is essential if all learners are to be equipped with the knowledge tools they need to cope with and compete in an increasingly complex and competitive world. The World Declaration on Education for All (1990) says that “whether or not expanded educational opportunities will translate into meaningful development—for an individual or for society—depends ultimately on whether people actually learn as a result of those opportunities.” In the global economy, it is not sufficient that children simply attend school as a right. It is essential that they learn, that they acquire the basic tools of literacy and numeracy as well as skills in problem solving, critical thinking, and the work habits of diligence, creativity, and personal responsibility.26 The evaluation team reviewed the literature in light of the following issues: (1) What is “school effectiveness” and “quality education”? (2) Why are school effectiveness and quality education important to international agencies? In other words, what does international research say about the need for school effectiveness and quality education? (3) What does research in developing countries reveal about quality education? and (4) What is the status of education in Ghana and what challenges do Ghanaian schools face in their attempts to improve quality? 3.1 INTERNATIONAL RESEARCH In the past decade different international agencies have committed themselves to investing in a variety of interventions in developing countries. A large volume of research also has addressed school effectiveness and improving the quality of education in basic schools at both the international and the national level. 3.1.1 What Is “School Effectiveness” and “Quality Education”? Although there is a growing consensus among international and local researchers that school effectiveness and quality education are important, there is still disagreement about what the terms mean. The literature warns of a danger in confusing the terms “efficiency” and “effectiveness” and their descriptors “internal” and “external.” According to Lockheed and Hanushek, (1988), “Efficiency refers to a ratio between input and output. The output 21 26 Heyneman, 1989. 22 of education refers to that portion of pupil growth that can be attributed to specific educational experiences.” Inputs are conceived in broad terms to include the complex interactions of pupils and teachers, as well as textbooks, teachers’ salaries, and so forth. Lockheed and Hanushek restrict the term “efficiency” to monetary inputs and use ”effectiveness” for non-monetary inputs. They identify outputs expressed in non￾monetary terms (e.g., learning) as internal and those expressed in monetary terms (e.g., earnings) as external. Despite confusion in the use of these two terms, it can be argued that school “effectiveness” refers to all aspects of a school that are non-monetary, including learning, pupil-teacher interactions, and textbooks. 3.1.2 Why Are School Effectiveness and Quality Education Important? Several researchers (Creemers, Peters & Reynolds 1989; Raudenbush & Wellms 1991; Lockheed & Verspoor 1991) argue that to increase the pace of economic and social development in developing countries, schools must teach most school-age children the essential skills targeted by the primary curriculum: literacy, numeracy, communication, and problem-solving skills. The literature (see, for example, Lockheed & Verspoor 1991) gives evidence that variations in the characteristics of schools are associated with variations in pupil outcomes. The findings of Haddad and others (1990) show a consistent trend that variation in such school inputs as teacher experience, teacher motivation, the presence of textbooks, homework, and time spent in school during the year contribute to variances in pupil achievement, even when differences in family background are accounted for. In other words, if schools can improve these aspects of education quality, learning too will increase and the academic performance of pupils will improve. According to the EFA Global Report (2004), “It seems likely that the achievement of universal participation in education will be fundamentally dependent upon the quality of education available in the schools.” It adds that how well pupils are taught and how much they learn can have a crucial impact on how long they stay in school. The instrumental role of the school, the literature makes clear, is to help individuals achieve their own economic, social, and cultural objectives. Schools also may help society to be better protected, better served by its leaders, and more equitable—but only if the education is of a high quality. Another goal of schooling is to help children develop creatively and emotionally and to acquire the skills, knowledge, values, and attitudes necessary for responsible, active, and productive citizenship. Schools will be effective and beneficial to children and others only if they are able to achieve these goals. 23 The World Conference on Education for All held in Jomtien, Thailand, in 1990 recognized that expanding access alone would be insufficient for education to contribute to the development of the individual and society. The conference recognized the need to improve on the poor quality of education globally and recommended that education be made both universally available and more relevant. More recently, the Millennium Declaration and the Dakar Framework for Action (2000) set specific goals to ensure that all nations achieve free and compulsory primary education of good quality. Today, international agencies have realized that development will not be sustainable without investment in education. Numerous countries in Africa have benefited from USAID support for reforms in educational systems. In Uganda, for example, USAID helped the government to turn the teaching profession around: The Ministry of Education got rid of thousands of ghost and incompetent teachers. With the support of USAID, teachers' salaries were raised from $8 to $72 per month. The Ugandan Ministry liberalized the textbook market and adopted new procedures for procuring and distributing books to schools, breaking the monopoly of two publishers, Longman and Macmillan. In 1996, four local and 13 international publishers won contracts to supply instructional materials.27 Textbooks in Uganda are now less expensive than in neighboring countries. 3.1.3 What Does International Research Say about “Educational Quality”? Clearly, from the literature, quality education is multidimensional, depending on the researcher, organization, declaration, philosophical tradition, and approach. Bacchus (1991) identifies three major thrusts in efforts to improve the quality of education: (1) raising the academic performance of pupils in subjects offered in schools using currently available resources; (2) providing children with the education that is most likely to help them improve the quality of their lives when they become adults (also referred to as attempting to raise the effectiveness of schools); and (3) increasing the rate of school enrollment by providing more places and reducing inequalities between the sexes and the different regions in a country.28 The 1990 Jomtien Declaration identified quality as a prerequisite for achieving the fundamental goal of equity. The conference recommended that the cognitive development of children should be emphasized as an indicator of quality education. The Millennium Conference at Dakar agreed that quality was “at the heart of education and a fundamental determinant of enrollment, retention, and achievement.” The Dakar Framework for Action (2000) expanded the definition of quality to cover such dimensions as the characteristics of learners (how healthy and motivated they are), processes competent teachers use (active pedagogies), content (curricula), and systems (equitable resource allocation). Although the Framework established this 27 Christensen et al. 1997. 28 Bacchus 1991. 24 approach for achieving good quality, it did not ascribe any weighting to dimensions of the approach. UNESCO identified social change, the notion of life-long learning, relevance, and emphasis on science and technology as factors to improve the quality of education. It stated that “improving the quality of education would require systems in which the principles of scientific development and modernization could be learned in ways that respected the learner’s socio-cultural contexts.” 29 The International Commission on Education saw quality education as resting on four pillars: • Learning to know, which acknowledges that learners build their own knowledge daily, combining indigenous and external elements. • Learning to do, which focuses on the practical applications of what is learned. • Learning to live together, which addresses skills critical to a life free from discrimination, where all have equal opportunity to develop themselves, their families, and their communities. • Learning to be, which emphasizes the skills individuals need to develop to their full potential. UNICEF also promotes good-quality education as a human right. Within its rights-based approach, learning is perceived to be operated on at two levels – the level of the learner and the level of the learning system. UNICEF emphasizes five desirable dimensions of quality education: “learners, environments, content, processes, and outcomes that are founded on the rights of the whole child and all children to survival, protection, development, and participation.”30 There is an international consensus that quality education should uphold a child￾centered approach to learning that allows children to reach their fullest cognitive, emotional, and creative potential. To promote this type of education, curriculum content, textbooks, teaching processes, and environment should all promote children’s rights. Beyond quality education, international education agencies agree on the following three principles to guide and inform educational content and processes: (1) the need for relevance; (2) the need for greater equity of access and outcome; and (3) the proper observance of individual rights. It is believed that, given the right learning environment, all children can develop basic cognitive skills. “The failure of pupils to develop these skills at school is due in part to a 29 Education for All Global Monitoring Report (EFA Report) 2005. 30 UNICEF 2000. 25 deficiency in education quality or to poverty, rural residence, and gender inequalities and to poor instruction.”31 In summary, quality education is truly multidimensional. In the EFA Report (2005), the following are discussed as core dimensions that promote quality education: • Learner characteristics (how people learn is influenced by their capacities and experience). • Context: the link between education and the values and practices in society. Context is likely to constrain opportunities to increase resources for education. • Enabling inputs: the success of teaching and learning is influenced by the resources made available to support the process. Schools without teachers, textbooks, or learning materials will not be able to do an effective job. • Teaching and learning dimensions: instructional time, teaching methods, class size, assessment, and feedback. It is here that curricula have an impact. • Outcomes: these should be assessed in the context of agreed-upon objectives. 3.2 EDUCATIONAL QUALITY IN GHANA Studies conducted in Ghana32 show that the quality of basic education there is low. Despite funding by international donors and different interventions to improve the quality of basic public education, Kraft (2003) confirms that serious problems remain in the educational system of Ghana, especially the teaching of English language and math. The results of the Criterion-Referenced Testing (CRT) introduced into the Ghanaian education sector with the assistance of USAID showed that Grade 6 pupils’ achievement has been poor. In the 1994 sample, for example, only 3% of Grade 6 pupils scored satisfactory marks in English, and a dismal 1.5% in mathematics. 33 Several studies34 confirm that the failure of Ghanaian pupils to learn English can be attributed to the methods teachers use in the classrooms. Instruction took the traditional form of teaching a foreign language.35 Kraft (2003) confirmed that didactic modes of teaching characterized by rote learning are still prevalent. As Dzameshie (1997) put it, the teaching of English in Ghanaian classrooms has been more analytical and grammar-based than meaning-oriented. Benzanson and Hawkes (1972) described the teaching methods of Ghana as traditional, whole-class, and teacher-dominated. The early stages of reading often consisted of alphabetic and look-and-say work, with words 31 EFA Report 2005. 32 Centre for Research on Improving the Quality of Primary Education in Ghana (CRIQPEG) 1993, 1996; Angmor, Jakalia, Dzaka and Asante 1997; Etsey 2004; Kraft 1993, 1994, 2003. 33 CRIQPEG, 2003. 34 CRIQPEG 1996; Etsey 2003; Dzameshie 1997; Kraft 2003. 35 Freeman 1998. 26 or sentences mechanically repeated aloud. Actual reading is still delayed until Grade 2, when teachers copy short texts on the blackboard for children to read aloud in unison. Reading of actual texts starts only in Grade 3; the common belief is that children need to master structures of language and reading readiness before moving on to read text. Instruction in the lower grades is focused on teaching the blending of letters and sounds to form words. Teachers move from teaching two-letter words to three-letter words and beyond. As Etsey (2004) put it, this is the typical bottom-up, skill-based instruction encouraged by the behaviorist tradition.36 In field visits to Ghanaian classrooms Etsey observed little instruction about how to comprehend passages. The method of teaching comprehension was to test comprehension; the process of learning comprehension was neglected. The missing element was the failure to teach strategies pupils can use in constructing meaning from written passages: Questions from teacher and textbook were factual. The children did little thinking; they just located the answers in the textbook and copied them into exercise books. The Ghanaian approach to teaching reading and writing did not help the children become independent readers and writers. 3.2.1 Equity Issues. Equity is a prerequisite of quality education. According to Kraft (1995) there is a dramatic difference between the educational opportunities available to children in rural settings and those who attend school in towns, regional centers, or the national capital. Also, there is overwhelming geographical disparity between the southern, central, and northern zones of Ghana in every aspect of schooling: infrastructure, toilets, textbooks, management, parental wealth, the training of teachers, instructional materials, etc. In 2003 Kraft found most of these issues still unresolved. The growth in private schools and their impact on social mobility has been studied by the former Vice-Chancellor of the University of Ghana, Ivan Addae-Mensah (1974, 2002). He shows that over a 25-year period, entry into tertiary education in Ghana has been primarily through 25 public secondary schools. Access to one of these elite secondary schools has been the driving force behind investment in private primary education by a large number of middle-class families. A recent study of private schools in Ghana (EARC 2002) shows parents enroll their children in private schools because they believe these schools offer a better quality education, so an increasing number of children are entering the public secondary school system after completing private primary education, usually in a town or district capital. GES figures for 2004 indicate that of 16,000 primary schools in the country, 3,600 are approved private primary schools (22%). The number of private primary schools in a district capital north of Kumasi increased from four in 1993 to 64 in 2004.37 36 See, for example, Finn 1990. 37 DEO, Techiman, Brong-Ahafo. 27 3.2.2 Teaching and Learning Issues. The international community agrees that teaching and learning is the key arena for human development and change. Kraft’s reviews of teaching, learning, and the curriculum (1994, 2003) reveal that in developing countries like Ghana, home and community contexts are not the variables that exert the most influence on teaching and learning, as they are in the first world. Fuller (1992) hypothesized that in third world countries, the school is a more powerful influence on the success of children. In Ghana, although parents are supportive of their children attending school, there is a wide range of other factors, particularly in the rural areas, that explain why Ghana is plagued with large numbers of primary and Junior Secondary School (JSS) dropouts. Among them are high levels of poverty, parent illiteracy (prevalent in rural areas), and lack of access to television, radio, and reading materials.38 Moreover, some children, especially girls, miss school due to illness, working on the farm or in the home, or selling items on city streets. 3.2.3 School Factors. Kraft (1994) reports that a range of school factors correlate with higher achievement. Some factors that affect teaching and learning in Ghana are: • Large class and school sizes. • How involved head teachers are in decision-making, teacher supervision, monitoring of the class schedule and curricula, and ensuring that textbooks, syllabuses, and handbooks are available to teachers. • School-wide policies on discipline, attendance, tardiness, and absenteeism and their enforcement by heads of schools. • School cultural factors, such as orderly atmosphere, high expectations, leadership by head teacher—all characteristic of high-achieving schools. • Access to schooling: through the World Bank and other international agencies, Ghana has made a concerted effort to make primary schools more accessible. • Design and delivery of curriculum and instruction. The government has mandated that English be the language of instruction from Grade 4 on, but reading achievement in the school’s official language suffers when children speak another language at home (Kraft 1994). In Ghana, few children speak English or hear it used at home. This makes it difficult for the children to understand what they are taught at school. Moreover, not only is the amount of time limited that Ghanaian children are exposed to learning English at school, but teachers themselves are not often comfortable in the language, so trained teachers end up using a local language for much of the formal curriculum. Other factors that affect the quality of teaching and learning in Ghana are: 38 Manu 1993; Fianu 1992; Kraft 1994. 28 • Irregular and late distribution of textbooks. • Inadequate teacher training in pedagogy (Kraft 1994). Instructional practices in Ghana are overwhelmingly teacher-centered, and dominated by rote learning and copying off the board. Pupils find this boring and repetitive. While some Ghanaian teachers use a questioning/recitation strategy, their questions tend to be simplistic, and are answered only by the best pupils—most children are not actively involved in lessons. It has been globally demonstrated that this does not promote quality education and achievement in school. • Lack of teachers, especially in rural areas, where teachers are reluctant to accept posting. • Minimal teacher commitment because, teachers are underpaid; they are often absent or arrive late and leave early, especially in rural settings. (Kraft 1994) • Poor classroom management. • Traditional pupil-teacher interactions: Positive responses to questions from other pupils and teachers are correlated with achievement (Kraft 1994). In the Ghanaian classroom, Kraft reported, not only were 100% of questions teacher￾initiated but they were simple yes-or-no questions to which pupils repeated correct answers in unison. He also found no evidence that cooperative learning and small group instruction were ever used. (This is an instructional strategy QUIPS tried to introduce.) • Quantity of instruction: In Ghanaian schools there are endless interruptions of instruction by such activities as sporting and cultural festivals and teacher absenteeism and tardiness. • Pupil characteristics: In Ghana, there has been limited research into how learning characteristics affect achievement. The only aspect mentioned by Kraft (1994) is gender studies—how boys did marginally better than girls. Kraft recommended that teachers be trained in the use of meta-cognitive strategies and how to use alternative tools to regularly evaluate pupils’ mastery of knowledge and then adjust teaching approaches to areas of need. • Irregular distribution of textbooks and school resources: Scarcity is exacerbated by poor central record-keeping, leading to over- and under-supply of educational materials in schools. • Inadequate textbooks, a problem exacerbated by teacher and learner characteristics and the lack of supplementary instructional materials. • Unavailability of supplementary reading materials, a key to achieving literacy. • Absence of workbooks and exercise-books to give children more practice in mathematics and other problems. • Shortage of teachers’ handbooks, which are in any case repetitive and restrict the teacher to a limited variety of teaching strategies. • Functional illiteracy: A large majority of Ghanaian school children cannot function well in either English or their mother tongue. • Shortage of libraries: The few libraries in Ghanaian schools are, moreover, not fully utilized by either pupils or teachers. 29 Most of the problem areas Kraft identified in 1994 are still evident. The five most critical, demanding immediate attention, are the policy on language of instruction, the prevalence of reading difficulty among children, lack of an appropriate pedagogy to teach English as a second language, lack of textbooks, and teacher time on task. 3.3 ISSUES PREVALENT IN GHANAIAN SCHOOLS 3.3.1 Language Policy Issues. Kraft has criticized the latest Ghanaian English-only language policy, explaining how this policy does not work in the current Ghanaian educational environment: “The latest Ghanaian language policy reads as if the nation had sought to unilaterally disarm itself and to commit intellectual, cultural and educational suicide…. Ghanaians, not unlike most Africans, are among the world’s great linguists, often with the ability to communicate in three or more oral languages. To effectively deprive them of the ability to become literate in one of those languages, as a gateway to the mastery of neighboring languages or of English, can only be described as educational malpractice.”39 If the practice of ignoring the support the mother tongue gives in learning is not dealt with, Ghanaian children will continue to have a crisis in literacy. When children are first literate in their mother tongue, they can easily transfer basic literacy skills into the second language (Cummins, 1991, 1994; Kraft, 2003). 3.3.2 The Reading Issue. Unless the culture of reading is re-introduced into Ghana, libraries are established, and enough supplementary reading materials are provided for the schools, Ghanaian children will continue to have difficulty reading. A culture of reading can come into being only when materials, even pupil-written, become more widely available (Kraft 2003). 3.3.3 The Issue of Pedagogy for Teaching English. In his detailed comparison of the literacy learning and pedagogical situation in the U.S and Ghana, Kraft (2003) showed the inherent problems of literacy instruction in Ghana. Ghanaian teachers need pre- and in-service training in methods for teaching English. 3.3.4 The Textbook Issue. New textbooks need to be written to match the content of the syllabus. 3.3.5 Teacher Time-on-Task Issue. Schools need to make good use of all the instructional time available. That appears not to be the case currently. 39 Kraft, 2003, page 19. 30 3.4 CONCLUSIONS This brief review of national and international educational research on effective schooling, quality of education, and factors that affect learning describe the context within which the QUIPS program was rooted. Our conclusions are as follows: • There are high social and individual returns on investment in primary education. • School inputs consistently contribute to school achievement. • The five core dimensions influencing quality education and teaching and learning are the characteristics of the learner, the social context, inputs, the teaching and learning process, and educational outcomes. Quality school programs must be built around these dimensions. • There has been little impact from the international resources Ghana has received to enhance the performance of its primary school system. Ghanaian children still fail to master reading and writing in the English language. • There are severe disparities in the educational opportunities available to children in rural areas compared to those in district centers and towns; rural areas lack teachers, textbooks, readers, and instructional materials. • Problem areas identified by Kraft in 1994 are still evident: including policy on the language of instruction, reading difficulties, pedagogy for teaching English as a second language, textbook issues, and teacher time on task. • Parents are sending their children increasingly to private primary schools, then enrolling them in public secondary schools. Private schools are perceived as offering a better quality education. CHAPTER 4: OVERVIEW OF QUIPS INPUTS, RESULTS FRAMEWORK AND RELATED PERFORMANCE This chapter reviews the findings from the evaluation of QUIPS performance in terms of the Strategic Objective (SO2) Results-Based Framework, using QUIPS performance measurements during the QUIPS program cycle and the data stored at that time. 4.1 CORE INPUTS OF THE PROGRAM The QUIPS partnership school program was implemented through an integrated program of training and support that targeted improvements in teacher instructional practices and school management, community involvement in education, and national education policy in support of quality primary schooling. The GoG and USAID worked with three primary contractors and subcontractor affiliates to implement QUIPS nationwide. These were the school-level inputs the QUIPS program provided: 4.1.1 Training and Teacher Support Inputs • A series of in-service training sessions were delivered on site at each primary school (for 2,202 teachers, 367 head teachers, 880 Circuit Supervisors) with demonstration lessons, curriculum review, and reporting to the community about the training. • Support was provided to 38 teacher training colleges (320 tutors trained) using the primary school teacher in-service course content that focused upon the use of teacher aids in the classroom. • Training of trainers was provided for 550 education officers and Circuit Supervisors on English language teaching strategies. • Training of trainers was delivered to district teacher support teams in 35 districts during Cohorts 5 and 6. • Manuals were prepared for teachers and head teachers (i.e., reading, teacher aid use, continuous assessment, lesson notes, weekly forecasts). • Six handbooks were harmonized to fit with the inputs of various donor agencies and NGO’s. • HIV/AIDS education materials were developed. 4.1.2 Infrastructure Development Inputs 31 • QUIPS undertook 275 projects in 86 districts. (A total of 707 new classrooms were built, and 254 classrooms were renovated, 129 teacher’s quarters and 9 school libraries were constructed.) 32 4.1.3 District and National Level Inputs • Training of trainer sessions were provided for 550 district and national officers on the use of performance appraisal instruments. • Training was provided in 110 districts for managing the district grants program. • The Circuit Supervisor Handbook and the Addendum to the Head Teacher's Handbook were prepared and distributed. • Training of trainers sessions were delivered for 550 district and national officers in using the Circuit Supervisors Handbook and Addendum to the Head Teacher's Handbook. 4.1.4 Community Level Inputs • Participatory learning and action (PLA) studies were prepared. • Community development facilitators were provided for two years. • Support was provided to communities to design projects. A total of 439 communities planned and implemented 1,317 projects using School Performance Improvement Plans (SPIPs). • Training was given to SMCs and PTAs in 439 communities in 110 districts. • Training and support was delivered to a total of 2,200 communities through the District Grant Mechanism, which included 7,700 primary school PTA/SMC representatives trained in communications, budgeting, fund management and advocacy skills. • Training was delivered to SMC/PTA representatives from 12,000 school/communities as a part of the SMC/PTA national expansion program. Practical handbooks were developed for the SMC/PTA training (80,000 copies produced and distributed nationally). • Micro-grants were provided to 1,317 school improvement projects. • Community drama and remobilization activities were undertaken to renew the commitment and performance of communities. In addition, school feeding through the Food for Peace Program (PL480) was provided by Catholic Relief Services in 92 schools in the north. USAID used the best practices emerging from the QUIPS community, school and district interventions to engage in policy dialogue with the MOES and GES. The intention was to formulate and implement policies to improve the effectiveness of primary schooling within the context of decentralized management of primary education. 4.2 GENERAL APPROACH The QUIPS Program, as mentioned above, was implemented in three schools in each of 110 districts and provided 24 months of training and technical support to partnership school/communities in a roll-out consisting of six cohorts. 33 After the mid-term evaluation in 2001, a shift in strategy led to a transfer of Non-project Assistance support to Project Assistance directed toward spreading the QUIPS interventions using a District Grant Mechanism, including district-level monitoring and evaluation nationwide. District teams were trained to provide leadership in three areas: teacher training and supervision; monitoring and evaluation; and planning and development. 4.3 OVERVIEW OF THE SO2 RESULTS FRAMEWORK The Strategic Objective 2 results framework includes: (1) the overall program objective; (2) planned intermediate results (sub-objectives); (3) indicators to measure progress towards the objective and sub-objectives and (4) annual indicator targets. This section reviews each of these elements of the framework with particular attention being given to achievement of the 2004 targets, the final year of the program. 4.3.1 Strategic Objective 2: Increased Effectiveness of the Primary Education System Indicator 2.1: Percentage of pupils participating in QUIPS schools passing the Criterion Referenced Test (CRT) in English and in math. • 2004 Target: Pupils in QUIPS schools demonstrate CRT results 4% higher than those for the nation. • 2004 Result: The CRT was discontinued in 2001. Therefore, results comparing partnership schools to the nation were available only in 2000. At this time, pupils in the partnership schools did show an achievement advantage on the CRT compared to the nation, with scores 3.6% higher in English and 2.1% higher in mathematics. • Comments: Although Grade 6 pupils who had participated in a QUIPS partnership school did present improved performance, it is remarkable that the pass rate on the CRT for English (13.2%) and mathematics (6.3%) was notably low, even for “QUIPS pupils.” It is noteworthy that the gap (i.e., favoring pupils who participated in QUIPS) declined from 1999 to 2000. Indicator 2.2: Increased pupil learning in mathematics, English reading, and spoken English in QUIPS partnership schools • 2004 Target: Percentage of correct scores of 60%, 55%, and 60% for mathematics, English reading, and spoken English. • 2004 Result: Performance on this indicator surpassed the target for mathematics at 64% and fell just short of the target for English reading (53% vs. 55%) and spoken English (53% vs. 60%). • Comments: Performance measurements on this indicator are based on measures of static achievement among pupils who have just completed two years of schooling within a QUIPS school. Pupils’ scores were aggregated at the 34 school level and then compared across schools, class (i.e., Grades 4 and 6) and cohorts. Although the static achievement performance provides a mechanism for tracking overall performance within the QUIPS population of schools over time, it fails to provide information about the gap between QUIPS and control schools. Moreover, it does not address the pattern of pupils learning growth during the two-year cycle of intervention. The QUIPS Achievement Tests, however, were designed primarily to provide information about pupils’ learning growth. The QUIPS monitoring and evaluation system allowed for comparisons with “non￾QUIPS schools.” A more meaningful indicator of success on pupils’ achievement outcome is to measure the relative gap in pupils’ achievement growth between the QUIPS and non-QUIPS or control school populations. Indicator 2.3: Percentage of communities demonstrating sustained community involvement in the education process. • 2004 Target: 75%. • 2004 Result: Performance on this indicator surpassed the target, with 97% of the communities demonstrating sustained involvement, with communities sampled in the years after the QUIPS interventions were withdrawn. • Comments: The quantification of sustained performance rarely is observed in development projects, particularly with regard to community practices. The investment taken to develop a valid and reliable system to address sustainable impact is noteworthy and its importance cannot be overstated. Indicator 2.4: Rate of pupils' dropout in partnership schools compared to national rate. • 2004 Target: 5% dropout rate in partnership schools. • 2004 Result: Performance on this indicator surpassed the target, with dropout rates estimated at 3.8% and 4.6% for boys and girls, respectively. • Comments: Dropout rate is a chronic problem in developing countries as there is extraordinary competition for a child’s time, particularly when there is a need for children to supplement the family income and/or care for young children in the home while parents work. The collection of data needed to accurately measure pupils’ dropout rate often is difficult to obtain. Indeed, this was the case for Ghana during the QUIPS Program as the MOES was, during this time, in the process of putting into place a new Education Management Information System or EMIS. Aggregate statistics for sub-populations of schools such as the QUIPS schools that are taken from national primary school statistics always should be cross-validated. The QUIPS Program put this process in place in 1999 but the cross-validation process was discontinued by 2002. 4.3.2 Intermediate Result 2.1: Improved Teaching and Supervision Indicator 2.1.1: Percentage of teachers using pupil-focused instructional practices. • 2004 Target: 62%. 35 • 2004 Result: Performance on this indicator surpassed the target. Aggregated performance (i.e., across cohorts) was calculated at 73% of the teachers demonstrating pupil-focused instruction, including: effective questioning, encouraging pupils’ participation, provision of feedback, applying a variety of methods and materials, and stimulating critical thinking. • Comments: Even though performance on this indicator was high relative compared to the planned result (i.e., 62%), results from current field work observed limited spread and sustainability, even among QUIPS-trained teachers. Once teachers were no longer provided the support and guidance that was characteristic of the QUIPS interventions, the practices were impeded. Sustained classroom reforms were observed among QUIPS schools where the majority of QUIPS-trained teachers were still working in the school and where the head teacher championed the approaches and provided ongoing support to teachers. Indicator 2.1.2; Percentage of teachers using effective classroom management. • 2004 Target: 50%. • 2004 Result: Performance on this indicator surpassed the target, with aggregated performance (i.e., across cohorts) at 56% of the schools demonstrating effective classroom management techniques including: minimal use of time for routine non-teaching tasks, good use of classroom space, strategic grouping, discipline, and interaction. • Comments: None. Indicator 2.1.3: Percentage of head teachers using effective supervisory techniques. • 2004 Target: 61%. • 2004 Result: Performance on this indicator failed to meet the planned target of 61%. Thirty-one percent of the schools demonstrated effective supervision as measured by head teacher visits to classrooms and regular staff meetings. • Comments: Observation of the quantitative data given for measuring this indicator showed that there was very little variability on measurements of “regular staff meetings.” That is, almost every school successfully met the requirement of holding two staff meetings per term. However, some variability was observed in teacher visits to classrooms, although it was not possible to ascertain (i.e., from head teacher records) the nature of these visits. The validity of these indicators as measures of “effective supervisory techniques” therefore is questionable. These indicators probably should be re-visited in future programs. 4.3.3 Intermediate Result 2.2: Improved Education Management Indicator 2.2.1: Number of partnership schools that demonstrate the ability to manage material resources. • 2004 Target: 239 QUIPS schools. 36 • 2004 Result: Performance on this indicator fell just below the 2004 target, with 235 schools demonstrating effective management of resources as measured by accurate and up-to-date records of both construction and school materials. • Comments: Inspection of these data brings into question the reliability of the measure of this indicator. On the surface it would seem that the establishment of good record keeping is a critical first step in effective management. Indicator 2.2.2: Number of district education offices that demonstrate the ability to improve the planning, implementation, monitoring and evaluation of education programs. • 2004 Target: 46 district education offices. • 2004 Result: Performance on this indicator fell eight districts short of the 46 DEOs expected to successfully plan and implement the District Grants. • Comments: The process by which this indicator was measured appeared to be relevant, practical as well as valid and reliable. 4.3.4 Intermediate Result 2.3: Increased Community Participation Indicator 2.3.1: Percentage of communities active in school decision-making. • 2004 Target: 65%. • 2004 Result: Performance on this indicator surpassed the target with aggregate performance (i.e., communities across six cohorts) of 85% active in school decision-making processes as demonstrated by community members empowered to act in support of school quality improvement. • Comments: None. Indicator 2.3.2: Percentage of communities using participatory methodologies in initiating school projects. • 2004 Target: 70%. • 2004 Result. Performance on this indicator approximated the planned target with an aggregate performance (i.e., communities across six cohorts) of 71% demonstrating effective action planning in addressing school quality improvement aligned with the interests/concerns of the school. • Comments: None. 4.3.5 Intermediate Result 2.4: Improved Learning Environment Indicator 2.4.1: Number of schools adopting techniques to promote equity. • 2004 Target: 239 QUIPS schools. • 2004 Result: Performance on this indicator surpassed the target with a total of 331 schools where the majority (i.e., 75%) of the teachers demonstrated encouragement of all learners, and girls in particular, to participate in class. • Comments: Encouragement of pupils, including girls, was observed in high performing QUIPS schools visited during the field study. Furthermore, 37 quantitative results identified “encouragement of girls’ participation” as significantly related to learning growth and identified a significant relationship between communities that supported girls’ education and schools demonstrating a high pupil achievement outcome. Indicator 2.4.2: Number of communities and schools implementing a school improvement plan. • 2004 Target: 239 • 2004 Result: Performance on this indicator fell below the planned target with a total of 176 school-communities successfully demonstrating the ability to meet both annual targets set on the School Performance Improvement Plan (SPIP) and modify targets annually to meet the changing requirements of school improvements. • Comments: Although responsive community action related to school improvement was observed during the current field study, the use of planning was not as clear as one would expect considering the emphasis on systematic action planning (i.e., the SPIP). 4.4 CONCLUDING COMMENTS ON THE SO2 PERFORMANCE INDICATORS The USAID/Ghana results framework has a set of quantifiable performance indicators and an exceptionally rich source of data from which to measure them. The instruments used to measure pupil achievement, teacher instructional practices, and community participation were reliable and valid. The evaluation team was able to conduct this evaluation, combining both qualitative and quantitative results and linking inputs to learning outcomes, only because these data are so comprehensive. The breadth of the monitoring and evaluation data that characterizes the system applied in QUIPS is not common. Because of this, the utility of the information generated from these data far surpasses their use merely as a tool for reporting annual program results. Thirdly, issues related to internal validity need to be mentioned: • All implementing agencies must follow exactly the same processes in collecting data that are being compiled for evaluation and reporting. There is no evidence that the processes for collecting the classroom and community data were consistent in northern and southern Ghana. • And finally, a third issue is that data regarding teacher and community practices were not collected for the control schools. 4.5 PERFORMANCE ON SO 2 The QUIPS program was successful in achieving the SO as measured by all four performance indicators: The Grade 6 CRT, QUIPS pupil achievement tests (math, English reading, and spoken English), sustained community participation, and reduced 38 dropout rates. Indicators related to pupil achievement, although the most important outcome in any school quality or effectiveness initiative, are the most difficult to address. The effectiveness of these indicators is discussed briefly below. QUIPS used the national assessment program, Ghana’s primary school criterion referenced testing (CRT) program, as one indicator for the SO2. Using national assessments as indicators in development programs, although of strong validity, can present problems. National assessments attempt to understand the degree to which pupils completing school programs meet the performance expectations of those programs, in this case the expectations of primary school programs in Ghana. They measure pupil performance on a national achievement assessment instrument tied strictly to the national curriculum. Ghana’s CRT was based on the Grade 6 curriculum, as it should be. However, most children completing Grade 6 in Ghana’s public schools are functioning well below grade level—even in QUIPS schools. Since the national testing program is not aligned with the actual level of most pupils (even though generally aligned with the nation’s expectations of a pupil completing public primary school), it is difficult to compare children in the QUIPS and control schools based on this criterion. That being the case, it is impressive that by mid-term 2001, a higher rate of pupils in QUIPS schools had passed the CRT than in the nation as a whole, even though the gap in the pass rates was small. The QUIPS Achievement Tests were designed to provide information about pupil learning over time and the QUIPS monitoring and evaluation system allowed for comparisons with control schools, but the results framework indicator reported aggregated results of static performance only among QUIPS schools. Aggregating results in a program where schools are at different levels of the implementation can confound results. The database included pupil achievement information on both QUIPS and matched control schools even in the out-years; it might have been better to use the comparative data as part of the performance indicators for achievement. If comparisons of QUIPS and control schools had been made at the end of the two-year cycle and during the years after QUIPS, we would have a measure of both immediate and sustained outcome. It is impressive that the program was able to reduce the dropout rate in QUIPS schools in comparison to the national average. This is noteworthy, given the opportunity costs of schooling to parents in Ghana today: the cost of losing children to help in the home or to bring in additional income. QUIPS investment in cross-validating primary school statistics based on national EMIS programs in the first half of the program is commendable. It would have been good to maintain this effort throughout the program. 39 4.6 PERFORMANCE ON INTERMEDIATE RESULTS The QUIPS program was successful in shifting teacher instructional practices, notably by teachers using pupil-focused instruction and better classroom management. The challenge discussed at length throughout this report is how to sustain these teacher outcomes when the integrity of the public school teaching profession and conditions of service are still poor. The target for teacher supervision was not met. The validity of the performance indicators measuring “effective supervisory techniques” was questionable. They should be revisited in future programs. Head teacher supervision and district management are difficult to quantify; they are, and should be treated as process indicators. For process indicators, it may be best for USAID Missions to adapt a qualitative or hybrid approach, such as those used in measuring community best practices. Such approaches help to quantify, inter alia, processes like district monitoring and evaluation systems, EMIS, school and district teacher supervision systems, and district planning processes. CHAPTER 5: LEARNING OUTCOMES OF QUIPS 5.1 INTRODUCTION Three sets of comparative analyses, comparing aggregated performances of QUIPS and control schools, were conducted to study learning outcome. They were: (1) comparisons of pupil achievement growth during the two-year intervention cycle; (2) exploratory study of the residual effects from the end of the intervention cycle through the second year after the withdrawal of active QUIPS interventions; and (3) descriptive analysis of the types of skills and general class performance of pupils completing Grades 3, 4, 5, and 6 in QUIPS and control schools. During the QUIPS program pupil achievement was measured with an emphasis on learning growth; the performance of individual pupils in QUIPS and control schools was tracked for the two years of the QUIPS intervention cycle through repeated testing. The QUIPS achievement tests were administered to the same children on three occasions: at baseline, at the end of the first year of intervention, and at the end of the second year. This allowed for the longitudinal study of pupil achievement and analyses of achievement growth during the two-year period of active QUIPS interventions. In addition, static achievement performance of Grades 4 and 6 in QUIPS and control schools was assessed in the years after QUIPS interventions ceased. This allowed for investigation of potential residual effects of QUIPS on learning outcomes. Finally, the items that make up the QUIPS math and English reading achievement tests span the full range of the primary school curriculum related to essential skills in these subjects. As a result, the level of performance (general grade level attained) and the types of skills acquired by the majority of pupils can be described based on analyses of test item data. (See Tables 5.9 and 5.10 for details.) 5.1.1 Test Instruments and Administration. As part of the QUIPS monitoring and evaluation program, pupils in Grades 3 and 5 were tested in math, English reading, and spoken English narrative. The curriculum-based test instruments include items that span the primary school classes, Grades 1 through 6, thereby capturing a baseline for all children regardless of their entry-level abilities and allowing room to observe change even for the highest-performing pupils. The mathematics tests focus on basic operations, including story problems and basic concepts such as “geometrical figures” and “measurement.” All test instructions and story problems were presented orally in both the local language and English, avoiding the “reading bias” imposed when pupils are required to read in order to complete math 40 41 test items. The English reading test used a multiple choice format and emphasized ability to read with meaning. Both math and English reading tests were conducted in group classroom settings, with from 20 to 30 pupils sitting for each test. The spoken English narrative consisted of a prompted story-telling activity in which pupils were asked to retell a story read to them in English while looking at a picture book. Before telling the story, pupils heard the English story on three occasions, in a small group setting with all group members participating together in the re-telling experience or on an individual basis. The final English narrative assessment was conducted on an individual basis. About eight Grades 3-4 pupils and eight Grades 5-6 pupils were sampled from each school. The numbers of girls and boys in the study were equal. Given the limited exposure of many Ghanaian primary school pupils to formal test￾taking, pupils were trained with practice tests; there was individual and group support in the process. Pupils were given as much time as they needed to complete the tests so as to avoid any time-constraint bias in soliciting a child’s best performance. The math and English literacy tests were scored in a conventional fashion, with each item scored as “correct” or “incorrect.” The performance tests were scored using a series of response categories measured on an ordinal scale.40 The final step was an assessment of the quality of the test instruments used by QUIPS to assess pupil performance. (For the results of this critical review of the instruments, see Annex 10, Independent Review of the QUIPS Achievement Tests, by Professor Wes Snyder.) Overall, the QUIPS tests met most criteria for good indices for measuring education reform. 5.1.2 Analysis Procedures. A multilevel modeling procedure (HLM: “hierarchical linear modeling”) was applied for analyzing differences in QUIPS and control schools with regard to pupil achievement growth during the period of direct QUIPS support and training. (See Section 5.2 below.) The repeated measures analysis of covariance (ANCOVA) was applied for exploring trends in achievement and the gap between QUIPS and control schools two years after the QUIPS intervention ended. (See Section 5.3 below.) The descriptive analyses of the general class level of performance and the types of skills children know at the end of Grades 3, 4, 5, and 6 consisted of ranking the test items, in descending order, according to the percentage of pupils who correctly 40 Pupil responses on the storytelling task are scored according to the following categories: (1) “5” = self￾initiated exceptional verbal response; (2) “4” = self-initiated grammatically accurate verbal response; (3) “3” = self-initiated with grammatical inaccuracies; 4) “2” = correct verbal response to a direct question given by the test administrator; (5) “1” = pointing response to a direct pointing request given by the test administrator; (6) “0” = no response even after test administrator cues given by direct questions and pointing responses. 42 answered them. (See Section 5.2.4 below.) A discussion and rationale for the HLM and ANCOVA procedures is given in Note 5.1 at the end of this chapter. 5.1.3 Data Used in the Analyses. Data for comparisons of pupil learning growth in QUIPS and control schools consisted of longitudinal pupil achievement data from schools in Cohorts 4, 5, and 6 collected at baseline at the end of the first year of interventions and at the end of the second or last year of intervention. Data for Cohorts 1 and 2 were not entered into the analysis because the achievement tests designed to measure learning growth were not finalized until the middle of the 1999-2000 school year, toward the end of the intervention cycle for these cohorts. Although the tests were administered in Cohort 3 schools, the baseline tests were not administered until midway through the academic year, in March. Because this scheduling compromised the extent of learning growth that could be observed for the first year, the results were not comparable with the longitudinal data from Cohorts 4, 5, and 6. Results for the longitudinal study of pupil achievement growth are presented in Section 5.2. Data for the study of residual effects (e.g., exploratory investigations of any sustained impact on pupil learning) consisted of longitudinal data from Cohort 4 schools collected on five occasions: (1) baseline; (2) end of the first year of intervention; (3) end of the second intervention year; (4) one year after the withdrawal of interventions; and (5) two years after the withdrawal of interventions. In the analysis only schools where data were collected on all five occasions were included. This made it possible to compare performance in the QUIPS schools in two “out-years” to performance of QUIPS schools at the end of the two-year intervention. The performance difference or “gap” between QUIPS and controls could also be studied longitudinally, from the end of the QUIPS intervention cycle to the second year after. 5.1.4 Evidence for Matched QUIPS and Control Schools at Baseline. The longitudinal nature of the learning outcome studies, with pupils serving as their own controls, strengthens the validity of results in the face of subtle differences between treatment and control groups. Moreover, the QUIPS testing program set up rigorous selection criteria to minimize any differences between QUIPS and control schools. Comparative analyses between QUIPS and control schools at baseline indicate that the selection process was successful in yielding homogeneous entry-level achievement and demographics. No statistically significant differences between QUIPS and control schools were identified for any subject areas at baseline. Furthermore, no significant differences between pupils in QUIPS and control schools were identified for any of the key demographics: sex, age, parent literacy, or exposure to spoken English in the home. This situation held true for regional subdivisions, rural and urban school locations, and cohorts. Tables 5.1 through 5.5 and Figure 5.1 summarize baseline results for QUIPS and control schools. Table 5.1: Baseline School Mean Scores for QUIPS and Control Schools School Type Grade 5 Math Grade 5 English Reading Grade 5 Spoken English Narrative Grade 3 Math Grade 3 English Reading Grade 3 Spoken English Narrative QUIPS 36.69 35.08 56.35 35.78 29.14 33.40 Control 37.79 35.82 56.89 37.56 30.07 34.92 Total 37.20 35.42 56.60 36.61 29.58 34.13 * Percent correct scores Figure 5.1: Baseline Class Means for QUIPS and Control Schools Baseline Class Means QUIPS and Controls 0.00 10.00 20.00 30.00 40.00 50.00 60.00 P5 Math P5 English Reading P5 Spoken English P3 Math P3 English Reading P3 Spoken English Subject Areas Tested Class Means ( % Correct) QUIPS Control Table 5.2: Age of Pupils at Baseline: QUIPS and Control Schools Grade School Type N Mean SD Grade 3 QUIPS Control 3,605 2,732 10.23 10.15 1.86 1.81 Grade 5 QUIPS Control 3,162 2,445 12.16 12.10 2.36 2.42 Table 5.3: Sex of Pupils at Baseline: QUIPS and Control Schools Sex School Type Female Male Total QUIPS 3,042 45.1% 3,703 54.9% 6,745 100.0% Control 2,349 45.5% 2,808 54.5% 5,157 100.0% Total 5,391 45.3% 6,511 54.7% 11,902 100.0% 43 44 Table 5.4: Pupils’ Exposure to English: QUIPS and Control Schools Do people at home speak to you in English? School Type Yes No Total QUIPS 4,296 64.3% 2,386 35.7% 6,682 100.0% Control 3,165 62.0% 1,939 38.0% 5,104 100.0% Total 7,461 63.3% 4,325 36.7% 11,786 100.0% Table 5.5: Home Literacy: QUIPS and Control Schools Father reads at home School Type Yes No Total QUIPS 3,435 51.4% 3,252 48.6% 6,687 100.0% Control 2,590 50.5% 2,534 49.5% 5,124 100.0% Total 6,025 51.0% 5,786 49.0% 11,811 100.0% 5.1.5 Evidence for Matched QUIPS and Control Schools at Baseline. Differences in achievement growth in QUIPS and control schools during the period of direct QUIPS support and training were analyzed using HLM. The repeated measures ANCOVA, with schools being the unit of measurement, was used to explore longitudinal trends in the gap between QUIPS and control schools. The gap was measured at the end of the QUIPS intervention cycle and in the first two years after. A discussion and rationale for applying these analysis procedures is given at the end of this chapter in Note 5.1. 5.2 RESULTS OF THE ANALYSES OF PUPIL LEARNING GROWTH There is overwhelming evidence that QUIPS made a difference in pupil learning growth during the period of active QUIPS interventions. The findings were consistent across regions, urban and rural school locations, and cohort sub-samples of schools. Furthermore, the significant effect of the QUIPS treatment was maintained when controlling for competing pupil and school factors, such as pupil gender, exposure to spoken English, class size, and availability of texts. (See Chapter 9.) It is important to emphasize here that the hypotheses of interest are the slope coefficients representing learning growth rather than comparisons of static achievement at the end of the 24 month intervention cycle. As part of the HLM procedure (see Section 5.1.2 and Note 5.1), estimates of slope coefficients are generated for schools based on the actual pupil growth curves given by repeated testing of pupils at baseline, mid-cycle, and end of cycle. The slope coefficients represent the achievement growth for the school. Table 5.6 and Figure 5.2 present aggregated slope coefficients for QUIPS and control schools for each subject area tested: Grades 3-4 math; Grades 3-4 English reading; Grades 3-4 spoken English narrative; Grades 5-6 math; Grades 5-6 English reading; and Grades 5- 6 spoken English narrative. It can be seen in Table 5.6 that, with the exception of Grade 5 spoken English narrative, the learning growth curves for QUIPS schools were higher (i.e., steeper) than those for control schools. Table 5.6: Slope Coefficients for QUIPS and Control Schools. School Type Grade 5 Math* Grade 5 English Reading* Grade 5 Spoken English Grade 3 Math* Grade 3 English Reading* Grade 3 Spoken English* QUIPS 4.02 3.89 3.93 5.48 3.94 4.40 Control 3.39 3.33 3.24 4.79 3.14 3.59 Nation 3.73 3.63 3.61 5.15 3.56 4.02 *Significant group differences: p < 0.05 Figure 5.2: Learning Growth for QUIPS and Control Schools 5.2.1 Comparative Trends in Grade 3 Learning. Comparisons of the performance of Grade 3 pupils across two years of primary schooling showed that, on average, the rates of achievement growth as reflected in Table 5.6 and Figure 5.1 were higher for QUIPS schools compared to control schools for math, English reading, and spoken English narrative. For all subject areas, the differences were statistically significant.41 45 41 Using HLM, analysis of fixed effects: Grade 3 Mathematics (p<.001), Grade 3 English reading (p<.001); Grade 3 spoken English narrative (p<.03). Learning Growth Rates QUIPS and Controls 0.00 1.00 2.00 3.00 4.00 5.00 6.00 P5 Math P5 English Reading P5 Spoken English P3 Math P3 English Reading P3 Spoken English Subject Areas Tested Slope Coefficient QUIPS Control 46 This situation held true across urban and rural locations and across all three regions: northern Ghana, the middle region, and southern Ghana. Furthermore, the gains observed were constant even when Cohorts 4, 5, and 6 were analyzed separately. (See Section 5.2.5.) The performance for Grades 3-4 for QUIPS and control schools over time are presented in Table 5.7 and the means are plotted in Figures 5.3 to 5.5. Table 5.7: Summary of Grade 3 Class Means over Time: QUIPS and Control Schools School Type Baseline End Year 1 End Year 2 Grade 3 Mathematics Mean N SD Mean N SD Mean N SD QUIPS 36.35 130 10.54 50.43 130 11.94 64.69 130 11.39 Control 38.25 114 12.56 49.97 114 13.15 62.05 114 12.18 Total 37.24 244 11.54 50.22 244 12.50 63.46 244 11.81 Grade 3 English Mean N SD Mean N SD Mean N SD QUIPS 30.90 130 4.78 36.11 130 7.62 45.90 130 10.28 Control 32.26 113 6.70 36.09 113 8.51 43.60 113 11.29 Total 31.54 243 5.78 36.10 243 8.03 44.83 243 10.80 Grade 3 Spoken English Narrative Mean N SD Mean N SD Mean N SD QUIPS 19.22 124 7.08 26.14 124 6.48 28.01 124 7.15 Control 20.45 113 8.09 26.07 113 7.63 27.27 113 7.39 Total 19.81 237 7.59 26.11 237 7.04 27.66 237 7.26 Figures 5.2 to 5.4 present the actual pupil performance on all three occasions of testing. When inspecting these trends, it is important to consider differences in the relative steepness of the curve. It is the estimated slope coefficients for the linear curves that the statistical tests were based on. The relevance of impacting learning growth and the result with regard to static achievement differences gained by the end of the intervention cycle are discussed further in Section 5.2.2. Figure 5.3: Means Plots: Math from Grade 3 to Grade 4 P3 Math 30 35 40 45 50 55 60 65 70 Baseline End Year 1 End Year 2 Test Occasion Class Mean (% Correct) QUIPS Controls Figure 5.4: Means Plots: English Reading from Grade 3 to Grade 4 P3 English Reading 30 35 40 45 50 Baseline End Year 1 End Year 2 Test Occasion Class Mean (% Correct) QUIPS Controls Figure 5.5: Means Plots: Spoken English Narrative from Grade 3 to Grade 4 P3 Spoken English N 15 17 19 21 23 25 27 29 Baseline End Year 1 End Test Occasion Class Mean (% Correct) arrative Year 2 QUIPS Controls 47 5.2.2 Comparative Trends in Grade 5 Learning. Comparisons of the performance of Grade 5 pupils across two years of primary school showed that, on average, pupil achievement growth curves were higher for QUIPS than for control schools for math and English reading but not for spoken English. Again, the result was constant, even after controlling for such competing factors as location of school, region, cohort, and class size. The group means for Grades 5-6 are presented in Table 5.8 and plotted in Figures 5.6 to 5.8. Table 5.8: Summary of Grade 5 Class Means over Time: QUIPS and Control Schools School Type Baseline End Year 1 End Year 2 Grade 5 Mathematics Mean N SD Mean N SD Mean N SD QUIPS 36.72 129 10.62 48.11 129 11.86 57.18 129 12.31 Control 38.06 111 10.94 47.41 111 12.82 54.39 111 10.71 Total 37.34 240 10.76 47.79 240 12.30 55.89 240 11.66 Grade 5 English Mean N SD Mean N SD Mean N SD QUIPS 35.26 129 6.32 41.30 129 8.12 49.90 129 9.17 Control 36.13 111 7.13 41.04 111 9.01 47.72 111 9.71 Total 35.67 240 6.71 41.18 240 8.52 48.89 240 9.47 Grade 5 Spoken English Narrative Mean N SD Mean N SD Mean N SD QUIPS 35.69 125 11.53 42.04 125 8.78 43.80 125 8.36 Control 36.47 108 11.95 40.99 108 9.79 42.65 108 9.16 Total 36.06 233 11.71 41.55 233 9.26 43.27 233 8.74 Figure 5.6: Means Plots: Math from Grade 5 to Grade 6 P5 Math 30 35 40 45 50 55 60 Baseline End Year 1 End Year 2 Test Occasion Class Mean (% Correct) QUIPS Controls 48 Figure 5.7: Means Plots: English Reading Grade 5 to Grade 6 49 P5 English Readin 30 35 40 45 50 55 Baseline End Year 1 End Y Test Occasion Class Mean (% Correct) g ear 2 QUIPS Controls Figure 5.8: Means Plots: Spoken English Narrative Grade 5 to Grade 6 P5 Spoken English Narrative C 30 35 40 45 lass Mean (% Correct) QUIPS Controls Baseline End Year 1 End Year 2 Test Occasion 5.2.3 Relevance of the Shifts in the Learning Growth Curves of QUIPS Pupils. The finding that the QUIPS two-year interventions did result in a significant shift in the lthough significant, 42 achievement gains at the end of QUIPS training and support. Figure 5.9 presents the group means for achievement adjusted for subtle baseline differences43 between QUIPS and control schools. Final achievement gains of pupils in QUIPS compared to control schools were statistically significant (p<.001) for Grades 4 and 6 mathematics and English reading, but not for Grade 4 or 6 spoken English.44 learning growth curves, the “learning efficiency,” of pupils is important enough to deserve further clarification, especially considering what could be considered small, a 42 The ANCOVA statistical test is based on adjusted mean differences between QUIPS and control schools—mean comparisons after adjusting for any subtle differences across these groups that may have existed at baseline. 43 See Table 5.1 above for baseline aggregates for QUIPS and control schools. 44 Using the HLM procedure and repeated measures ANOVA, slope coefficients for the Grade 3 and 4 spoken English narrative learning curves were steeper and statistically significant for QUIPS as compared to control schools. 50 Figure 5.9: Differences in Final Achievement Results: QUIPS and Control Schools Adjusted Mean Differences at the End of the Two Year Intervention Cycle 0 10 20 30 40 50 60 70 QUIPS Control QUIPS 65.41 46.84 28.26 57.67 50.30 43.96 Control 61.27 42.56 27.01 53.86 47.27 42.48 P3 Math* P3 English P3 Spoken P5 Math* P5 English P5 Spoken Although gains in math and English reading were statistically significant, the achievement gains at the end of the two-year cycle are not large. However, one can get a better sense of how the QUIPS program affected learning by considering shifts in arning growth over time—looking at whether the school reforms and associated gains ains in the growth curves in the first place were sustained, a much more noteworthy . f reading of pupils have acquired at the end of Grades 3, 4, 5, and sheds some light on the problems teachers face and helps explain the small ifferences QUIPS achieved. le in learning growth were sustained. Figure 5.10 helps to illustrate the long run importance of positively impacting learning growth using a hypothetical case of a Grade 1 pupil. Although predicted gains in achievement at the end of two years (see Figure 5.10, “End Grade 2” gap) at first seem very small,45 if the education reforms (i.e., QUIPS good practices) that affected the g achievement advantage may be predicted for the end of six years of primary school 5.2.4 Explanation of Gains in Static Achievement. In spite of the statistically significant differences observed between QUIPS and control schools at the end of the two-year intervention, these differences are not large. The shifts in achievement in Grades 4 and 6 were approximately 3.2% in math, 2.2% in English reading and 1.2% in spoken English narrative. Information about class performance and the types o and math abilities the majority 6 d 45 Mean achievement differences, even at the end of two years, were statistically significant for Grade 4 and 6 English reading and Grade 4 and 6 mathematics (ANCOVA, with baseline entered as covariate, p<.001). 51 pothetical Group of Grade 1 Pupils, Predicted Means Plots for Six Years of Primary School 0% . The tables are based on the erformances of pupils on the QUIPS math and achievement tests. For example, pupils nt that upils in QUIPS and ontrol schools. It was apparent from the descriptive results that pupils in QUIPS st by t program. Most pupils had been achieving at two years behind age xpectations. In the absence of accelerated learning programs in basic numeracy and ing of Figure 5.10: Hy Pupils throughout Ghana, whether or not they attended a QUIPS school, are performing far below development expectations in math and English reading. A content review of the achievement test results for children completing Grades 3, 4, 5 and 6 underscores how serious children’s deficiencies are in basic numeracy and literacy. Tables 5.9 and 5.10 summarize overall performance and describe the types of skills the majority (5 or more pupils) of primary school pupils were able to use P3 Math: Predicted Outcome over Six Primary Classes given Sustained Quality Reforms 12 17 22 27 32 37 42 47 52 Be Primary School Class ginning P1 End P1 End P2 End P3 End P4 End P5 End P6 Raw Score QUIPS Control p who had completed Grade 4 were able to do Grade 3 mathematics items such as adding with carrying and multiplication without carrying. Tables 5.9 and 5.10 below underscore the serious delays in academic developme characterize Ghanaian primary school pupils. From the item analyses, it was difficult to identify specific math abilities that could differentiate between p c schools were better able to read with meaning than pupils in control schools, at lea the second year of intervention at the end of Grades 4 and 6. The narrowness of the gap between QUIPS and control schools by the end of the treatment period may be due to the severe delays in learning by public schools pupils a the beginning of the e literacy, these pupils cannot be expected to make appreciable gains within a two-year treatment period. Although teachers were exposed to some subject-based training to improve teach math and English, the training emphasis in QUIPS was on general child-centered 52 ouraged to teach to the grade-level syllabus even though children were nctioning two or more grades below grade level. Furthermore, many of the critical them n the form of supplementary readers or hildren’s books at home, they have little opportunity to practice reading or to apply co e school blackboard for memorization. : Summary of Performan nd Skill Clusters for Mathema s Class Completed wered G l ** practices and active learning, not on literacy and numeracy enhancement. Teachers were enc fu supports for literacy and numeracy learning were not available to the teachers and pupils. Textbooks rarely were available in classrooms and pupils were not allowed to take home to use. Teachers in any case need specific instruction in how to use textbooks and there is little evidence teachers, QUIPS-trained or not, have received it. Because children have few or no printed materials i c ding skills to new reading experiences other than the words and sentences written on th Table 5.9 ce Level a tic Skill Clusters (Items that 50% or more of the pupils ans correctly) rade Leve Grade 3 Basic concepts from Grade 1, no number lines or measurement Grade 2 addition and subtraction with no carrying or renaming Grades 2 to 4 story problems (no reading required) Grade 2 multiplication with no carrying involved Grade 2 – Grade 3 Grade 4 luding number lines but not but not subtraction with renaming (not at Basic concepts from Grade 2 and 3, inc Grade 3 measurement concepts Grade 3 addition with carrying, Grade 2 level) Grade 3 multiplication with no carrying Grade 3 division Grades 2 to 4 story problems Grade 5 ic figures n denominator Grade 3 to Grade 4 Basic concepts, lacking Grade 4- level geometr Addition with carrying Subtraction with renaming Multiplication, but no carrying Grade 4 addition problems involving decimals Unable to add or subtract fractions, even with commo Grades 2 to 4 story problems Grade 6 s, No addition and subtraction of fractions, even with common denominator Basic concepts lacking for Grades 4 to 5 geometric figures, percentage Grade 4 measurement Addition with carrying and subtraction with renaming Multiplication with carrying, but not Grade 3 multiplication with double digits Grade 4 addition problems involving decimals *Summaries observed from tests administered at the end of each hool year (June/July). **Levels based on items that 50% or more of the pupils answered rrectly. sc co 53 T 0: able 5.1 Summary of Performance Level and Skill Clusters for English Reading Class Level* Skill Clusters (Items that 50% or more of the pupils answered correctly) Grade Level ** Grade 3 picture support Grade 2 Single letter and word recognition at Grades 1 and 2 levels Match single words to pictures No reading of simple sentences, not even with Grade 1- Grade 4 ills of pupils in QUIPS versus s and in QUIPS Subtle and favorable differences in level and Grade 2 – control schools. More test items involving read sk ing of sentence passages at Grades 2 and 3 levels were completed by the majority schools compared to the control schools Neither group reading without picture support Vocabulary recognition from Grades 3 and 4 Grade 3 Grade 5 Match single words to pictures: Grades 3 and 4 vocabulary Word recognition from verbal stimulus: Grades 3 and 4 vocabulary Beginning reading Grades 2 and 3 level sentences without picture support Grade 3 Grade 6 ssages with and without Some vocabulary knowledge at Grade 5 level Grade 4 – Grade 5 Noticeable difference in level and skills of pupils in QUIPS versus control schools Most QUIPS pupils in Grade 6 reading short pa picture support, some taken from the Grade 6 curriculum Control school pupils were beginning to read simple sentences and short passages without picture support *Summaries observed from tests administered at the end of each school year (June/July). **Levels based on items that 50% or more of the pupils answered correctly One observation of the final evaluation was the advantage that relevance presents to pupils in mathematics. The item analyses in the tables show that when reading was not a confounding factor, pupils were able to complete math story problems at higher leve than they completed problems that were not presented in a real-life context. This underscores the importance of bringing pupils’ life experiences into all learning con 5.2.5 Performance Differences Related to Cohort Group. One of the pressing uestions coming out of the results recorded in the annual SO ls texts. 2 performance data ab sch l con c il learning by testing ifferent cohorts. H q t les had to do with the apparent closing of the gap between QUIPS and control oo s during the final years of the program. To investigate this question, the team du ted four supplemental analyses: • Repeated measures ANOVA, testing for differences in pup interactions between the cohort and linear trend (i.e., learning) effects. • ANCOVA, testing for differences in the final static achievement outcome for QUIPS and control schools relative to d • LM analyses of fixed effects of cohort on pupil learning. • ANCOVA, testing for differences in final static achievement across cohorts, whether in QUIPS or control schools. 54 ound for Grades 5-6 math and for Grades 3-4 and 5-6 poken English narrative. That is, for these groups and subjects, although analysis did in The bar graphs in Figures 5.12 to 5 ve difference in performance of UIPS and control schools by cohort. Table 5.11: Aggregated Slo C n on hools across Co Type G M Gra Eng Rea Grad Spo Eng Narr G M Gra Eng Rea Grad Spo Eng Narra 5.2.6 Cohort Comparisons of Pupil Learning Growth. Repeated measures ANOVA results showed no statistically significant cohort differences in the QUIPS/control gap with respect to pupil learning as measured by slope coefficients comparisons. However, for some subjects there were significant differences between cohorts in the learning curves. Significant cohort differences in pupil learning in QUIPS and in control schools were f s not identify any significant differences in the gap between QUIPS and control schools pupil learning during the intervention, across QUIPS and control schools, it did identify differences across cohorts. Table 5.11 presents the slope coefficients disaggregated by school type and cohort. .15 show the relati Q pe Coefficients ompari horts g QUIPS and C trol Sc Cohort School rade 5 ath de 5 lish ding e 5 ken lish ative rade 3 ath de 3 lish ding e 3 ken lish tive QUIPS 3.86 3.92 3.41 5.69 4.02 4.63 Cohort 4 Control 3.42 3.52 4.05 4.72 2.94 3.82 QUIPS 4.49 3.82 3.45 5.60 3.72 2.65 Cohort 5 Control 3.69 3.14 1.61 4.98 3.15 1.61 QUIPS 3.72 3.95 4.95 5.15 4.05 5.68 Cohort 6 Control 3.13 3.30 3.80 4.69 3.31 4.92 Figure 5.11: Gap in Pupil Learning for Grade 3 Mathematics by Cohort : Gap in Pupil Learning for Grade 5 Mathematics by Cohort Figure 5.13: Gap in Pupil Learning for Grade 3 English Reading by Cohort Figure 5.14: Gap in Pupil Learning for Grade 5 English Reading by Cohort Figure 5.12 55 56 Figure 5.15: Gap in Pupil Learning for Grade 3 Spoken English by Cohort Figure 5.16: Gap in Pupil Learning for Grade 5 Spoken English by Cohort etter than the counterpart groups in Cohorts 4 and 6. The degree of hown in s It can be seen in Figure 5.12 that both the QUIPS and control schools in Cohort 5 performed b difference in Cohort 5 performance is not large, however. In effect, the results s the bar graph are difficult to interpret. Further study is necessary to better grasp the relevance of these comparisons, especially given the variety of contextual factor present.46 46 Using HLM analysis of fixed effects for cohort, controlling for urban and rural, regional, and school differences, the results were somewhat different. Significant cohort effects for Grades 5- 6 math were not identified, though an effect for Grades 3- 4 English was. Similar results were found for spoken English for Grades 3- 4 but not for Grades 5- 6. 57 wer for both Grades 3-4 and Grades 5-6, although the ap between the QUIPS and the control schools was larger (although not statistically a he ificant cohort differences in e QUIPS/control gap with respect to final achievement measured at the end of Grades ken nglish narrative, however, there were statistically significant cohort differences in ontrol schools in each of the two years following the withdrawal of direct QUIPS ontrolled (“equalized”) and the statistics tests are conducted on adjusted means over rol -year uggest noteworthy trends, which varied by grade level and subject area. Figures 5.1 In looking at the performance of QUIPS and control schools on spoken English narrative, it can be seen that more learning (as shown by the steepness of the slope) took place in Cohort 6 than in Cohorts 4 or 5. (See Figures 5.14 and 5.15) The change—the learning—was lo g signific nt) for Cohort 5. For Cohort 4 there was little difference between QUIPS and control schools. In Grades 5-6 the control school slope coefficients were higher than those of the QUIPS schools. 5.2.7 Cohort Comparisons on Static Achievement Measured at the End of t Intervention. ANCOVA results showed no statistically sign th 4 and 6 for any subject. For Grade 6 English reading and Grades 4 and 6 spo E achievement, whether the schools were QUIPS or control. 5.3 ANALYSIS OF ACHIEVEMENT AFTER THE WITHDRAWAL OF ACTIVE QUIPS Repeated-measures ANCOVA was applied to study how well the achievement advantage for pupils attending QUIPS schools was sustained. Data on math and English reading were collected from Grades 4 and 6 in a small sample of QUIPS and c support. Data from Cohort 4 QUIPS and control schools were used to investigate sustained effects on pupil learning. The Grade 3 sample consisted of 34 schools (18 QUIPS and 16 control). The Grade 5 totaled 47 schools (32 QUIPS and 15 control). In the repeated measures ANCOVA, group differences in achievement at baseline are c the remaining years. After controlling for baseline differences, the QUIPS and cont schools were compared longitudinally from Test Occasion 2, the end of the first year of the intervention period, to Test Occasion 5, two years after the withdrawal of QUIPS. ANCOVA results failed to identify any significant differences between QUIPS and control schools, but the mean plots from Test Occasion 2 through the second out 47 s through 5.20 show the trends from mid-cycle, Test Occasion 2, through the second out￾year. The figures present plots of the adjusted class means—the aggregated means after controlling for differences between QUIPS and control schools at baseline. 47 A number of factors may relate to the lack of significance in the between-groups effect: It may be, e.g., that the result is correct and there is no gap between QUIPS and control schools after controlling for baseline differences, or there may be increased error variance in the data, or there may be reduced power because the small numbers of schools in each group reduces the sensitivity of the statistical tests. 58 en the intervention (when e pupils were finishing the upper grade). The means plotted in Figures 5.19 and 5.20 re These are pupil nd years after e two-year intervention cycle. (Baseline results are not plotted in these figures ecause the scores were entered into the analysis as the covariate.) Figure 5.17: Adjusted Class Means for Grade 3 English from Mid-cycle through the Second Out-year for QUIPS and Control Schools Figure 5.18: Adjusted Class Means for Grade 3 Mathematics from Mid-cycle through the Second Out-year for QUIPS and Control Schools The first two means plotted in the figures are results from the Grades 3-4 and 5-6 tests administered to the same pupils at the end of the first year of the intervention (wh pupils were still in the lower grade) and the end of the second year of th present results from the Grade 4 or 6 tests administered to a new sample of pupils. s in Grade 4 and 6 in the out-years, i.e., the first and seco th b 59 Figure 5.19: Adjusted Class Means for Grade 5 English from Mid-cycle through the Second Out-year for QUIPS and Control Schools Figure 5.20: Adjusted Class Means for Grade 5 Mathematics from Mid-cycle through the Second Out-year for QUIPS and Control Schools 60 ortant 4 pupils tested in the two years after QUIPS were not those ut￾e a sustained gap in performance between QUIPS and control schools at e read. QUIPS and control ls e ear tests had been tested in Grade 3 during the QUIPS cycle. In any case, all pupils in ear e ty, with a 5.3.1 Trends Observed from Plots of the Grades 3-4 Class Means. It is imp to note that the Grade tested during the QUIPS intervention. In the first year after QUIPS, the Grades 3-4 population had already moved on to Grade 5. The Grade 4 pupils tested in the first o year had been in Grade 2 at the beginning of the QUIPS intervention. Those tested in the second out-year had been in Grade 1. However, all pupils in QUIPS schools potentially benefited, not just those who were repeatedly testing during the active QUIPS interventions. In principle, the early years of schooling provide a foundation for later academic development. If the QUIPS interventions did increase rates of achievement growth, there should b least for as long as the pupils sampled in the out-year testing had been present also during the QUIPS interventions. However, other factors could narrow the gap. First, some of the effects of QUIPS on pupil learning might attenuate in the years after th withdrawal of active intervention. Second, if the spread of the QUIPS interventions and the harmonization of materials were successful, effects on learning might become more widesp Inspection of the Grades 3-4 math means plots (Figure 5.18) suggest a sustained gap between QUIPS and control schools at the end of the intervention. Grade 4 class performance after QUIPS showed that the relative gap between schoo was maintained, although a gradual reduction was observed in the second out￾year. Grades 3-4 English showed an increase in performance of the school one year after th intervention cycle; Grade 4 math performance dropped in the second out-year, but the drop in performance for control schools was more substantial. 5.3.2 Trends Observed from Plots of the Grades 5-6 Means. The Grade 6 pupils who participated in the first out-year testing had been in Grade 4 at the end of the QUIPS intervention. However, the Grade 6 pupils who participated in the second out￾y QUIPS schools at the time of the active intervention potentially benefited from the program. Thus, one would predict a sustained gap in performance between QUIPS and control schools at least for as long as pupils sampled in the out-year testing had been present during the QUIPS interventions. The factors discussed in the previous section as possibly narrowing the gap, however, are just as likely to be present in this group. Interestingly, despite the difference in the learning curve for Grades 5 and 6 that favored the QUIPS schools, after controlling for group baseline differences there did not app to be any difference in the group means for math at the end of the QUIPS intervention shown at Test Occasion 2 (Grade 6, end year 2). The two years following show a gentl rise in performance. For Grades 5-6 math, the control schools had less continui fall in the first out-year and a steep rise in the second out-year. 61 in chieving a shift in the rates of pupil learning is sustained, with no appreciable rates among the comparison group, the gap may be expected to iden over time. Although the Grade 6 performance of both QUIPS and control schools If the plot of the Grades 5 and 6 English test is truly representative of the population of QUIPS and control schools (which may be questionable, given the small sample of schools tested in the out-years), this graph represents a potential longitudinal benefit from positively impacting the rates of pupil learning growth. That is, if an advantage a difference in learning w shows improvements from the first to the second out-year testing, the plot of QUIPS school means reflected steeper learning growth curves. This produced a greater gap between QUIPS and controls after the second out-year testing than after the first. 5.4 CONCLUSIONS It appears that the training and support QUIPS provided during the two-year intervention positively shifted the rates of learning for those pupils enrolled in QUIPS compared to control schools. The results given by the analyses of fixed effects (e.g., school type) using the HLM procedure are consistently significant. They were reinforced through repeated measures of ANOVA. The final shifts in achievement measured through group comparisons between QUIPS and control schools were also significant, favoring the QUIPS schools. Inspection of the gains in achievement between QUIPS and control schools at the end of the two-year cycle are not necessarily impressive but it is important to consider the long-term benefits of affecting the pupil learning curve in programs interested in enhancing school effectiveness or school quality. If the targeted school quality improvement (e.g., classroom reform, school management improvement, or community involvement) that affected pupil learning in the first place were to be sustained, there should be a greater shift in the final achievement differences over time. (See Section 5.2.3 and Figure 5.1.) Section 5.3 gives evidence of the potential long-term effects that can be obtained when development programs attend to shifts in pupil learning curves, underscoring the importance of making such shifts a part of program monitoring and evaluation systems. The USAID Mission understandably has been concerned about the limited achievement gains demonstrated at the end of QUIPS, and it is clear from an understanding of what primary school children are able to do in math and English reading that there is reason for concern. (See Section 5.2.4.) Yet with delays like those in Ghana today, pupils cannot be expected to make appreciable gains within a two-year intervention period in the absence of accelerated learning programs in basic numeracy and literacy. Although teachers were exposed to some subject-based pedagogy, the emphasis was primarily on general child-centered practices and active learning, rather than on literacy and numeracy enhancement. Teachers are encouraged to teach to the class-level 62 syllabus even though children are functioning two or more grades below level. Furthermore, the country lacks many critical resources for literacy and numeracy learning: Textbooks were rarely available in the classrooms and pupils could not take books home to use. In any case, there is little evidence that QUIPS-trained or other teachers have received the specific instruction they need in how to use textbooks.48 Because there are few or no printed materials, whether supplementary readers or children’s books at home, children have little opportunity to practice reading or apply coding skills to new reading experiences. Relevance presents a distinct advantage in mathematics. When reading is not a confounding factor, pupils are able to complete more story problems than other types of math problems that are not presented in a real life context. It is important to bring the life experiences of pupils into all learning contexts. Note 5.1. Education systems have a hierarchical structure, also called a multilevel or nested structure—pupils are “nested” within grades, which are nested in schools, which are nested in districts or regions. The educational processes occurring within each level of the structure, such as the classroom, ultimately influence learners' achievement and interaction with educational processes at other levels, such as school processes that influence teachers. Conventional analysis methods, such as ANCOVA techniques, are based on the assumption of independence—the assumption that the dependent measures are uncorrelated. This assumption is violated where individual data from pupils in the same grade are entered into the analysis without first aggregating at the grade or school level. The assumption of independence, however, is tenable when the grade or school mean is the unit of measurement rather than individual scores. Violations of the assumption of independence are avoided by aggregating data, using grade or school means as the unit of measurement rather than individual scores. However, aggregation raises the risk of losing important pupil information such as sex and age which cannot be integrated with school information, such as location or type (e.g., QUIPS or control). Most important, conventional methods are not as informative in telling us why some schools are more effective than others. That is why a mixed model, hierarchical linear modeling (HLM), is the analysis of choice when attempting to understand factors related to school learning, although repeated measures ANOVA and ANCOVA are still excellent procedures for analyzing fixed effects, such as comparisons across treatment and control schools. 48 In its review, the Academy of Educational Development commented: “QUIPS in-service training did indeed include instruction in how to use the math and English textbooks. In fact, these texts were brought by the teachers to the training sessions, and when not available to teachers were provided by the project. The observation by the evaluators that they found that the texts were not properly used is an important but separate matter.” 63 CHAPTER 6: SCHOOL-BASED OUTCOMES OF QUIPS This chapter reviews the main school-based outcomes the QUIPS evaluation team observed. 6.1 KEY INPUTS AND INTERVENTION STRATEGIES OF QUIPS QUIPS school-based interventions emphasized four areas of training and support: • Training and on-site support to teachers in preparing lesson notes and teaching aids, classroom management, record keeping, and instructional techniques; • Training and on-site support to head teachers in the area of teacher supervision, staff management, and school administration and financial management; • Construction of new classroom blocks, or other facilities where new classrooms were not needed; and • Provision of micro-grants to schools and communities to support priority community school-improvement projects. 6.2 INFLUENCE OF QUIPS ON SCHOOL ENROLLMENTS Although improved access and increased enrollments were not a targeted result of the QUIPS program, improvements in the quality of education are known to be associated with increased enrollments, which were indeed one outcome of the QUIPS program, particularly for girls. Although increased enrollments are generally viewed as favorable, the evaluative field work suggests that they present systemic problems for Ghana. Reportedly, because QUIPS partnership schools quickly developed a reputation for being “good” schools, parents in surrounding communities began transferring their children out of nearby schools and into QUIPS schools. This produced an imbalance in enrollment distribution in communities adjacent to QUIPS schools. As enrollment in surrounding schools was depleted, the enthusiasm and general support for education in nearby schools was attenuated. Educationalists throughout Ghana considered this imbalance favoring the novel and “very exciting” QUIPS program a negative outcome of QUIPS. It introduced unforeseen problems in the delivery and administration of local and district primary schooling. District personnel in particular reported that the costs of implementing the QUIPS program in terms of level of effort, staffing, and travel compromised the support they were able to provide to non-QUIPS schools and communities. (See Chapter 8.) 64 Tables 6.1 and 6.2 present data collected from EMIS data in the year immediately after the QUIPS program ended.49 After two years of interventions, both total enrollments and the percentage of girls enrolled were higher for QUIPS than for control schools (ANOVA results showed statistical significance, p< 0.05). The gap in enrollments was observed in northern, middle, and southern Ghana and in both urban and rural locations; it was accentuated slightly in middle Ghana. Table 6.1: Enrollments in QUIPS and Control Schools by Region School Type Regional Division Total School Enrollment* Percentage of Girls** Mean N Standard Deviation Mean N Standard Deviation QUIPS South 227.71 85 94.42 47.88 85 7.43 Middle 247.96 75 121.73 48.19 75 3.95 North 330.13 61 151.51 46.44 61 7.13 National 262.85 221 128.29 47.59 221 6.38 Control South 191.02 44 79.26 46.04 44 6.07 Middle 199.29 31 137.89 46.50 31 4.46 North 280.19 27 176.00 45.88 27 5.62 National 217.14 102 133.24 46.14 102 5.46 *Significant differences (ANOVA): p = 0.004. ** Significant differences (ANOVA): p = 0.048. Table 6.2: Enrollments in QUIPS and Control Schools by Location School Type School Location Total School Enrollment* Percentage of Girls** Mean N Standard Deviation Mean N Standard Deviation QUIPS Rural 235.59 142 108.75 46.57 142 5.18 Urban 311.85 79 145.79 49.42 79 7.81 Nation 262.85 221 128.29 47.59 221 6.38 Control Rural 202.64 72 123.90 45.50 72 5.35 Urban 251.93 30 149.91 47.68 30 5.51 Nation 217.14 102 133.24 46.14 102 5.46 *Significant differences (ANOVA): p = 0.026. ** Significant differences (ANOVA): p = 0.012. 49 For example, interventions in Cohort 4 schools were completed in July 2002. Therefore, the National EMIS data used for Cohort 4 schools was taken from the national primary school census data collected at the beginning of the 2002/2003 school year. By using national census data taken from the school year immediately following the withdrawal of active QUIPS interventions and applying the rule to all cohorts, references pertaining to school demographics are consistent for all QUIPS and control schools. 65 6.3 TEACHERS AS INPUTS TO QUIPS Tables 6.3 and 6.4 present a comparative view of teacher staffing in QUIPS and control schools. QUIPS schools had a higher percentage of trained teachers and a lower pupil￾to-teacher ratio. (ANOVA results showed statistical significance, p< 0.05.) This discrepancy was consistent throughout Ghana. Table 6.3: Teacher Staffing in QUIPS and Control Schools across Regions School Type Region Total Teachers* Percentage of Trained Teachers** Pupil Teacher Ratio Mean N Standard Deviation Mean N Standard Deviation Mean N Standard Deviation QUIPS South 6.74 85 2.41 93.39 85 14.99 34.55 85 10.90 Middle Belt 7.09 75 2.33 84.68 75 24.50 34.68 75 10.18 North 6.98 61 2.78 68.25 61 22.35 50.48 61 23.90 Nation 6.93 221 2.49 83.49 221 22.95 38.99 221 16.93 Control South 6.32 44 2.10 82.94 44 27.49 30.92 44 9.91 Middle Belt 6.55 31 2.66 77.51 31 24.82 29.86 31 8.91 North 5.33 27 2.76 57.90 27 31.81 58.23 27 33.16 Nation 6.13 102 2.48 74.66 102 29.54 37.83 102 22.36 *Significant (ANOVA) mean differences between QUIPS and Controls for total teachers, p = .008 **Significant (ANOVA) mean differences between QUIPS and Controls for percentage trained, p = .004. Table 6.4: Teacher Staffing in QUIPS and Control Schools: Urban and Rural Settings School Type Region Total Teachers* Percentage of Trained Teachers** Pupil/Teacher Ratio Mean N Standard Deviation Mean N Standard Deviation Mean N Standard Deviation QUIPS Rural 6.33 142 2.28 82.28 142 23.17 38.99 142 18.40 Urban 8.00 79 2.50 85.67 79 22.53 39.00 79 14.00 Nation 6.93 221 2.49 83.49 221 22.95 38.99 221 16.93 Control Rural 5.63 72 2.11 70.61 72 31.03 39.01 72 24.78 Urban 7.33 30 2.92 84.38 30 23.31 34.99 30 15.04 Nation 6.13 102 2.48 74.66 102 29.54 37.83 102 22.36 *Significant (ANOVA) mean differences between QUIPS and controls for total teachers, p = .008 **Significant (ANOVA) mean differences between QUIPS and controls for percentage trained, p = .004 The QUIPS teacher in-service training program involved substantial training delivered on-site at the schools, so there was a need for them to be sufficiently staffed during the 66 intervention cycle.50 School-level interventions emphasized teacher training and general classroom reforms within a somewhat contrived schooling context where all teachers were available. As a result, sustainability of the school-level outcomes was to a large extent dependent on the availability of teachers trained and actively implementing QUIPS targeted classroom reforms. The evaluation study results discussed below reinforce this conclusion. Where QUIPS teachers were no longer available in schools, neither were QUIPS targeted instructional practices. Improvements in instruction cannot be expected to be sustained without attempts to reform systems, not only in relation to the targeted instructional practices but in formulating policy for improving the conditions and thus the integrity of the teaching profession, including remuneration. 6.4 CLASSROOM TEACHING OUTCOMES OF QUIPS INTERVENTIONS One approach to promoting teacher growth that was used as part of in-service teacher training was to conduct formative assessments of their classroom performance, stimulating them to reflect on their own teaching as it related to targeted classroom behaviors. The QUIPS Classroom Observation Instrument (COI) was the basis for this regular formative assessment. Data collected at the end of the intervention were recorded, archived, and available for analysis in the current evaluation.51 For Cohort 4 schools, there were also baseline data allowing for comparison of teacher performance from baseline to the end of the two-year QUIPS cycle. The Cohort 4 data provide information related to 14 targeted classroom practices: • Minimum use of time for non-teaching activities in classroom; • Materials used by pupils; • Teacher questioning; • Pupil questioning; • Stimulating critical and creative thinking; • Discipline in classroom; • Strategic grouping; • Learning behavior in groups; • Use of a variety of teaching methods; • Opportunities provided to observe pupils’ abilities; • Interaction in the classroom; • Use of feedback; • Encouraging participation of girls; and • Encouraging participation of all pupils. 50 This criterion was not always met in very remote communities, particularly in the north. 51 Limitations of this data include the fact that it was often collected by people who were directly involved in the training of teachers themselves and may have been biased. 67 Four quantitative analyses investigated teacher in-service training and support outcomes using COI data collected in the classrooms of Cohorts 4, 5, and 6 schools across the nation. The first analysis compared shifts in teacher performance on each of the above targeted instructional behaviors at baseline and again at the end of the QUIPS cycle. The second analysis compared performance on the 14 targeted teacher practices by rural and urban location. The third did the same analysis for regions of the country.52 The fourth analysis explored the relationship of each of the 14 practices to pupil-learning outcomes measured during the QUIPS cycle. 6.4.1 Analysis 1: Shifts in Targeted Instructional Practices. Analysis 1 investigated the extent to which teacher practices improved from baseline to the end of the two-year cycle. As teacher baseline data were available only for Cohort 4, the sample for this analysis was Cohort 4 teachers. Cross-tabulations and related chi￾square statistical tests demonstrated a positive and statistically significant shift in the performance of teachers by the end of the QUIPS cycle. Significant teacher change was demonstrated on all 14 targeted classroom “good practices” measured through COI. This first analysis suggests that the Cohort 4 teachers as a group responded positively to the in-service training provided by QUIPS. Consequently, it was reported that the trained teachers used more of the 14 classroom practices, which would be expected to be a more efficient form of instruction. These applications would then be expected to lead to increased motivation and learning by pupils. 6.4.2 Analysis 2: Exploring Rural and Urban Differences in Teacher Instructional Practices. Data from teachers of Cohorts 4, 5, and 6 who were observed at the end of the cycle were entered for cross-tabulation and related statistical analyses of teacher classroom practices by rural and urban area. The results showed a difference favoring the teacher located in an urban area on 12 of the 14 items. A few more rural teachers in the sample responded that they encouraged girl pupils to participate in class more than the urban teachers responded. In sum, the reported application of the learning techniques was confirmed by the findings and analysis. 6.4.3 Analysis 3: Exploring Regional Differences in Teacher Instructional Practices. Data from teachers in the same Cohorts 4, 5, and 6 who were observed in classrooms were analyzed to see differences in their teaching practice over the different regions of the country. The results identified a consistent regional difference for 13 of the 14 target behaviors (all except pupils’ use of questions). The performance of teachers in the north was lower than that of teachers in middle and southern Ghana, and the observed differences were statistically significant. 52 North: Northern, Upper East, and Upper West; Middle: Brong Ahafo, Ashanti, and Western; South: Greater Accra, Central, Volta, and Eastern. 68 The explanation given during the fieldwork in the north was that the basic education level of a number of QUIPS teachers limited what they absorbed during the series of in￾service training sessions. Untrained teachers on contract were reported to have been allocated by the GES to the QUIPS schools. Their training along side the Certificate A teachers became problematic. Eventually, CRS organized teacher in-service courses purposely for the untrained teachers in an effort to increase their basic skill levels as well as introduce them to improved methods of teaching. The lower level of use of QUIPS classroom practices in the north may be attributable to the role of the untrained teachers. It suggests that training in pedagogy needs to be built upon a solid academic foundation and that the in-service training needs adequate hands-on practice teaching. It suggests also that implementers need to be very quick in making alterations in their program plans in the face of unexpected gaps discovered during implementation. Earlier efforts to deal explicitly with untrained teachers in QUIPS schools might have led to a level of use of classroom practices in the north equal to the rest of the country. 6.4.4 Analysis 4: Exploring the Relationship of Teacher Practices and Pupil Learning Growth. Data from Cohorts 4, 5, and 6 teachers observed at the end of the intervention cycle were entered for cross-tabulation and related statistical analyses of how teacher classroom practices correlate with a grouping variable related to pupil learning growth.53 Considering the regional differences already mentioned in the responsiveness of teachers to the QUIPS in-service training, there was concern that there might also be an imbalance in the distribution of schools in the high, medium and low learning groups. That is, more schools in middle and southern Ghana would demonstrate high learning growth and more schools in northern Ghana would fall into the low learning group. The result given by cross-tabulations of region by learning group revealed no such imbalances. There were no significant regional differences (p > .178) in the distribution of high, medium, and low-performing schools. (See Table 6.5 below for details.) This relative balance among the regions provided some assurance that the association between instructional practices and pupils' learning was not confounded by regional differences in teacher practices themselves. Yet the evaluation team decided that analyses questions about this relation should be analyzed separately for the north as one group and for the Middle Belt and South as a second combined group for both the quantitative analysis and the qualitative field work. The reason was that how teacher factors affect the learning of pupils might very well be different in the two groups. 53 Pupil learning was measured from a calculation of the slope coefficients of the learning growth curves for pupils in Grade 3, tested repeatedly through Grade 4, and pupils in Grade 5, tested repeatedly through Grade 5. Individual slope coefficients were then aggregated at the school level and based on the aggregated slope coefficients, schools were assigned to high, medium, and low “learning groups.” 69 In fact, 7 of the 14 previously listed teacher instructional practices showed a significant relation to pupil learning, but only one was common to northern as well as southern and middle Ghana. That practice was "encouragement of girls to participate in class.” Tables 6.6 and 6.7 give the results for this practice across the regions; Figures 6.1 and 6.2 present the data graphically. Table 6.5: Distribution of Pupil Learning Groups across Southern, Middle and Northern Ghana Learning Group Region Low Medium High Total South 15 18.8% 43 53.8% 22 27.5% 80 100.0% Middle Belt 26 30.6% 35 41.2% 24 28.2% 85 100.0% North 24 27.9% 33 38.4% 29 33.7% 86 100.0% Total 65 25.9% 111 44.2% 75 29.9% 251 100.0% Table 6.6: For Northern Ghana, “Encouraging Girls to Participate”* Encouraging Girls to Participate (North) Region Neither encouraged nor discouraged Girls and boys given equal encouragement Girls given special encouragement Total Low Group 10 20.4% 39 79.6% 0 0% 49 100.0% Medium Group 14 10.6% 113 85.6% 5 3.8% 132 100.0% High Group 7 6.6% 92 86.8% 7 6.6% 106 100.0% Total 31 10.8% 244 85.0% 12 4.2% 287 100.0% *Significant association (chi-square, p = 0.045). Table 6.7: For the South and Middle Belt Combined, “Encouraging Girls to Participate” by Learning Group* Encouraging Girls to Participate (North) Region Neither encouraged nor discouraged Girls and boys given equal encouragement Girls given special encouragement Total Low Group 4 5.3% 72 94.7% 0 0% 76 100.0% Medium Group 2 1.9% 103 96.3% 2 1.9% 107 100.0% High Group 3 1.1% 247 93.6% 14 5.3% 264 100.0% Total 9 2.0% 422 94.4% 16 3.6% 447 100.0% *Significant association (chi-square, p = 0.028) Figure 6.1: “Encouraging Girls to Participate” (North) Encouraging Girls to Participate (North) 0 10 20 30 40 50 60 70 80 90 100 Low Group Medium Group High Group Learning Group Percent Neither encouraged nor discouraged Girls and boys given equal encouragement Girls given special encouragement Figure 6.2: “Encouraging Girls to Participate” (South and Middle) Encouraging Girls to Participate (South, Middle Belt) 0 20 40 60 80 100 120 Low Group Medium Group High Group Learning Group P e r cen t Neither encouraged nor discouraged Girls and boys given equal encouragement Girls given special encouragement 6.4.5 Summary of Findings Based on Results from the Classroom Observation Instrument (COI). The quantitative analyses of teacher outcomes were based on evaluations of teacher practices observed at the end of the two-year intervention period; these are not sustained outcomes. Sustained school impact as reflected in the evaluative field work is presented in the remaining sections of this chapter. Where QUIPS teachers were still present in school, those in high-performing schools in general demonstrated many of the teaching practices identified from COI results as being associated with learning. That is, they were observed to encourage pupils and use good questioning techniques. Their classrooms were “friendlier” and there were 70 71 more interactions observed between teachers and children—even if they were mostly teacher-initiated interactions. The following points summarize the quantitative findings drawn from the classroom observation instrument (COI) results: • Teachers were responsive to the in-service training, consistently showing meaningful (and statistically significant) improvements in teaching practices from baseline to the end of the intervention cycle. • Improvement in teaching practices was smaller for teachers in the north than in the south and middle of the country. • Six teaching practices emphasized in the in-service training program were identified as significantly related to improved pupil learning during QUIPS: (1) special encouragement of girls’ participation in class; (2) quality of materials used; (3) teacher questioning; (4) facilitation of critical and creative thinking; (5) interaction in the classroom; and (6) encouragement of all pupils to participate in class. 6.5 RESULTS FROM QUALITATIVE FIELD WORK: INTRODUCTION The main findings from the evaluation team’s field work are organized by the themes that emerged from the study: infrastructure, textbooks and supplementary readers, availability of teachers, teacher in-service training, teacher practices, head teacher and district education office support, and observed teacher performance. These themes are discussed in relation to findings from field visits in the 18 school-communities selected. (See Section 5.3.) 6.6 INFRASTRUCTURE This section of the analysis describes school buildings and the state of furniture, toilet and urinal facilities for the periods before, during, and after QUIPS. (See Table 6.14 for a summary of their completion level at the end of the intervention.) 6.6.1 Classrooms. The condition of schools before the QUIPS program was not good. (See Annex 7.) The school infrastructure was made up of old dilapidated buildings with leaky roofs, broken windows, and doors and uncladded or uncovered classroom pavilions that required schools to close any time there was rain. In the north, only one out of eight sampled schools had the full complement of classrooms but it also had a leaky and partially ripped off roof. The other seven schools in the north had uncladded pavilion structures and classrooms made of mud bricks. In the middle of the country, schools visited in the sample had managed to construct six-classroom units, but all were in a bad state. Some schools in the south, QUIPS and control, had old three-unit classroom blocks that had not been renovated for a long time. 72 The evaluative fieldwork found that QUIPS provided financing for all its partnership schools to help put up new classroom blocks or renovate old ones. However, in some cases particularly in the north, communities were unable to finish constructing classrooms due to lack of financial and organizational support from the community. For instance, Nawuni (Q-Lo-N)54 could not support the classroom project, so the block remained incomplete with sandy floors and no roofing. The three-unit classroom block built by the Catholic Church houses the entire school from Grades 1 through 6. Infrastructure development by other donors was evident across the country, particularly in the north. For instance, the QUIPS comparison schools received support for the provision of classroom blocks from District Assemblies, the African Development Bank (ADB), NGOs, and religious organizations; these organizations also supported all but one of the control schools in the north. All the QUIPS schools investigated in the south and middle belts were able to erect three- to six-unit classroom blocks. The control schools in these areas had no support from QUIPS or any other source to construct new classroom blocks. Several control schools (Ntonso, Kunsu-Dotiem, and Nyamebekyere) are still functioning in dilapidated buildings with inadequate classroom blocks.55 The evaluative fieldwork found that school infrastructure across the four regions improved tremendously during the QUIPS program within both types of schools. All the QUIPS schools across the regions were provided with infrastructure to support their improvement. Only in Nawuni is the school building not completed. Comparisons failed to identify any systematic relationship between infrastructure and the overall performance of pupils in a school. Table 6.14 shows that while most of the high- performing QUIPS schools had a full complement of classroom blocks, some high￾performing control schools had limited facilities.56 Table 6.14 also reveals that many schools categorized as low- and medium-performing had enough classrooms. Meanwhile, Nyamebekyere (C-Hi-S), had a three-unit classroom block for over 10 levels of education from kindergarten 1 through junior secondary school 3 (KG1 – JSS 3). 6.6.2 Furniture. Basic furniture was supplied to most QUIPS schools through micro￾grants to be used for school quality improvement. According to all the field reports, the furniture situation before the QUIPS program was very poor. For instance in several communities in the north it was reported that, prior to QUIPS, children were lying on their bellies to read and write (e.g., Lonto Presby, Q-Hi-N). In southern and middle Ghana, some children had to carry their own furniture to school (Ntiribuoho, Q-Hi-M). 54 The following shorthand is used to characterize the schools: QUIPS or control schools (Q or C); high performing or low performing school (Hi or Lo); and location in the south, middle or north of Ghana (S, M or N). 55 Ntonso (C-Hi-M) still housed still in an old 6-unit classroom block with a leaky roof and broken doors and windows. The SMC/PTA has, however, started renovating. All the classes, KG1-JSS3, at Nyamebekyere (C-Hi-S) share an old three-unit classroom block. 56 Gbangu-Bongbini (C-Hi-N) has a three-unit classroom block. 73 Nevertheless, most of the stakeholders interviewed and observations of the classrooms indicated that QUIPS schools had an adequate supply of dual desks after the program ended. The furniture situation appears to have improved tremendously during and after the QUIPS programs in all schools in all parts of Ghana, QUIPS and control, rural and urban, and high- and low-performing schools. Dual desks now are available in all schools. Unfortunately, dual desks are not the best design for participatory child￾centered approaches to learning where desks need to be moved easily into small groups. More consultation with QUIPS implementing agencies may have helped communities recognize different furniture options and ensured that furniture more appropriate to child-centered participatory approaches (e.g. single chairs and tables) could have been adopted in order to improve learning outcomes. 6.6.3 Toilets and Sanitation. Before QUIPS few schools had toilet and urinal facilities; the sampled schools instead had improvised bamboo/wooden structures and unroofed pits. The evaluation teams found no significant differences between QUIPS and control schools in relation to toilet facilities; most still lacked basic toilet facilities. In only two QUIPS schools, both in southern Ghana (Ntiribuoho, Q-Hi-M and Dunkwa Q￾Hi-S) the communities had constructed toilet facilities with the support of QUIPS or NGOs. Three non-QUIPS schools (Ekumpe, C-Hi-N; Tusundo, C-Lo-N and Kunsu C-Lo￾M) had modern toilet and urinal facilities. The remaining 13 schools shared toilet facilities with the JSS, had improvised structures, or had no facilities. 6.7 TEXTBOOKS The evaluation team found no marked differences across the three regions in terms of availability of textbooks. Before QUIPS, it appears, the schools did not have adequate textbooks, especially in English and math. Currently, throughout the country there is still an extreme shortage of basic primary textbooks and readers. (See Table 6.15.) Although some field teams reported a few books in the schools, these were often locked up or in very poor condition. Although all the field sites reported that the lack of textbooks made teaching and learning difficult, the most extreme shortages were in the north. The field teams that visited the north found that although there were some lower primary textbooks, they were mostly torn and missing many pages. In some northern communities, there was only a single reading book in each of the upper primary classes. 74 Table 6.8: QUIPS Infrastructure Completion Level School Building Toilet Urinal NO COMMU NITY LOCA TION COMP. UNCOMP REMARK COMP UNCOMP. COMP UNCOMP REMARK 1 NAWUNI NORTH Yes at a standstill Yes Yes could not start them 2 KPANLORI NORTH Yes roofing sheets, 2 classrooms Yes Yes Community initiated project 3 LONTO NORTH Yes 3 classrooms Yes Yes 4 NAKPALE￾KWOLE NORTH Yes 3 classrooms Yes Yes 5 NTIRIBUOHO MIDDLE Yes 3 additional rooms Yes Yes completed with community top-up 6 ADUKROM MIDDLE Yes 6 classroom, office & store Yes Yes -do￾7 WATRESO SOUTH Yes 3 additional rooms Yes Yes -do￾8 DUNKWA ON￾OFFIN SOUTH Yes 6 classrooms, making it a storey building Yes Yes -do￾9 TUOBODOM MIDDLE Yes 6 classrooms Yes Yes -do￾Table 6.9: English and Mathematics Textbooks in Sampled Schools SCHOOL P 1 P 2 P3 P4 P 5 P 6 NAME ENG MATHS REMARKS ENG MATHS REMARKS ENG MATHS REMARKS ENG MATHS REMARKS ENG MATHS REMARKS ENG MATHS REMARKS NAWUNI R/C (Q) 10 KPANLORI E/A (Q) 10 0 20 0 Very bad 16 5 10 12 Eng bks bought by community GBANGU L/A ( C) 24 65 17 64 23 42 8 56 9 18 8 23 TUOBODO M NURIYA (Q) 15 23 37 23 31 14 31 17 34 23 29 12 Neatly packed KWAKU PAMFO (Q) 1 1 1 1 1 1 1 1 1 1 1 1 Teac hers copie s 75 6.7.1 English and Mathematics Textbooks. The absence of textbooks is one of the greatest obstacles to learning, particularly the development of basic literacy, and was a recurring theme in the field work. Although the problem of textbooks was reported in all field sites, predictably there were differences in the availability of texts related to school location. Schools that are very near the District Education Office (DEO) are more likely to receive books than those in remote areas. Schools in rural locations, far from the DEO, had extreme shortages of textbooks compared to more centrally located urban schools. As GES officials in the Basic Education Directorate Headquarters explained in an interview, “The reality of the distribution of materials is a function of proximity. Not everyone gets them. They often don’t get to the hinterlands.” For instance, Nawuni (Q-Lo-N) had 10 copies of the English textbook for the Grade 6 class and the other classes in Nawuni had no textbooks at all, not even a teachers’ copy. In Kpanlori (Q-Hi-N) there were no math textbooks for grades 1, 2, 3, or 4 and no English textbooks for Grades 1-2, although the community had managed to purchase a few English textbooks for the upper primary grades. At Gbangu-Bangbini (C-Hi-N), located closer to the DEO, the textbook scenario was considerably better. All the classes had 16 to 65 math textbooks and a few copies of English textbooks. At Tuobodom (Q-Lo-M), there were some math and English textbooks, although far from a sufficient number. Kwaku Pamfo (Q-Lo-S) had an extreme shortage: All six classes had only the teacher’s copy of the English and the math textbook. The field research also identified many situations where head teachers packed all the school’s textbooks in boxes or trunks that were inaccessible to teachers. In one school in the north, where there were reports of severe textbook shortages, piles of new English and math textbooks were found in the storeroom. Demographic data on pupils collected before the QUIPS interventions revealed that a small percentage of pupils had textbooks to use at home: Nationally, on average only 8% to 22% percent of children have a math textbook to use at home, although the percentages are lower for pupils in northern Ghana and remote locations in general. Only 13% to 20% of Ghanaian primary school children have English texts to use at home, and again the percentages in the north and the rural areas are lower. (See Tables 2.6 and 2.7 above.) 6.7.2 Supplementary Readers. The fieldwork revealed an extreme lack of supplementary readers across the 18 evaluative sites. In most of the schools where supplementary readers and library books were available (e.g., Tuobodom, Q-Lo-M) they were kept in heavily padlocked metal boxes, usually in the head teacher’s office, making it difficult for teachers to access teaching aids for their pupils and preventing access by teachers and pupils at their own convenience (field work, north). The same practice 76 prevailed in a number of other schools, QUIPS and control, across the country (Kunsu￾Dotiem, C-Lo-M; Nyamebekyere, C-Hi-S; Gbangu, C-Hi-N; Nakpale, Q-Lo-N; Tusundo, C-Lo-N; and Watreso, Q-Hi-S). Only one school (Dunkwa, Q-Hi-S) had a library with books. Results from descriptive analyses of national demographic data reinforce the field observations of the scarcity of children’s books. Only 21% to 47% of the pupils surveyed had any children’s books to use at home. As usual, the lowest percentages were in northern Ghana and in rural communities. Given the scarcity of reading material, it is no wonder that texts were not a factor in differentiating low- and high-performing schools or QUIPS and control learning environments. It also helps to explain why learning in public schools across the nation is seriously impeded, in spite of considerable donor activity and development programs. 6.8 AVAILABILITY OF TEACHING STAFF During the project, all the QUIPS schools in southern and middle Ghana had the full complement of six GES-trained teachers—one of the selection criteria, although this criterion was not applied strictly in northern Ghana, where there is an extreme shortage of trained teachers because they are unwilling to be posted to remote parts of the country.57 In northern Ghana, DEOs did ensure that at least one trained teacher was posted to each of the QUIPS schools.58 After the QUIPS interventions ended, some QUIPS schools in southern and middle Ghana were able to retain most of their complement of six trained teachers. This was not true in northern Ghana. The teacher staffing findings (see Annex 8) show that the urban schools visited in southern and middle Ghana are staffed with GES trained￾teachers while staff in the schools in northern Ghana consisted of a mixture of trained teachers, untrained instructors and volunteer teachers. The QUIPS schools visited in rural areas had lost most of their trained teachers when the QUIPS interventions ended. Adukrom (Q-Lo-M) was the exception; it was the only rural QUIPS school that retained its complement of six trained teachers to the time of this evaluation. In northern Ghana, an average of one to two QUIPS-trained teachers 57 Based on interviews with CRS. 58 In its review of the evaluation the Academy of Educational Development stated that: “This statement is not correct. Not all schools were in fact fully staffed. In the southern and middle part of the country most schools started with or gradually built up a full staff, but there were a number of schools that never had a full complement. Moreover, some schools lost teachers that were not replaced during the two year period. In addition, there were instances where teachers were on long medical or maternity leave - up to a term or longer and who therefore could not be replaced while on leave status.” 77 were still teaching in each former QUIPS school; in southern Ghana retention averaged from two to five.59 The staffing situation in northern Ghana at the time of the evaluation had largely returned to the situation before the QUIPS program, according to field reports. Most of the schools in the rural areas were and are seriously understaffed. For instance, Adukrom (Q-Lo-M), said there had only four teachers, untrained, before QUIPS; the school had never received a full complement of teachers until QUIPS arrived. During the project, all the QUIPS schools in southern and middle Ghana were fully staffed. Even though QUIPS schools in northern Ghana were only assured one GES-trained teacher, findings from the field indicate that the DEO made no concerted effort to retain teachers, trained or untrained, in former QUIPS schools. At Nawuni, (Q-Lo-N) the number of teachers, which had increased to four during QUIPS, is now three. An elder at another school expressing his disappointment in the staffing situation, said that since the end of QUIPS, “There is no longer any help from the Assembly. The education office is even taking away our teachers without replacing them.” The staffing situation in schools visited in southern and middle Ghana is different. The DEO has made efforts to keep trained QUIPS teachers where they were posted. For instance, Adukrom (Q-Lo-M) had six trained teachers, five of them QUIPS-trained, on staff during the time of the evaluation.60 It appears that the DEOs in southern and middle Ghana have maintained trained teachers at the QUIPS schools even though this may have created an imbalance in staffing in the district. District officers claimed repeatedly that staffing of QUIPS schools was accomplished at the expense of other schools in the districts. Staffing in urban and rural schools was as inequitable as the distribution of textbooks. Schools located in district capitals and towns close to a district or regional capital were better staffed than rural schools in the same district. For instance, Dunkwa Presby (Q￾Hi-S), which is in a district capital, and Ntonso (C-Hi-M), a school twenty-one kilometers from Kumasi, the regional capital, and just five kilometers from the district capital, both had the full complement of teachers. Ntiribuoho (Q-Hi-M), also about 10 kilometers from Kumasi, had six trained teachers. Generally, rural control schools continued to have severe staffing problems. Kunsu￾Dotiem (C-Lo-M) and Nyamebekyere (C-Hi-S), as well as all the control schools visited in the north, fall into this category. 59 At Nakpale-Kworle the only QUIPS-trained teacher had been appointed head teacher. Lonto Presby had two, one QUIPS-trained professional teacher and one untrained QUIPS teacher that had attended in￾service training, Nawuni and Kpanlori had two QUIPS-trained teachers each. 60 Ntiribuoho (Q-Hi-M) also had six trained teachers, two of them QUIPS-trained. Ntonso (C-Lo-M) had six, all trained, while Kunsu-Dotiem (C-Lo-M) had four teachers, three trained and one untrained. Dunkwa Presby (Q-Hi-S) had six teachers, three QUIPS-trained and three untrained. Watreso (Q-M-S) had six trained teachers. 78 In summary, there are persisting and severe staffing problems, particularly in northern Ghana, where both QUIPS and control schools in the rural areas were understaffed. Furthermore, the staffing criterion of six trained teachers (or even one in the north) created an “unnatural” context for QUIPS and its outcomes within the community; it also posed a challenge to the sustainability of the program. 6.9 TEACHER IN-SERVICE TRAINING The fieldwork report found that QUIPS in-service training (INSET) took five forms: • National/cohort level training, where all QUIPS teachers in a particular cohort were trained by QUIPS-trained personnel when the project began. • National/cohort level training, where all QUIPS teachers in a particular cohort were trained by QUIPS-trained personnel on-site at each school. • District-level training, where QUIPS trainers and District Teacher Support Teams (DTST) trained QUIPS school teachers. • Cluster-based training workshops, where the DTST further organized training workshops for QUIPS school teachers at the circuit level. • School-based training workshops, organized on premises by head teachers of QUIPS schools, where teachers discussed challenges they encountered in the classroom as well as head teacher observations based on regular supervision (field reports, middle belt). In addition to these training programs, DEO staff, particularly the Circuit Supervisors, visited the schools to provide professional support. Teachers interviewed said that the Circuit Supervisors organized demonstration lessons, observed their teaching in the classroom, and discussed new teaching techniques with them. In some cases, control schools benefited from three out of the four levels of training (district, cluster, and school-based). Some districts made concerted efforts, using funds from DFID/GES, to assemble QUIPS materials, handouts, and manuals at the district level and distribute these among control schools. Four of eight districts visited had shared materials to spread QUIPS best practices for improving teacher performance and overall school quality. Only the control schools included in the district grant process benefited from funds for school-based in-service training, although others benefited from the district cluster - based INSET that was funded by Whole School Development (WSD). QUIPS manuals and handouts were found in a few of the 18 schools sampled but there was little evidence that they were used. Interviews with teachers in the schools that received district wide training on QUIPS best practices said that INSETS for QUIPS and control schools were similar. 79 6.10 TEACHER PRACTICES AND THE PROFESSIONAL COMMITMENT OF TEACHERS This section addresses teacher regularity and punctuality, types of preparation for lessons, lesson plans, teaching aids, interaction and questioning techniques, and head teacher supervision. The issue of teacher absenteeism and punctuality was raised repeatedly throughout the evaluation. In most school and community interviews it was said that teachers were not regular and punctual before the QUIPS program. However, they improved during the QUIPS period, perhaps because of increased supervision by head teachers and Circuit Supervisors and perhaps because districts selected teachers for the program carefully to ensure good performance. After QUIPS, however teacher absenteeism and lateness resurfaced, especially in the north. Pupils, head teachers, chiefs and elders, and SMC/PTA members all mentioned absenteeism as a major barrier to learning. At one school in the north, community leaders and PTAs reported that during the program teachers were known to be regular in attendance and punctual. “Today, they come to teach when they want to,” The head teacher and the trained teachers of that school were absent during the first day the evaluation team visited the school. In another school the elders maintained that the “Grade 6 teacher frequently is absent from school and provides false reasons for being away … claiming to be the district sports coordinator.” Community members in a school in the middle belt reported that because of the irregular attendance and punctuality of the teachers, including QUIPS￾trained teachers, there had been a total breakdown of QUIPS good practices. Enrollment of children in the school had dropped. In a number of low-performing schools, PTA/SMC members commented that not only was teacher attendance irregular, but when teachers were in school their time on task was minimal. The EARC report on teacher time on task concluded that one-third of teacher time in school was lost because the teacher was engaged not with pupils but with visits talking on the veranda, meeting visitors, going out to the head teacher’s office or to the bank, sick days or arriving at school late and departing early.61 6.10.1 Preparation and Use of Lesson Plans. Before the QUIPS program, most teachers were not planning lessons beforehand. That problem prevailed afterwards as well. From lessons observed during evaluation, it appears that in both QUIPS and control schools the teachers had not prepared lesson plans. At QUIPS low performance school, the teacher had not prepared a lesson plan since July 2004, when QUIPS ended there. 61Education Assessment and Research Centre (2002), Study of Teacher Time on Task. 80 The teachers interviewed said they had not prepared lesson notes because they had not been supplied with notebooks. In some schools, teachers explained they had not prepared lesson notes because the term had just started. However, in some schools teachers had prepared comprehensive lesson notes and used them in teaching (e.g. Adukrom (Q-Lo-M), Dunkwa (Q-Hi-S), Ntonso (C-Hi-M), and Tusundo (C-Lo-N). In several QUIPS schools teachers observed had not prepared lesson plans for the lessons observed by the evaluation teams, as was true also in some of the control schools. 6.10.2 Preparation and Use of Teaching and Learning Materials. A few teachers in the northern schools demonstrated the use of teaching aids during the evaluation team’s observations of teaching practice, although it was clear that across all regions, teaching aids were nonexistent, insufficient, or dumped in head teachers’ offices. Of the 18 schools, four used teaching aids - Ntonso (C-Hi-M), Adukrom (Q-Lo-M), Dunkwa (Q￾Hi-S) and Tusundo (C-Lo-N) but only two, both QUIPS schools, had displays of them on the walls.62 Among those that did not use teaching aids, at one school in the north, “teaching aids, including the sturdy wooden items supplied by UNICEF, were haphazardly dumped in a corner of the unlocked portion of the office building.” Another kept teaching aids at the office, gathering dust. Yet another simply had no teaching aids in the school. These situations were fairly typical. Thus, although all the DEOs had mentioned use of teaching aids as one of the best practices introduced as part of the QUIPS program, in the majority of sampled schools, most teachers, QUIPS trained or not, were still not using teaching aids regularly. 6.11 HEAD TEACHER AND DISTRICT SUPPORT TO SCHOOLS Effective head teacher supervision and support was observed in five of nine QUIPS evaluation sites— Adukrom, Dunkwa Presby, Watreso, Kpanlori, and Lonto. That was true also in three of the nine control schools—Ekumpe, Tusundo, and Ntonso. These findings from the field suggest that the presence of effective head teachers in any school may be one of the most important factors for improving learning within the classroom setting and can make a difference in the performance of both teachers and pupils. These findings support research conducted during QUIPS on the role of the head teacher in school performance (Klauss et al., 2004). 62 At Ntonso, it was reported that teaching aids were used during the lesson but not displayed in the classroom. At Adukrom and Dunkwa Presby, teaching aids were not only displayed in classrooms but were also actually used in the lessons observed. Adukrom (Q-Lo-M), that displayed and used TLMs in class, was the only low school to do either. 81 The scaling up of QUIPS influenced the practices of head teachers across the districts observed and appears to have been successful in spreading good practices to non￾QUIPS schools. Effective head teachers in both QUIPS and control schools organized school-based INSET for inexperienced teachers and were themselves very reliable and punctual. The head of Ekumpe (C-Hi-N) even encouraged his teachers to offer extra classes to pupils free of charge. There were also several examples of ineffective head teachers in QUIPS schools. For instance, the head teacher at one of these schools did not take the initiative to provide support to the first grade class in which 109 children were enrolled but had no assigned teacher even though there were not enough teachers at the school and he had taught first grade for six years before becoming the head. At another QUIPS school, the evaluation team encountered the head teacher at the lorry station, where he claimed he was seeking transport to the district capital even though he was not dressed appropriately and was carrying his baby. The unreliability of these head teachers seriously undermined the reliability and punctuality of other teachers and of pupils. Absenteeism, tardiness, roaming in town, and lastly ineffective teaching were what the team observed at some schools in all three regions. The qualitative study revealed that the low performance of children in schools across all regions was correlated with poor head teacher performance and lack of supervision and support to the teachers. Head teachers in all but two of the low-performing schools were reported to be ineffective. It was clear from the fieldwork reports that DEO staff visits to schools to provide professional support were more common in urban areas, which all received frequent DEO staff visits.63 The only exception was one QUIPS school, where lack of DEO supervision contributed to the poor state of affairs in the school. Another control school in the middle belt never received regular visits by the DEO staff, even after chiefs and elders had reported the irregular attendance of the head teacher and teachers to the District Director of Education. 6.12 TEACHER PERFORMANCE QUIPS-trained teachers who had prepared lesson plans performed better in the classrooms observed and made more use of QUIPS practices. Where committed QUIPS-trained teachers were still in place, QUIPS practices were still visible. In the two out of nine QUIPS schools evaluated that were continuing to use these practices (Dunkwa, Q-Hi-S; and Adukrom, Q-Lo-M), each still had their QUIPS-trained teachers. These were observed to use child-centered teaching approaches and small group learning. The field workers reported in particular that at Adukrom, the teacher observed was self-confident and demonstrated familiarity with QUIPS teaching methods, lesson preparation, and use of teaching aids. The teacher used group activity to teach 63 For example: Ntiribuoho, Ntonso and Dunkwa. 82 mathematics: Children worked in small groups with leaders they themselves appointed, and at the end of the activity, the leader presented the solution of each group on the board to be discussed by the class. The QUIPS-trained teacher at Dunkwa (Q-Hi-S) considered the different aptitudes of children in preparing his lesson plans and is clearly committed to the use of groupings and involving children in lessons; he used flexible ability grouping practices in the math lesson the field team observed. There were indications that group work went on in some schools, both QUIPS and control. The teacher observed at Ekumpe (C-Hi-N) created an environment that was very relaxed and conducive to active participation, although the teacher did much of the talking and children participated only when asked questions. At Ekumpe, in the Grade 4 class where the teacher was absent, the children sat in groups that had obviously been organized by the pupils themselves. In a few of the QUIPS schools Adukrom (Q-Lo-M), Dunkwa (Q-Hi-S), and Watreso (Q-Hi-S)—children sat permanently in small groups. 6.12.1 Review of Written Exercises. The performance of teachers in the sampled schools was partially evaluated in terms of the number of math and English exercises pupils completed during the school year. (See Table 6.16.) Table 6.16: Range of Grades 4 and 6 Mathematics and English Exercises Completed by Pupils in their Exercise Books Region School Mathematics English Middle Ntribuoho L/A QUIPS 3-12 5-18 Middle Ntonso SDA Control 14-49 33-61 Middle Adukrom L/A QUIPS 39-52 33-61 Middle Kunsu Dotiem L/A Control 8-9 1-18 South Watreso D/C QUIPS 29-45 26-45 South Nyamebekyere D/C Control 0-4 2-13 South Dunkwa Presby QUIPS 37-49 45-50 South Achiase R/C Control 16-20 25-37 North Lonto Presby QUIPS 6-15 8-12 North Ekumpe L/A Control 6-8 7-8 North Nakpale Kworle QUIPS 5-9 3-9 North Tusundo L/A Control 3-9 4-8 Clearly, there was no substantial difference in pupil exercise book activities between control and QUIPS schools, but there was a significant difference between the north and the south and middle combined. Pupils in the north, from QUIPS and control schools alike, accomplished fewer exercises than their counterparts elsewhere in Ghana. On average, most children in the north had only three to nine exercises in their workbooks for the period from September 2004 to April 2005. 83 Children in middle Ghana, in both QUIPS and control schools, completed on average 30 to 40 exercises during the study period.64 The differences in written exercises completed by children in the QUIPS and control schools were not noteworthy. In the south, there was a significant difference in the number of exercises pupils completed in math and English between the high- and low-performing schools. In Dunkwa Presby and Watreso LA, both QUIPS high-performing schools, teachers gave pupils 45 to 50 math exercises from September 2004 to March 2005; while two control schools had administered only three and 37 exercises. The trend for English exercises was similar. The picture drawn from site visits in northern Ghana is that the highest number of exercises given between September 2004 and March 2005 was 25 in English at Lonto Presby (Q-Hi-N), which had given 22 math exercises. In the north there was no noticeable systematic difference between QUIPS and control or high- and low-performing schools. Teacher feedback on pupils’ completed exercises was generally better in southern and middle than in northern Ghana. Even at Lonto Presby, the school that reported the highest number of exercises assigned, the teachers had not marked most of the exercises completed. The field research found that in the south and middle, the few schools that retained QUIPS-trained teachers also had given more exercises than control schools. 6.12.2 Teacher Performance at Lesson Delivery. Teachers who prepared lesson plans and taught a lesson aligned with that plan were more likely to be QUIPS-trained than not. Their lesson plans demonstrated clearly that QUIPS-trained teachers evidenced more of the targeted good teaching practices. At one control school, the teacher observed had prepared a comprehensive lesson plan, used teaching aids, had them on the walls, and engaged children in a rectangular seating arrangement. He made an attempt to actively involve all pupils, although because he did more of the talking, there was little child involvement. However, he demonstrated excellent questioning skills. At two QUIPS schools, different QUIPS-trained teachers used teaching aids, the question and answer technique, and discussion methods. These teachers engaged children in group activities, which resulted in pupil–pupil and pupil–teacher as well as teacher–pupil interactions. Children asked questions, and each group was given an 64 In the middle region, Ntonso (C-Hi-M) an urban school, and Adukrom (Q-Lo-M) a rural school, had done the highest number of exercises in math and English. In Ntonso the teacher observed had administered as many as 68 math exercises, while at Kunsu (C-Lo-M) pupils were given as few as 18. The trend in English exercises was similar: Ntonso gave 49 and Adukrom 52, while Ntribuoho could manage only 12 and Kunsu only nine. A high number of exercises by a pupil is relative: GES standards for math and English would be a minimum of 3 exercises each per week or 72 for the study period from September 2004 to April 2005. 84 exercise that suited the level of performance of its members. Pupils therefore were involved. 6.12.3 Professional Commitment of Teachers. Professional commitment of teachers helped sustain QUIPS gains in the schools sampled. High performance attained in some QUIPS schools during the QUIPS period has since slowed down because of the movement of QUIPS-trained teachers, loss of commitment of some retained QUIPS-trained teachers, and ineffective supervision in the schools by both head teachers and Circuit Supervisors. Nationally at least two of the eight control schools—Ntonso and Gbangu-Bangbini— were performing better than QUIPS schools. At one of the QUIPS high performing schools none of the teachers were using lesson plans although each had two QUIPS￾trained teachers. The teachers were not using teaching aids or small group methods; although they used questions and answers, the dialogue was one-way, from teacher to pupils. The main fieldwork finding here is that while there were many devoted and committed teachers in QUIPS schools, there was also high teacher turnover and transfer out from QUIPS to non-QUIPS schools due to teacher dislike of the higher workload the QUIPS program required. This greatly limited the sustainability of QUIPS. In the interviews, several QUIPS teachers revealed that they saw the training component of QUIPS as an extra duty that should attract extra income and but did not consider this extra effort part of their ongoing professional work.65 6.13 CHALLENGES TO SUSTAINABILITY The evaluation exercise found that QUIPS had limited outcomes at schools where QUIPS teachers had not remained in place. More lasting results were felt where non￾QUIPS teachers and schools had adopted some of the best practices from the project. Wherever the teams visited, QUIPS-trained teachers talked about QUIPS with fondness. Some expressed the view that QUIPS had provided a model of primary education in Ghana that could lead to great improvement if the model could be adopted universally—but most of the inputs had been concentrated in too few schools. At the school level, key outcomes were provision of infrastructure and furniture, and training of teachers and head teachers. To some degree these outcomes did translate into a change in classroom management as well as instructional behavior during QUIPS, but to a large extent these behaviors were not fully sustained where there was not strong head teacher leadership and support. 65 More fieldwork is required to learn more about QUIPS teachers who have been posted to non-QUIPS schools. 85 Inadequate supplies of textbooks were also a major challenge both to classroom teaching and learning and to insuring that QUIPS investment in training teachers was maximized.66 To compound the problems, head teachers and teachers were found to be unfamiliar with simple approaches to using texts and literature in classes and many had reverted back to teacher-centered classroom behavior. Challenges to sustaining QUIPS inputs for teachers in QUIPS schools included a lack of funds to purchase materials for preparing teaching aids. According to teachers from all the QUIPS schools, grants to purchase materials that should have come from the communities have not done so. Head teachers also were not able to sustain INSET training because funds were not forthcoming from DEOs. Lack of funds thus limited the spread and retention of changes in teaching practice. Ineffective head teacher support to and supervision of teachers in QUIPS schools was also a great challenge to successful teaching and learning. Most head teachers in the schools visited were not fully effective; lack of visits by Circuit Supervisors and other district office staff compounded the problem. The lack of teachers, especially trained teachers, affected the education in all the schools, especially those in the north.67 One of the greatest challenges to the sustainability of QUIPS is the professional commitment of teachers. All the teams reported that teachers complained of the lack of compensation for the extra duties associated with the QUIPS program. One team noted that, “until teachers’ salaries are modified significantly, teaching will remain for the vast majority, especially those who are heads of household, a part time job.” Teachers interviewed recommended that the time demands of training take this into account and that compensation for lost income be considered if the best practices of the QUIPS program are to be shared and sustained. A major challenge in the schools surveyed was that when a QUIPS-trained teacher was transferred out, QUIPS practices within the school often disappeared. It is therefore recommended that the best practices of QUIPS be introduced into the curriculum of the teacher training colleges so that a greater number of teachers become familiar with child-centered approaches to teaching. 6.14 CONCLUSIONS Both quantitative and qualitative reports suggest that school-level interventions directly impacted the quality and effectiveness of QUIPS schools during the period of active QUIPS training and support. This is evidenced by the positive change in teacher 66 Watreso, for instance, has only three reading books and five science textbooks in Grade 6, while Kwaku Pamfo has only the teacher’s copy for all classes in math and English. 67 As many as five of the eight schools visited had only three untrained teachers each (excluding volunteer teachers); indeed, five of the eight schools in the north had not received a full complement, whereas only three schools in the middle and south had not received a full complement. 86 performance from baseline to the end of the two-year period and by the reflections from interviewees about the “QUIPS era” and the quality the program brought to QUIPS schools. For the most part, QUIPS-trained teachers were committed during the QUIPS program; they completed lesson plans and made good use of teaching aids. Supervision on the part of both school and district was enhanced and infrastructure significantly improved. These gains, however, were not without cost. Parents in nearby schools began to place their children in QUIPS schools, introducing an imbalance in the distribution of pupils across communities near the QUIPS school. In maintaining the required supply of teachers and complying with the supervision requirements of QUIPS, DEOs found that their support to other schools in the district was being compromised. There were opportunity costs for teachers, who worked long hours to comply with the demands of QUIPS at the expense of their attention to home and community affairs or after-school income-generating activities. The overwhelming conclusion to be drawn is that “where QUIPS teachers were no longer available in schools, the short-term benefits of QUIPS . . . were not sustained.” QUIPS-targeted instructional practices disappeared in schools when QUIPS-trained teachers left. On the other hand, where the majority of QUIPS-trained teachers were still in place in a former QUIPS school, quality ingredients that had characterized QUIPS were still observed, although the level of commitment and certainly the level of supervision were observed to have waned. During the period of active QUIPS interventions, teachers enjoyed the benefit of teacher-to-teacher support and, as the head teacher participated in the program, all learned and changed together as a team. The enhanced supervision from the district and the support of parents and community leaders translated into improved teacher self-esteem and confidence. During the two-year period of active QUIPS inputs, there was a shift in the school climate and conditions of service improved, if for no other reason than that external attention and support for teachers increased. In the aftermath, however, with many teachers and even head teachers transferring out of the district, reinforcement of best teaching practices deteriorated. The extent to which QUIPS-trained teachers could maintain high commitment and use best practices in their schools was also challenged because there were no peer QUIPS-trained teachers and head teachers to encourage the use of QUIPS good practices. Unless there are attempts to affect systemic reforms by working directly with Ghana’s formal teacher training system and to legislate policy directed to improving the integrity of the teaching profession, it is naïve to expect reforms in classroom instruction to be sustained. This problem does not appear to have been addressed in QUIPS. 87 In addition to these systemic constraints related to the conditions of service of teachers and the integrity of the profession, the effectiveness and sustainability of QUIPS were further undermined by such systemic problems as the lack of textbooks68 and supplemental readers, and teacher misunderstanding of how best to use teaching aids when they are available. These deficiencies need to be addressed in a deliberate way throughout the country. There are a number of good models in other developing countries where cost effectiveness, quality, and availability of reading materials for primary schools, as well as teacher support for their use, has been improved through privatization and competition. Another constraint to the effectiveness and sustainability of QUIPS is related to the ability of the system to maintain high levels of teacher support and supervision. One of the results of the evaluative field work and the Klauss study (2004) was a better understanding of the prominent role of the head teacher in school quality and effectiveness. Schools where learning was taking place, QUIPS or control, were schools where the head teacher was taking a strong stance as a leader and champion of quality schooling and pupils’ learning. These head teachers gave direct support and guidance to their teachers, and were active in facilitating and sustaining community interest and involvement in school matters. School-level education development programs must work together with teacher professional development programs to build a strong force of head teachers across Ghana, identifying and systemically addressing gaps in head teacher capacities to provide the kind of leadership that struggling schools need. One somewhat different conclusion drawn from the results on school level outcomes is the impact of infrastructure improvements. Throughout the field work, the team heard testimonials about the benefit of infrastructure improvements, primarily related to the fact that pupils and teachers were able to stay in school during periods of heavy rainfall. Yet there was no direct evidence that the infrastructure improvements were tied to school performance and pupil learning. However, one benefit of the infrastructure development program was the sense of ownership facilitated by involving the community in the construction process and by mobilizing community involvement in school improvement through the micro-grant process. Infrastructure programs should first define their goals. If the main goal is to improve school effectiveness as measured by pupil learning, an infrastructure project might consider building teacher housing to encourage retention, keep teachers punctual, and 68 In June 2005, the GES and the World Bank awarded contracts to local publishing companies to produce primary school textbooks (English, math, science and local languages) and deliver them to all 138 DEOs for the 2005-2006 academic year. Import duties on raw materials for the textbooks have been suspended. The quantities agreed upon aim at a 1:1 textbook to pupil ratio in private as well as public primary schools. Daily Graphic Nii Armah Hanson article on the printing industry June 23, 2005 and SEDCO Publishers. 88 build their commitment and sense of accountability to the communities where they teach. On the other hand, if the goal is to facilitate community sense of ownership in the schools, the building of classroom blocks and acquisition of furniture may be more useful. In both cases, the responsibility for part of the construction or procurement process needs to be shared with the community leadership. CHAPTER 7: COMMUNITY-LEVEL OUTCOMES OF QUIPS 7.1 KEY INPUTS AND INTERVENTION STRATEGIES OF QUIPS The community component of the QUIPS program was significant in terms of the type of community support and training provided to targeted communities when QUIPS began and during the scaling-up activities at the district level. The interventions planned to strengthen community support towards the school in order to ensure learning were: • Community investigation into priorities, problems, and solutions using participatory rural appraisal (PRA) and participatory learning appraisal (PLA) techniques; • SMC/PTA training for all QUIPS communities and schools, and later district and nation-wide training; • Training and community support by National Service Volunteers, posted to the districts to facilitate, and coordinate community activities; • Development of school improvement plans (SIP); and • Creation of community support units as part of BED’s ongoing work. In most cases the implementing agencies, Catholic Relief Services (CRS) in the north and the Community Schools Alliance (CSA) in the south, used community drama and participatory learning and action techniques to engage the community in a self-analysis of problems and potential solutions related to local education and schooling. 7.2 COMMUNITY-LEVEL OUTCOMES ON THE QUIPS COMMUNITY BEST PRACTICE INSTRUMENT The “Community Best Practices Assessment Instrument” was designed to guide quantification and evaluation of community involvement in education during the QUIPS period. The instrument was drafted during the first two years of the QUIPS program with the full participation of members of Cohorts 1 and 2 communities. The instrument measures community performance against ten focus areas considered critical to effective and sustainable community involvement in education. It uses a five￾point rating scale to guide the assignment of communities into low, low to average, average, average to high, or high categories. Low to high values were associated with a developmental continuum.69 The ten focus areas of community participation assessed through the Community Best Practices Assessment Instrument were: 89 69 Low performance was characterized by a lack of evidence for the targeted community practice; low to average as a community that was getting started in development of the targeted practice; average as a community well on the way to the targeted best practice; average to high as a community beginning to consolidate the new good practice; and high as a stable community with the potential for sustaining the targeted best practice. 90 • Trust in the school system; • Support for quality education; • Support for girls’ education; • Empowerment of local people to act; • Community support for strengthening school management; • Productive links with external agencies; • Use of participatory planning and design; • Mobilization of local and district resources; • Monitoring of school finances and assets (transparency); and • Help to strengthen school management structures. One limitation in the quantitative analyses was the differences in the data collection processes used by the CSA and CRS. The CSA project implementing the community component in the south used external data collectors; CRS used its own staff. Data collection by members of the actual implementation team weakens the validity of results. It often introduces bias in the direction of good performance.70 Moreover, the community intervention itself was not consistent across regions. CSA used new university graduates or National Service Personnel who were trained in the facilitation techniques that form the basis of community interventions. CRS, on the other hand, used education professionals living in the same community or nearby. Thus, results in the north and south may be intrinsically different. Three quantitative analyses investigated QUIPS outcomes with regard to community training and support using the Community Best Practice Assessment data from communities in Cohorts 4, 5, and 6. The first analysis compared community performance on the ten community practice focus areas from baseline to the end of the two-year intervention cycle.71 72 The second analysis compared the community practices in the three regions: south, middle and north.73 The third analysis, compiling 70 In its review of the draft evaluation, Catholic Relief Services stated that: “The comparison between CSA and CRS in terms of personnel used for data collection is not correctly captured. CRS used GES staff because they were our main partners in the implementation of the project. Using them presented the opportunity for them to build their capacity in such activities and thus make the issue of continuation and sustainability more meaningful and attainable. However,[ ] the GES staff never worked in their own districts. Nonetheless, in allocating districts, care was taken to give due consideration to the fact that the data collectors understood not only the language of the communities but knew also the culture so as to be functional and collect accurate and reliable data.” 71 One of the ten focus areas, “helps to strengthen school management structures,” could not be included in the analysis. 72 It is recognized that there was some support and training beyond the two-year cycle in selected communities; however, data for the analyses presented are based on community data collected at the end of the two-year cycle. 73 North: Northern, Upper East, Upper West; Middle Belt: Brong Ahafo, Ashanti, Western; South: Greater Accra, Central, Volta, Eastern. community data across regions, explored the relationship of the nine community best practices and the pupil-learning outcomes measured during the QUIPS cycle. 7.2.1 Analysis 1: Exploring Shifts in Targeted Good Practices. This analysis operated on data available at baseline and at the end of the intervention. Data from all QUIPS communities in the south were included, but only Cohort 4 communities had baseline data available from the north. The results for each of the nine targeted community practices taken at baseline and again at the end of the two-year period of training and support were cross-tabulated.74 The results showed that community involvement increased along all nine of the targeted community good practices and chi-square tests of significance applied to each cross￾tabulation indicated that the improvements in community-school relations and community support to the education process were statistically significant (p < 0.01). Table 7.1 presents the cluster of performance across the five-point rating scale at baseline and again at the end of the two-year period of training and support. Table 7.1: General Shift in Performance Clusters on the Community Best Practice Assessment Instrument from Baseline to QUIPS Termination Absent Getting Started Developing Consolidation Stabilizing/ Potential for Sustainability Baseline XX XX Outcome XX XX X Baseline XX XX XX Outcome XX XX X Baseline XX XX XX Outcome XX XX Baseline XX XX Outcome XX X Baseline XX XX X Outcome X XX Baseline X XX XX Outcome XX XX Baseline X XX X Outcome X XX X Baseline XX XX Outcome XX X Baseline XX XX XX Outcome XX XX XX Monitors and Accounts for School Finances and Assets *In this table, boxes marked "XX" identify areas on the continuum where the majority of communities are performing. Boxes marked with a single "X" show where some communities are performing, but not the bulk of them. Mobilizes Local and District Resourced Level of Development on Community Best Practices Instrument FOCUS AREA DATA COLLECTION OCCASION Empower Local People to Act Strengthen School Management Partnerships with External Agencies Participatory Planning and Design General Shifts in Performance Clusters on the Community Best Practice Assessment Instrument from Baseline to the End of the Two Year Period of Training and Support* Trust in the School System Support Quality Education Support Girls Education 91 74 Data from the north was not available on community practices related to “Support for Girls’ Education.” 92 nalysis of the results presented in Table 7.1 sheds light on the types of effective ing ality; “leap” l y and also he community practice where more resistance to change appeared was transparency of t much ly .2.2 Analysis 2: Exploring Regional Differences. Considering the obvious nces owever, contrary to expectations, cross-tabulations of the targeted community in .2.3 Analysis 3: Exploring the Relationship of Community Involvement in nity data A community practices that were most and least responsive to QUIPS-provided train and support. The greatest shifts in community behavior (which may suggest more responsiveness to training and support) were observed in: (1) supporting school qu (2) supporting girls’ education; (3) strengthening school management; and (4) participatory planning and design. These four best practices were observed to across the development continuum. Community practices related to “supporting schoo quality” and “strengthening school management” were either absent at baseline or minimal, yet by the end of the intervention the majority of communities were consolidating these community practices or were showing evidence of stabilit potential sustainability. Community practices related to “supporting girls’ education” demonstrated a marked change from baseline to the end of the period, with support for girls’ education observed at the highest level in a noticeable number of communities. T in financial management—the ability of communities to “effectively monitor and be accountable for school funds and assets.” Communities demonstrated a wide range performance on this practice at baseline: some showed little awareness of their responsibilities for school finances and little interest, others showed average performance, characterized by increased awareness and interest, although no initiative about monitoring funds. At the end of the QUIPS training and support, performance did improve, although the majority of communities demonstrated on increased awareness and interest and the beginnings of some accountability for SMC/PTA funds—but not for school finances. 7 regional socioeconomic and cultural differences in Ghana and the apparent differe in the implementation of community interventions in the north and the south by the two agencies, it was important to explore differences on community best practices across regions before compiling these data. H practices by region (south, middle, and north) revealed no meaningful differences performance on eight of the nine measured focus areas.75 This was confirmed during the evaluative field work phase, where no significant regional differences in community support to schooling were found, other than differences in styles of community leadership, which vary based on socio-cultural groupings. 7 Education and Pupils’ Learning Growth. Given the absence of substantial differences in the distribution of community best practice assessments, commu collected in the north and in the south and middle combined were analyzed. Data from 75 Information on “supporting girls’ education“ was not available from the north. 93 Cohorts 4, 5, and 6 communities that were also in the sample for the learning outcome study (Chapter 5) were used. Preliminary descriptive analyses of the Best Practice Assessment results based on data collection at the end of the QUIPS intervention found what was essentially a bimodal distribution with two performance clusters on the 9 best practices: (1) “low to average” up to “average” and (2) “average to high” up to “high.” Preliminary descriptive analyses showed further that, where community performance appeared to be correlated with learning, high- and low-learning groups could be differentiated the same way. The cross-tabulations of community performance and learning outcome were therefore based on a simplified two-by-two table. Results from the Community Best Practice Assessment were recoded into “high” and “low” community performance groups based on the bimodal clusters and then cross-tabulated only with the “high” and “low” but not the “average” learning groups. In the cross-tabulations of community performance and learning, five of the nine focus areas showed some systematic relation to learning growth, with high performance more likely in schools where the patterns for pupils fell in the “high” category. The five areas related to pupil learning growth were: (1) trust in the school system; (2) support for quality education; (3) empowerment of local people to act; (4) school management support; and (5) participatory planning and design. The cross-tabulations of results on each of these community focus areas and the high and low learning outcome groups are in Tables 7.2 to 7.6 and Figures 7.1 to 7.5, which demonstrate at least some relation between community practices in these five focus areas and pupils’ learning. Chi-square contingency table analyses identified two of these areas—support for quality education” and “empowerment of local people to act— as significantly related to pupils’ learning. The cross-tabulation of results on these two practices, shown in Tables 7.3 and 7.4 demonstrate the strong link between high community performance in these two areas and high learning outcomes. Table 7.2: Trust in the School System Trust in School System Learning Group Low to Average Involvement High Involvement Total Low 9 45.0% 11 55.0% 20 100.0% High 13 27.7% 34 72.3% 47 100.0% Total 22 32.8% 45 67.2% 67 100.0% Figure 7.1: Trust in the School System 94 Table 7.3: Support for Quality Education * Trust in School System 0 20 40 60 80 Low High Learning Group Perc ent Low to Average Involvement High Involvement Support Quality Education Learning Group Low to Average Involvement High Involvement Total Low 15 75.0% 5 25.0% 20 100.0% High 19 40.4% 28 59.6% 47 100.0% Total 34 50.7% 33 49.3% 67 100.0% * Significant chi-Square: p < 0.015 Figure 7.2: Support for Quality Education Support Quality Education 0 20 40 60 80 Low High Learning Group P ercent Low to Average Involvement High Involvement Table 7.3: Empower Local People to Act * Empower Local People to Act Learning Group Low to Average Involvement High Involvement Total Low 9 45.0% 11 55.0% 20 100.0% High 9 19.1% 38 80.9% 47 100.0% Total 18 26.9% 49 73.1% 67 100.0% *Significant chi-Square: p < 030. Figure 7.3: Empower Local People to Act Empower Local People to Act 0 20 40 60 80 100 LowLearning GroupHigh Perc ent Low to Average Involvement High Involvement Table 7.5: Strengthen School Management Strengthen School Management Structures Learning Group Low to Average Involvement High Involvement Total Low 10 50.0% 10 50.0% 20 100.0% High 17 36.2% 30 63.8% 47 100.0% Total 27 40.3% 40 59.7% 67 100.0% 95 Figure 7.4: Strengthen School Management Strenghten School Management Structures 0 20 40 60 80 Low High Learning Group Percent Low to Average Involvement High Involvement Table 7.6: Use Participatory Planning Utilize Participatory Planning and Design Learning Group Low to Average Involvement High Involvement Total Low 6 30.0% 14 70.0% 20 100.0% High 9 19.1% 38 80.9% 47 100.0% Total 15 22.4% 52 77.6% 67 100.0% Figure 7.5: Use Participatory Planning Utilize Participatory Planning and Desing 0 20 40 60 80 100 Low High Learning Group P ercent Low to Average Involvement High Involvement 96 97 Table 7.7: Outcomes in Relation to Each Community Involved in the Site Visits Community High /Low QUIPS/ Control Increased Awareness and Interest in the School Community School Relationship Positive and Sustained Strengthened Local Leadership Ongoing Community Contributions and Sustained Action to Improve the School. Middle Ntiribuoho L/A High QUIPS Yes Yes Yes Yes Ntonso SDA High Control Yes Yes (mission) Yes Yes Adukrom L.A. Low QUIPS Yes Yes Yes Yes Kunsu Datiem L.A Low Control No No No No North Lonto Presby High QUIPS Yes Yes Yes Yes Ekumpe L/A Medium Control Yes Yes Yes Yes Nakpali Kworle Low QUIPS Yes No No No Tusundo L/A Low Control No No No No Kpanlori E/A High QUIPS Yes Yes Yes Yes Gbangu L/A High Control Yes Yes No Yes Kpalgun Zion Low Control Yes No No No Tuobodom Islamic Low QUIPS Yes No No No 7.3 KEY OUTCOMES OF COMMUNITY SUPPORT BY QUIPS Five significant outcomes of the QUIPS program emerged from the qualitative field work carried out by the evaluation team. These findings support the quantitative findings of community best practices discussed above. The qualitative findings attributable to the QUIPS program include: • Increased awareness and interest in, and commitment on the part of communities and parents to, the process of education. • Strengthened relations between teachers and the community. • Strengthened community leadership for improving schools, through the SMC/PTA and traditional leadership structures. 98 • Improved community support of schools via enrollment drives, community contributions, and maintenance of the school, including food support, books, and volunteer teachers. • Heightened sense of parental responsibility for their children’s well being and education. The findings from the evaluative field work reveal that QUIPS assisted the communities in increasing their awareness and interest in improving school quality and community￾school relations; these outcomes often were sustained after the project closed, particularly where local leadership was strong. QUIPS strengthened the internal leadership structures in the community and encouraged communities to contribute to and plan for school improvement. Community-focused interventions had a positive correlation with high-performing schools and enhanced learning outcomes in children. Most of the high-performing QUIPS schools, and pupils, had received considerable financial, human, and social support from their communities. 7.4 INCREASED PARENTAL AND COMMUNITY AWARENESS, INTEREST, AND COMMITMENT The key outcome for all the high-performing schools in every region was the contribution QUIPS made in creating awareness about community responsibility for the local school. For instance, in the south, the elders and SMC/PTA members interviewed confirmed that before QUIPS the “school block was collapsing and teachers were not at all punctual. During and after QUIPS interventions the head teacher and the teachers are doing well in teaching during school hours and after school.” The increased awareness of parents was demonstrated by increased parental support for education in all regions. Interviews with district stakeholders and SMC members revealed that in some communities, parents made more visits to the schools to meet with teachers and were more supportive in providing the basic needs of their children. Those interviewed attributed these changes to sensitization of community members to the value of education and the need to provide support to children by making school resources available and taking a direct interest in their children’s education. In QUIPS low-performing communities, parents often were neglecting the basic needs of their children, such as providing minimal amounts of food and reducing household chores before and after school. Interviews with children revealed that most parents were not ensuring that their children’s basic food needs were met before and during school. This undermined the child’s ability to learn. Findings from the field suggest that USAID's school feeding program (PL480) was an excellent complement to the QUIPS program because it ensured that most children were able to obtain a basic meal that gave them enough energy to learn effectively, thereby ensuring that QUIPS interventions were supported. At the schools the evaluation team visited, the children took lunch after they had spent the day in class and then went home. In one school, Kpanlori (Q-Hi-N), on two occasions the team observed pupils continuing their studies after lunch. 7.5 HEIGHTENED AWARENESS OF THE ROLE OF PARENTS IN A CHILD’S EDUCATION The most visible outcome in these communities was the increased awareness and commitment of parents to their children’s education; for instance, they organized evening classes where children would study together and receive extra tutoring from teachers in the community. Among the communities doing this were Kpanlori (Q-Hi-N), Gbangu (C-Hi-N), and Edukrom (Q-Lo-M). Usually the community would release the children for evening classes and provide a lantern with kerosene to support the community volunteers who would supervise the children as they completed their homework and reviewed their lesson notes. The field teams also found heightened interest and awareness of parents for their role in supervising school work at home and creating a conducive place for learning outside school. This awareness of the role parents’ play in their child’s learning may be the most important outcome of the QUIPS program.76 Several studies suggest that in Ghana not enough emphasis has been placed on, or support given to, the role of the parent. The QUIPS program made a significant breakthrough in this area: interviews with children, teachers, and parents themselves confirmed a new sense of responsibility and a visible increase in parental participation in their children’s homework activities; parents reduced the workload of their wards during school days so that they could study at night and be ready for school in the morning. 99 76 It may only be possible to judge how much impact QUIPS had in this area when QUIPS has been withdrawn for a few years. In the case of Kpanlori, where there were night studies, QUIPS had ended three years previous. The Story of Lonto In the Lonto community before QUIPS, the SMC later reported, the relationship between community and school was “hostile.” The administration was “lackadaisical” and the teachers dressed “anyhowly.” One of two pavilions was uncladden, teaching and learning were not effective and enrollment was very low. At that time, there was no SMC or PTA. Mothers reported that the children did not want to go to school (field report, Lonto School, Q-Hi-N). Interviews in Lonto revealed that during QUIPS the community got to know the District Education Oversight Committee and Circuit Supervisors who were compelled to visit the school. When QUIPS arrived the community provided labor to complete cladding a pavilion. They also visited their children in the school. The District Assembly provided the sand and chips to wall one pavilion and QUIPS funds helped wall the other. These behavior changes had significant impact in both QUIPS and control high￾performing schools. Increased parental support and interest in education also has a significant impact on girls’ education, particularly where socio-cultural patterns restrict girls’ full involvement in education. Some of these practices have been sustained in two rural QUIPS community-schools across nine QUIPS sampled sites, Kpanlori (Q-Hi-N) and Adukrom (Q-Lo-M). 7.6 INCREASED ENROLLMENT IN QUIPS SCHOOLS A controversial yet visible impact of QUIPS interventions was increased enrollment, particularly in the north. The combination of food assistance and emphasis on professional teacher development and learning had a tremendous impact on attracting and retaining children during the QUIPS lifespan. For instance, QUIPS and control schools both engaged in awareness-raising activities to encourage parents to send their children to school. The increased complement of teachers that district offices provided attracted more children, not only from the immediate school communities but also from nearby communities, sometimes to the detriment of schools there. 100 Kpanlori consists of two villages with a total of 1,600 residents who farm and raise animals. Houses are made of mud and thatch. There is no electricity, but there are two boreholes and a river nearby. The nearest health clinic is in Nakpanduri, two hours away. The language is Lekpankpal. During the first half of every year, the community does not have enough food. The donor-sponsored feeding activities at both the day nursery and the primary school are very visible ways to attract children to school and help them learn throughout the entire school session. Very few adults in Kpanlori speak English or are literate. In recent times, secondary school graduates from the community have been volunteering as teachers in the primary school; they are trained by the head teacher. The GES has not provided untrained teachers or textbooks since the QUIPS program ended three years ago. Parents sent the head teacher to Tamale to buy textbooks for all classes, with funding from a community levy. The head teacher and the grades 5/ 6 teacher are Certificate A holders and received QUIPS training in cohort 4 in 2000 and 2001. The primary school scored high on the learning growth tests used by this evaluation. Neighbors noticed the behavior of the teachers and the community at the Kpanlori school and now enrollment has swelled, with some classes up to 65 pupils or more. The most senior leaders of Kpanlori are driven to give the present generation a quality education. The chief and elders see that the quality of life of Kpanlori can improve if they can produce educated elite, like their neighbors of other ethnic backgrounds. Neighborhood competition is intense. All the resources of Kpanlori have been mobilized for more than a decade to support schooling. The community is seeing returns on its investment; for instance, a graduate of the teacher training college returned home to teach and is earning a viable income. Six of nine pupils who sat the Basic Education Certificate Examination (BECE) in 2004 passed and are attending secondary education now. Kpanlori, East Mamprusi District, Northern, rural Q-Hi-N, Cohort 4. 101 An interesting case of how food and the QUIPS interventions enhanced the learning outcomes of children is Kpanlori, where the CRS Food for Education program and the QUIPS awareness-creation activities sustained their impact after three years. The community continues to have SMC meetings every two weeks. The case of Kpanlori demonstrates that, where there is strong community leadership, led by the chief and elders, SMC, and PTA, QUIPS capacity-building can be sustained and affect a community’s ability to improve the quality of education. 7.7 IMPROVED COMMUNITY-SCHOOL RELATIONSHIPS In almost all the high-performing QUIPS schools, community/school relationships had visibly improved. In most cases when QUIPS entered a community this relationship at first was fragile, but through the concerted efforts of both community leaders and teachers, the relationship improved because of the commitment on both sides. Communities noticed teachers teaching with more commitment and reducing their absenteeism; teachers noticed communities working to improve the school infrastructure and prepare children to attend school by rehabilitating or building schools, underwriting volunteer teacher salaries, and providing exercise books and in some cases food. Adukrom (Q-Lo-M) Lonto (Q-Hi-N) and Kpanlori (Q-Hi-N) were places that maintained strong community-school relations even after QUIPS. They provided housing and other ongoing support for the teachers. The community/school relationships of QUIPS schools are described in Annex 6, Community Participation and Teacher Performance. Schools observed during fieldwork that had high participation by their community and high performance of their teachers, have behaviors in place to support pupil learning. There were two schools that were once marked high on learning performance a few years ago but now have receded. They no longer have the same level of community support and involvement. Two factors have emerged to explain this trend: the movement of QUIPS teachers out of the school and the lack of sustained support, leadership, and unity within the community. Community/school relationships did not improve in every case under QUIPS. Communities where there was limited leadership to initiate, spearhead, and maintain school improvement, found it difficult to sustain change. There were several cases where teachers demonstrated low commitment and a non-professional attitude to their responsibilities for educating the school children. For instance, in a control school in Ashanti teachers would leave on Thursday evening to return to their home towns and arrive back at school on Tuesday afternoon to start classes on Wednesday. In two QUIPS schools in the middle and northern region, teams found poor community- school relationships, particularly after QUIPS ended. In these communities, the teachers were absent or late to school and did not practice child-centered approaches. The continued caning and corporal punishment of children was reported by both teams in the north. 102 7.8 STRENGTHENED LEADERSHIP CAPACITY QUIPS strengthened community-based institutions such as the SMC/PTA by training their members intensively on how to manage their schools. Findings across the 18 research sites suggest that local leadership and parental involvement were major factors in improving schools. In high-performing QUIPS schools, field work revealed the importance of the QUIPS role in strengthening leadership institutions like SMCs/ PTAs and respecting traditional structures to ensure that interventions were community-led. Great care was taken in CRS interventions in the north of Ghana to work closely with local leadership to ensure that school improvement continued to be nested within their sphere of influence. For instance, in Ekumpe several teachers were also chiefs and the town chief was a former District Director of Education and the current Regional Director of Education. Ekumpe (C-Hi-N) recruited several volunteer teachers from the community who were excellent role models for the children. Several communities recognized that the SMC/PTA training they received gave them more control over the destiny of their children. Many expressed the concern that the world is changing and their sons and daughters need education to survive in a world different from that of their parents and grandparents. 77 The heightened involvement of community leaders in the school improved teacher discipline and reduced their absenteeism, making a direct impact on learning outcomes. 7.9 POWER OF COMMUNITY AWARENESS AND INVOLVEMENT Once a well organized community has discovered its “inner voice” and has had it validated by positive experiences, its members can make a difference in how schools operate and pupils learn. While only half the schools visited had benefited directly from QUIPS interventions, because all SMC/PTA members were trained through the program, the evidence of greater change at the community level than in some QUIPS schools is understandable. The composition of the community and SMC/PTAs varied little in the time since QUIPS ended, although there were major changes at both school and district level. All communities were clearly aware of their role in education. Besides providing tangible support to maintain and expand school facilities, parents have learned that school visits not only keep teachers and children on their toes but also are appreciated by the children. “When I come to visit and my daughter sees me she points to me and tells her friends and she is proud that I have come.”78 77 Interview with parents in northern Ghana. 78 Interview with Mother, Nakpale Kworle, QUIPS School, northern Ghana. 103 All the communities visited talked about support to schools and demonstrated, understanding of their roles in ensuring quality education for their children. The participation of community members ranged from: • Communal labor to augment food when CRS supplies did not come. • Support for teachers by helping them with housing or food or by raising money to support volunteer teachers. • Providing children with their basic needs, both tangible (e.g., exercise books, pens, helping them to move quickly in the morning to school) and psychological (supervising homework, encouragement, rewarding school success). All communities spoke positively about teachers and the work they were doing under difficult circumstances, and most complained about the scarcity of teachers, books, and other teaching aids. One SMC/PTA reported having complained to the DEO about a teacher considered detrimental to the school and others and having requested that poor teacher attendance be addressed by district officers. Findings from northern Ghana suggest the importance of the chief, his elders, and other community opinion leaders, including the Assemblyman, to school improvement. This was very clear in three of the more successful school/communities visited, Lonto (Q-Hi￾N) Ekumpe (C-Hi-N) and Kpanlori (Q-Hi-N). Not only does the quality of leadership affect how the community operates, it has impacts, direct and indirect, on the school. Thus it is critical that all interventions work through the chief and elders to ensure the community support necessary to move forward. By contrast, in QUIPS low-functioning communities the chiefs played a limited role and school performance was low also. 7.10 INCREASED COMMUNITY CONTRIBUTIONS TO SCHOOL IMPROVEMENT AND SCHOOL PERFORMANCE IMPROVEMENT PLANS There is no doubt that the QUIPS inputs and strategies throughout the country translated into a stronger community process for improving schools. Across the 18 sampled sites there were examples of communities doing the following: • Raising funds for books and teaching aids; • Supporting payment for teachers by raising levies; • Making contributions to supplement the food provided to school children; • Setting up committees; • Providing labor and materials for constructing school classrooms, teacher bungalows, and latrines and toilet facilities; and • Monitoring the attendance and punctuality of both pupils and teachers. Apart from the heightened participation of communities in the life and improvement of their schools, during the program almost all the QUIPS schools finished implementing a school improvement plan, which helped the communities define their priorities and 104 make plans to achieve change within their schools. Unfortunately very few communities continued with their SPIPs once QUIPS ended. Only one community had done so: Kpanlori, a high-performing community in the north. Some of the sampled communities continued to support their schools sporadically but few had maintained regular SMC/PTA meetings or consultations in a systematic way. Five communities, however, stood out for their systematic support for their schools: Dunkwa (Q-Hi-S), Adukrom (Q-Lo-M), Lonto (Q-Hi-N), Kpanlori (Q-Hi-N), and Ekumpe (C-Hi-N). All these communities had either supportive agencies, such as a church group, or a strong head teacher and teachers driving community support for the school after QUIPS ended. 7.11 COMMUNITY-LEVEL CHALLENGES TO SUSTAINABILITY The communities that described high levels of community/school interaction and strengthened community support during QUIPS were not always able to sustain these outcomes. Watereso was a good example of a community with high community-school interaction during QUIPS but its SMC/PTA had not met in over two years. Such a finding differs from the final reports from the QUIPS implementing partners, which suggested a high likelihood of sustainability within the project sites (CSA final report, 2002).79 The importance of sustained and indigenous leadership to improve and maintain the community-school relationships was a theme that emerged from the fieldwork in all 18 sites. Where there was a “community champion” who believed in the importance of supporting the school, QUIPS not only added value to ongoing interventions at the school level but the community champion became a key to ensuring sustainability. How the champions operated and who they were took different forms. In Dunkwa, the champions were local church leaders; elsewhere, they were traditional leaders such as chiefs and elders; and in other cases they consisted of a few members of the community, parents or SMC members. To ensure that community-level interventions and structures were sustained required something from within the community itself. 79 In its review of the draft evaluation, Education Development Center stated: “To clarify, results presented in the CSA final report were based on external data collection, which supports that these things were sustained, at least in the period when CSA was collecting data. It may be helpful to review the out year data for the communities, if this has not been done. It is possible that the evaluation team went into the communities later.” Lonto after QUIPS Although QUIPS has left the community, school stakeholders in Lonto said that parents still visit the school regularly and the SMC/ PTA meets at least twice a term—more often if there is something special to discuss. Once issues are discussed by the SMC executive, they are usually presented to the community for action. The School Performance Improvement Plan is no longer evolving—the community does not see the need because there are no resources to allow activities to be carried out—but it was able to complete the SPIP drawn up during the QUIPS period. The story of Lonto suggests that the people interviewed had a need to see results if they were to stay engaged with their school and continue planning for it. The role of the chief and elders was critical to sustaining change and encouraging action and a positive attitude of the community towards its school. 7.12 CHALLENGES, SUSTAINABILITY, AND CONCLUSIONS The greatest shifts in community behaviors that suggest responsiveness to QUIPS training and support were increased community support for school quality, support for girls’ education, strengthened school management, and participatory planning and design. These four best practices were found to create change across all the QUIPS communities during the QUIPS intervention. The community practice that appeared most resistant to change was transparency (or rather, the lack of it) in financial management and the ability of communities to effectively monitor and account for school funds. Quantitative analysis also revealed that community performance and learning growth of pupils were systematically related in five of nine focus areas: community trust in the school system, support for quality education, empowerment of local people to act, school management support and participatory planning and design. The study findings suggest that high community performance was more likely in schools where the learning growth pattern of pupils fell also in the “high” category. Involvement of local leaders and parents was a major factor in improving school quality and children’s learning. For the first time in Ghana, a program was able to reach beyond simply raising awareness to ensure the direct participation of parents in their child’s learning. The QUIPS community intervention strategies and school-based support encouraged parents to get involved in their children’s learning by offering evening classes, reducing the children’s workload, and ensuring that their basic needs were filled so that they could learn. 105 106 Parents began to feel a sense of responsibility for the education of their children, sometimes for the first time. This was a significant step for parents who themselves had not been formally educated and were not always confident about helping their children. Several interacting factors played a role in this heightened sense of responsibility and ensuing action, among them SMC/PTA training, community dramas, and more home visits by teachers and community facilitators. The empowerment of communities was a sustainable change, particularly where there was a local champion, a leader who took up the cause of improving education. Where there were no local champions, community empowerment and participation in school improvement declined after QUIPS ended. CHAPTER 8: OUTCOMES AT THE DISTRICT AND NATIONAL LEVEL This chapter discusses how QUIPS operated at the district and national levels and what the results were. 8.1 IMPACT OF QUIPS ON DISTRICT EDUCATION OFFICES QUIPS had considerable visibility at the district level from start to finish. 8.1.1 Key Inputs and Intervention Strategies. QUIPS district programs centered on training: • Training of district staff, including statistics officers, inspectorate officers, girl child officers, circuit supervisors, and district-monitoring assistants; • M&E training for the district statistics and M&E teams; • Teacher training-of-trainers for circuit supervisors, the Inspectorate Division and, girls’ education officers, who were often members of the District Teaching Support Teams (DTSTs); and • Management, planning, and financial management training for the District Director of Education; Assistant Directors (ADs) of Supervision, the Inspectorate, Finance and Administration, and Human Resource Development; and Accountant and Budget Officer. QUIPS also wrote and distributed manuals and documents to support district-based training on management, monitoring and evaluation, and strengthening of SMCs and PTAs. A significant amount of information was also provided to the DEOs on improving the teaching of reading, English as a second language, and math, as well as manuals on how to prepare lesson notes and manage a classroom. Although some of these materials came at a very late stage in the program cycle, they were intended to help the DEOs to spread activities to non-QUIPS schools and teachers. 8.1.2 Key District Outcomes. The evaluation team identified six key QUIPS outcomes at the district level: • Improved supervision and monitoring of QUIPS and other district public schools; • Enhanced access to information technology for organizing, storing, and analyzing data; • Enhanced teacher support services and development through DEO training and monitoring; 107 108 • Strengthening of some district structures and processes such as the District Teacher Support Teams, the District Management Implementation Teams (DMITs), and the M&E process;80 • Enhanced appreciation of the role of the community participation process and ultimate creation of a new district post called the Community Participation Coordinator (CPC); and • Strengthening of SMCs, PTAs, and other local community structures. (The structures mentioned are discussed more fully in Section 8.2.1.) Concurrent with QUIPS, it is important to note, other district-directed initiatives, such as the Whole School Development program (WSD) and NGO-supported programs (e.g., Ibis, ActionAid) were operating. Thus the outcomes cannot be attributed solely to QUIPS interventions; their synergy with these other programs must be considered. The most visible effects of QUIPS interventions were at the district level. In all the districts visited, DEOs gave specific examples of how QUIPS-inspired capacity development, enhanced management procedures, and improved M&E had strengthened efforts to improve teaching and learning. 8.1.3 Background to the Entry of QUIPS at the District Level. QUIPS was designed to strengthen existing systems within the GES and enhance teacher professionalism in the classroom. Before QUIPS, GES in-service training of teachers at the district level was limited and not well funded. District personnel were not used; most of the trainers were drawn from teacher training colleges and GES headquarters. Even though the DEOs had subject matter specialists, because they were rarely involved with and thus not familiar with the content of the training programs, they could not follow up with supervision at the schools. GES had mandated the formation of SMCs/PTAs in 1995 but there were no training programs for them before QUIPS. SMCs and PTAs were confused about their responsibilities within in the community and in relation to the school. Relations between many communities and schools also were not conducive to improving the school environment for learning. Supervision of schools was described as “policing” teachers and inspection was seen as “fault finding.” Because there was little financial support, school supervision was carried out “once in a blue moon” (DDE)—officers visited schools only when there was a problem. Before QUIPS, the administrative structures of the district offices were described as hierarchical. Planning and decision making were conducted by the District Director of Education and the four frontline Assistant Directors, but only rarely; DEOs were given instructions but there was no consultative planning and most district-based activities had no budgetary allocations from Accra. 80 The production of a Circuit Supervisors’ Handbook and documentation on the Monitoring and Evaluation process were designed to help DEOs to effectively fulfill their roles. 109 After the QUIPS mid-term review, the District Grant Mechanism was set up to support the spread of good practices emerging from the QUIPS model schools and communities. At this stage, a significant amount of training was given to DEOs to enhance their capacity to spread the program. 8.1.4 Teacher Deployment, Training, and Support. One of the criteria for selecting QUIPS schools was that a school should have a full complement of teachers, except that in the north where there were very few trained teachers the criterion was reduced to one trained teacher and enough untrained teachers81 to reach the full staffing requirements.82 Interviews with district officials revealed that the untrained teachers were often not able to benefit much from the QUIPS training because they could not keep up with the context. Untrained teachers made up most of the teaching staff (40% - 60%) in the primary schools in the north.83 One District Director of Education explained that “untrained teachers in the north are mostly those who could not pass the Senior Secondary Certificate Examination, so if they are put together with trained teachers to undergo the same technical training in teaching, the assimilation level will by all means not be the same.” However, the training did equip many trained teachers with the necessary skills to perform well in the classroom. Still, the challenge of training such a mixed group was daunting for some district trainers. 8.1.5 Teaching staff Imbalances within the District. Because teachers often were transferred out of rural schools in order to fulfill the criterion of six trained teachers per project school, QUIPS created an imbalance in the numbers of trained and untrained teachers serving schools across the district. In evaluation interviews, DEOs reported moving trained teachers from already disadvantaged schools to the schools identified for QUIPS support. Not only did other district public schools not benefit from QUIPS training, they ended up with fewer trained staff. On the other hand, the QUIPS community schools benefited from both the deployment of GES trained teachers into their schools and from teacher training. The administration of QUIPS schools improved because the head teachers were also trained. As more teachers were given quality training with QUIPS funding, the attitude of pupils and parents toward the schools changed. QUIPS was designed to encourage district staff to give special attention to the instructional needs of the three selected QUIPS schools. The improved human and physical resources of the QUIPS schools attracted higher 81 In Ghana untrained teachers are also known as pupil teachers. 82 In its review of the draft evaluation, Catholic Relief Services stated that “for a school to be selected it was to have at least two trained teachers and four pupil teachers (ideal). Where there were only three teachers, at least one should be trained.” 83 Ghana Living Standard Survey and Casely-Hayford, 2001, 2004. 110 enrollment from the surrounding communities—which meant, in turn, that these schools then had even fewer resources and less attention from district offices. 8.1.6 Supervision and Monitoring. At the early stages of QUIPS, Circuit Supervisors, District Directors of Education, and a few Assistant Directors were involved in training and field work. After attending the teacher training, they were expected to supervise QUIPS teachers in between the scheduled school-based quarterly teacher training sessions. Until QUIPS funded the purchase of fuel for supervisory trips, little supervision took place. District staff did not take the lead in instructional training during the initial phases of the program but they were involved in the PRA/PLA needs assessments and in baseline studies in the communities. Once the DEOs were trained, a new pattern of behavior and attitude toward teacher supervision emerged; the approach changed from policing or fault finding to support and encouragement. The frequency of Circuit Supervisors and DEO visits to QUIPS schools increased and thus helped to augment teaching and learning in the classroom. Lack of funds at the district office often restricted the frequency of DEO visits to non￾QUIPS schools. As one officer noted, “Often visits to non-QUIPS schools were limited to just saying hello and signing in the log books to show the officer was present. Most importantly, very little was known of these schools because reports… were very sketchy.” Since the GES was not regularly providing DEOs with adequate funds for supervision, QUIPS created another imbalance within the system by providing monitoring support (funds for fuel) for the three schools. That focusing a high level of resources and investment in a few schools within a district created imbalances was one lesson learned from the QUIPS program. At the same time, without such inequalities, the quality of instructional improvements seen under QUIPS would not have been possible.84 In addition to the increased presence of GES officers, frequent visits by QUIPS officers also contributed to the improved performance of QUIPS schools. According to a District Director of Education in the northern region, “The regular visits of Circuit Supervisors and other officers to the QUIPS schools helped [keep] head teachers and teachers . . . on their toes.” Besides visiting QUIPS schools more often, GES officials also taught refresher courses and gave demonstration lessons to assist the teachers. This resulted in improved attendance and punctuality of teachers, who were thus better prepared to manage and apply the QUIPS child-centered methodologies. The written assignments teachers gave pupils also increased across most of the QUIPS schools visited. In 84 In its review of the draft evaluation, Catholic Relief Services stated that it “disagree[d] that too much support was devoted to QUIPS and that marginalized non-QUIPS schools. QUIPS only gave the minimum of support needed for supervision. In most cases (at least in the North), QUIPS schools were rather denied normal GES support during the implementation. As it was assumed that they were receiving ALL support from QUIPS. The experience proved to CRS that with minimum support and adequate motivation, GES staff has the capacity to achieve desired results.” 111 simple terms, QUIPS schools were given more supervisory attention because there were resources to fund these activities. The learning atmosphere in QUIPS schools was enhanced in comparison to control schools. There was more competition to excel among pupils and even teachers. The quantitative findings, which reveal increased learning growth and learning outcomes among pupils within QUIPS schools can be attributed, at least in part, to the greater support by the DEOs and the increased discipline and performance generated from the range of QUIPS community and classroom interventions. 8.1.7 SMC/PTA Training. The training of SMC/PTA members and other members of the community was designed to build local capacity. This helped rural communities realize their responsibilities to the school and the education of their children. Interviews with district officers suggested that communities were encouraged to begin taking on more school responsibility and management. They became more aware and active; some communities began to pay their own volunteer teachers to fill vacant teaching positions, (Kpanlori, Q-Hi-N). Some communities also began buying storybooks and readers to ensure that their school children had the necessary learning materials.85 Before QUIPS, SMC/PTAs had been set up in many schools, as required by GES, but were given little or no training. The head teachers or the chiefs and elders often selected the PTA members. There was often poor rapport between the community and the school before the QUIPS program. Consultations to discuss problems were infrequent and teachers made few visits to the communities. SMC/PTA members in QUIPS schools were selected after a four-day series of participatory activities within the communities. SMC/PTA executives were selected in line with GES membership criteria and introduced to a more democratic approach to school governance. Members were taken through a series of training sessions and were supported in their community activities by the District QUIPS Facilitators. Their new awareness empowered members to plan together for their schools using consensus and consultation with DEOs. GES officers visited QUIPS communities regularly to promote awareness. The recruitment of volunteer teachers, communal labor to improve infrastructure, and activities to promote the welfare of teachers and pupils were carried out in almost all the QUIPS communities. SMC/PTA training spread throughout the district and was one of the key outcomes of the QUIPS program. (See Section 8.1.2.) 8.1.8 Infrastructure. Provision of school buildings, furniture, and latrines were directed to the QUIPS schools, further stimulating shifts in pupils away from more deprived schools to QUIPS schools. This pattern emerged across all the regions evaluated. 85 Participatory Learning Appraisal (PLA) and drama were used to stimulate consultation on key educational issues. 112 The school infrastructure grants had a condition attached: communities were to make contributions so that their projects could be completed. The grant activities usually supported the construction of three -classroom blocks for the three QUIPS schools that qualified in each district. According to the DEOs, this arrangement created inequities within the district; the well-endowed communities made greater contributions and completed their infrastructure projects, sometimes adding more rooms and amenities to the school facility because community members contributed more. The extent of poverty, particularly in the north, was such that a more equitable approach to infrastructure development was needed to ensure that all communities benefited; communities in the north could not contribute as much as those in the south, so some of the QUIPS communities could not finish even the three-classroom block, such as Nawuni (Q-Lo-N). Almost all communities visited in the middle and south QUIPS communities completed not just the three -classroom block but often built a six-classroom block with urinal, toilet and office facilities. (See Table 6.14.) 8.1.9 Strengthening of District Education Offices. The District Management Implementation Teams, M&E teams, and the Community Participation Coordinators (CPCs) were put in place by QUIPS; they were new to the DEOs.86 The M&E teams had in some cases been dormant but were reinvigorated through the training QUIPS provided. The training plus the financial injections enabled the district offices to increase their visits to school- communities, give teachers better quality support, conduct in￾service teacher training, and sensitize communities to their rights and responsibilities. The school needs assessments M&E teams carried out also supported needs-based training for schools and communities. It was targeted at specific issues of concern for both groups. Targeted training and using existing data to help solve problems and track change were new practices for the DEOs. Field reports from the evaluation teams revealed that the skills imparted to DEOs through the QUIPS training enabled them to carry out a great deal of in-service training for both teachers and communities and helped them reform district administrative and decision-making procedures. However, the reports also suggest that few of these activities have been sustained. Interviews with the District Directors of Education in the four regions evaluated suggest that since QUIPS funding terminated, the GES has not 86 In its review of the draft evaluation, Education Development Center stated that it: “would like to clarify the issue of M&E teams, which is addressed in the evaluation report in Section 8.1.9. CSA did not form M&E teams. In fact, CSA tried to discourage the formation of these teams. CSA could only train 4 people in each district, and these four were trained with the intention that they would train others in the district. The districts themselves began referring to those CSA trained as the M&E team. CSA also included the larger DMIT in all the onsite training to convey the message that everyone should be involved in M&E. The direct trainees were an advanced small group of people who could support the M&E function, but should not be considered as district M&E teams.” 113 allocated budget to support continued training of teachers, communities, or SMC/PTAs. Moreover, such district education structures as the District Management Implementation Team, Community Participation Coordinators, and M&E teams “are not functioning at the level they were during QUIPS, even though they still exist.” For example, Circuit Supervisor visits to schools to support teachers have been irregular or nonexistent since the QUIPS program ended. Reports from district officers also suggest that planning is no longer consultative. However, the skills the DEOs learned allow them in some cases to effectively influence planning. For example, all the district offices adapted the QUIPS modules from the District Grant process, with important inputs from the M&E teams. According to an official at GES headquarters, who was echoed by many officers during interviews with the districts, “The benefits are not being sustained. The officers and teachers trained were hard working when they knew people [from QUIPS] would come and see them. Supervision to schools came more frequently because they had money and fuel and the QUIPS people also came. Now this is not happening and for many it is easier to slip back into our old ways if no one is there to push.” 8.1.10 Summary of DEO Impacts. Apart from injecting a new spirit of administration and management at the District Education Office, funds provided by QUIPS coupled with training programs enabled District Education Officers to train teachers and community members in new approaches to solving problems within their classrooms and communities. The availability and regular flow of resources and technical assistance to the District Education Office was a great relief to often severely under - resourced district offices. Nonetheless, some resource imbalances resulted from the focus on a few key schools within the district. More cohesion and synergy among QUIPS district capacity-building interventions and other bilateral project interventions would have reduced the burden on district officials and ensured more continuity and sustainability after QUIPS closed.87 8.2 DISTRICT-WIDE OUTCOMES This section presents the main outcomes related to the district-wide interventions that were implemented after QUIPS began its strategy to spread to other schools within the district. Some of the key interventions introduced after the mid-term evaluation to spread the best practices of QUIPS to other schools were: • The creation of structures to support the spread of QUIPS best practices to other schools (e.g., District Management Implementation Teams, District Teacher Support Teams, M&E Teams, and Community Participation Coordinators); 87 Officials in all districts complained of the intensity of district-based training, which often brought their continuing district responsibilities to a standstill during QUIPS implementation. 114 • Training of District Officers in M&E and community mobilization; and • Introduction of the DGM, supported by training. Field interviews suggest that the DEOs did not become fully involved with QUIPS until after the mid-term evaluation and the start of the DGM, which involved an effort to include a broader array of District Officers as co-trainers in QUIPS outreach efforts. 8.2.1 District Structures before QUIPS. The administration of the district offices was the sole responsibility of the District Director of Education, the four-frontline Assistant Directors, and to a lesser extent the accountant. Most other staff did not know the plans, programs, or vision of the district offices. As one district officer remarked, “You only receive instructions and you are to carry it out. Information came only from the top.” To enable District Education Officers to implement the grant program, it became necessary to train staff at the district to spread QUIPS good practices. The structure created for this was the District Management Implementation Team, which was responsible for operating the QUIPS-funded District Grant Plan. The District Teacher Support Team was responsible for teacher training. The Community Participation Coordinator was responsible for sensitizing the communities and training SMC/PTA members. The M&E team was responsible for monitoring progress against the District Grant Work Plan.88 All these officers were given training. According to interviews, the District Grant proved to be both innovative and effective in strengthening the DEOs by heightening their activity in the school communities. The QUIPS District Grant Mechanism also enabled districts to plan and implement more in-service teacher training and supervision. The QUIPS participatory approach engaged the DEOs in the change process, which transformed their mode of operation. Several district officers interviewed remarked that QUIPS moved their mode of operation from a hierarchical approach to a more horizontal management. The District Director of Education worked through the District Management Implementation Team; this made possible frequent office-wide meetings with members of the District Teacher Support Team and M&E team, the Community Participation Coordinator, and Circuit Supervisors. Besides lacking trained teachers, most districts in the north have never had the full complement of district staff; as a result, the same officers often had to be trained for several different tasks required for the grant mechanism. Field reports from the 88 Two of the four M&E members were given computer training in the hope that they would be able to train other district staff. 115 evaluative teams revealed that overlapping responsibilities were common in all the districts but were much more pronounced in the north.89 The project officials–master trainers responsible for different aspects of the QUIPS training programs—were from the implementing agencies. They were at first responsible for training teachers, community members, and SMC/PTA, depending on the larger responsibilities of each agency. Duration of training, processes, capital input, and monitoring tools were all changed when GES trainers took over after the mid-term evaluation. At that point, “Things were so rushed that sometimes the people were unable to grasp what was being taught, considering the time limit and number of schools under the grant program” (DDE interview). Officials in GES headquarters remarked that “the program was too ambitious.” The movement of staff and personnel depended on decisions made at the national level, which often affected the outcomes of QUIPS training and capacity-building within a district. In many districts, activities of the District Management Implementation Team, District Teacher Support Team, Community Participation Coordinators, M&E and to a lesser extent, the Circuit Supervisors were reduced when the QUIPS program ended in the district—mainly, according to the DEOs, because funds were short or staff members were transferred. Teacher and SMC/PTA training, community sensitization, M&E, and supervision were all affected. 8.2.2 The District Grant Mechanism. The decision to spread the good practices of QUIPS, based on the mid-term review, gave birth to the district grant mechanism. The approach of the grants program had a number of limitations. The DDEs that the evaluation teams visited contended that the three schools in the program could not effect district-wide change because the resources of the QUIPS schools were so much greater than those of the average district school. So the strategy became to replicate only a select few of the best practices through the grant mechanism. Secondly, instead of the original trainers (QUIPS program officers and other national trainers), district officials were taught to themselves conduct the training of teachers and community leaders in the QUIPS program. Even though each district was able to handle an additional 20 schools during the spread period, the officials reported, resources were not always adequate to spread good practices to the new schools. Moreover, infrastructure projects, such as school buildings, furniture, libraries, recreational facilities and latrines, were not included in the grant mechanism, which was limited to teacher training and community sensitization. 89 The northern evaluation teams had to combine meetings of the District Management Implementation Team, District Teacher Support Team, M&E Community Participation Coordinator and Circuit Supervisor because district offices were understaffed and same senior officers were serving on several district structures at the same time. 116 Finally, the duration and intensity of teacher training and community sensitization was limited compared to what had been provided for the three QUIPS schools. External monitoring, which included support from resident district facilitators, was not included in the spread phase of QUIPS activities either. The evaluation teams found little M&E reporting available on this phase of project activities at the district offices other than financial reports. 8.2.3 Capacity Development. QUIPS DEO capacity building was carried out in all 110 districts in Ghana by national trainers, most of whom using the same instruments.90 Training manuals for each team were written by project and GES staff drawn from national, regional, and district levels. These officers were given intensive training under the direction of specialists. The national trainers were then employed to conduct district￾based training using both cluster and on-site visits. The statistics officer and the M&E unit received residential and on site training from the implementing partners over the life of the project. The context included program planning and measurement skills as well as the traditional data management competencies needed for the annual school census, inventories, etc. The M&E team was also trained in the needs assessment for the instructional and community awareness initiatives, which eventually led to training within the DGM. The M&E team was responsible for monitoring the implementation and impact of the training and technical assistance funded under the DGM. It appears from interviews at the district level that the team members were so overloaded with training duties that they postponed the work of monitoring courses given by their colleagues. Thus it appears that the district grant management program was not monitored effectively. No reports were found in the district offices where the district grant management training that had been conducted.91 Late in the QUIPS project, all but one M&E team received a computer and training in managing databases. The computers had a positive impact on the districts’ ability to analyze and store data and to conduct official correspondence. 90 While in the north, CRS handled all training, elsewhere three main agencies were involved in the capacity building: ILP trained the District Teacher Support Team; CSA handled the SMCs/PTAs/Community Participation Coordinator, and EARC/CSA was in charge of M&E. 91 In its review of the draft evaluation, the Education Development Center stated that: “the final sentence [of this paragraph] states that no reports were found in the district offices where the district grant management training had been conducted. This is surprising, as CSA worked with every district in workshops to prepare reports on the data that was collected. The district reports were one of the remarkable outcomes of the project. It is true that due to available time and resources, the reports were not as comprehensive as CSA would have liked, but there was monitoring and reviewing of results.” 117 8.2.4 New Patterns of Behavior. All the districts visited felt the impact of QUIPS. The DGM management training was seen as effective and relevant to the daily work and needs of the DEOs. The gradual conversion of the pattern of management from hierarchical to horizontal helped the district offices to introduce more participatory approaches to planning, which heightened the sense of responsibility by staff as well as their level of performance. The evaluation teams also found that once QUIPS closed, participatory planning meetings in district offices were reduced. Some officers complained that management was reverting back to the “old approach,” with the District Director of Education and the four Assistant Directors handling it all. However, field reports revealed that the M&E teams continue to have a role in the district planning process. The emergence of the CPC also has helped some districts to encourage expanding the role of communities in improving and managing schools. Table 8.1: District Sustainability and Spread of QUIPS Activities DISTRICT GRANT PROGRAM SPREAD NO. DISTRICT TEACHER TRAINING SMC/PTA TRAINING C’MNTY SENSITIZ ATION. M & E ACTIVITIE S TEACHER TRAINING SMC/PTA TRAINING C'MNTY SENSITIZ ATION M & E ACTIVITIE S 1 TOLON KUMBUNGU √ √ √ √ 2 EAST￾MAMPRUSI √ √ √ √ 3 EAST￾GONJA √ √ √ √ √ 4 ZABZUGU￾TATALE √ √ √ √ √ 5 KWABRE √ √ √ √ √ √ √ √ 6 AHAFO ANO￾SOUTH √ √ √ √ √ √ √ 7 TWIFU￾HEMANG LOWER DENKYIRA √ √ √ √ 8 DUNKWA ON-OFFIN √ √ √ √ 9 TECHIMAN √ √ √ √ Some districts in the south and middle sectors of the country, such as Kwabre, Techiman, and Ahafo Ano South, have solicited funds from the District Assembly and other sources to continue replicating QUIPS good practices. For instance, Techiman District managed in 2005, to secure 9.7 million cedis ($1,050) to continue supervision of schools within the district. There are also reports that districts have replicated the materials developed by QUIPS for non-QUIPS schools. Table 8.1 above summarizes the type of activities, which have been sustained in the sample districts for the 118 evaluation visit. Kwabre and Ahafo Ano South districts in the middle belt have used their resources to spread the QUIPS good practices after the termination of the program. 8.2.5 The Relationship between QUIPS and the GES. Interviews with high-ranking GES officials suggest that the QUIPS design did not take into full consideration the needs and capacities of GES as implementer. Support for the project superseded considerations of whether GES could absorb the approaches proposed and fulfill the commitments required by the project. One GES officer interviewed at headquarters in Accra said that, “GES was not party to, nor was it consulted in, the original design of the QUIPS program.” According to officers at the Basic Education Division: “USAID made sure GES people were there to re-package the proposal…. without looking at GES’s own programs to make the project fit in. GES officers who were needed for implementation were taken especially at the district level, without necessarily informing the District Director of Education.” Officers interviewed at the national and district levels stated that the QUIPS program was “imposed on the districts, and due to the bureaucratic structure, the District Directors of Education could not object to the usage of their staff, which drained many of the offices, but some did complain to the GES Headquarters, Accra.” This pattern of project led intervention was confirmed by headquarters officials. According to a District Director of Education in the north, “The DDEs were even initially not considered (or consulted) for the various trainings and workshops. Their inclusion was only during the training for the District Management Implementation Teams. By virtue of being chairmen, they automatically qualified to be members of the team to be trained.” Some of the DEOs interviewed resented the QUIPS officers who visited the districts because they saw them as “invaders.” They preferred working with other education development partners, who gave them recognition, leadership, and respect. According to one officer at GES headquarters, the relationship between QUIPS and District Director of Education was “like an adult walking with a toddler— the toddler cannot keep up but the parent is in a hurry. It was a difficult relationship with QUIPS in the drivers’ seat.” Another high-ranking official put it this way: “[USAID] have their own procedures and it’s difficult for them to release full control … they want to lead in everything when the MOE /GES should actually be taking the lead.” The QUIPS program was implemented in a context of meager resources and often￾insufficient district staffing. Some District Directors of Education interviewed resented the fast pace of QUIPS implementation. For example, officers selected for M&E training in some northern districts withdrew from QUIPS training programs saying that they were tired. Other officers such as Statistics Officers and some Assistant Directors of supervision withdrew complaining of the additional workload as well as the pace. 8.2.6 Challenges to District Sustainability. While it was widely reported that QUIPS had a positive effect on district offices, there are a few important challenges 119 associated with its sustainability. Most DEOs saw the choice of only three schools per district as a significant drawback. Those interviewed made it clear that supporting three to five schools in a district during the first few years was not enough to help districts move forward with new approaches to improving quality in the classroom or effectively replicate good practices across the district. The two-year intervention cycle in each school was also seen as too short to sustain change within the community/school. Apart from these design limitations for sustaining change within the district, there were internal and contextual limitations, such as the constant transfer of teachers and district officers, poor retention of “trained“ teachers, particularly in deprived areas, and lack of support for change from GES headquarters. The DEOs faced other constraints imposed from Accra, among them: (1) limited resources and delayed arrival of funds at the district offices; (2) a decision-making process that restricted District Director of Education control over major decision making at the district level, such as hiring untrained teachers; and (3) poor conditions of employment for teachers and DEOs. Interviews with DEOs suggest that broader consultation was necessary to ensure that QUIPS design and implementation was both effective and relevant given the contextual constraints throughout Ghana. The project was not imbedded within these realities; if it were, it would have implemented its programs in harmony, at a pace that took into consideration the realities of people on the ground. In fact, DEOs felt constrained, had very little room to maneuver, and simply followed the lead of the project implementers. The Evaluation Team recommends that future USAID programming should: • Place greater stakeholder participation, particularly from GES headquarters and the districts in the design of a QUIPS-type program; • Provide districts with more resources for effective M&E activities; • Give more grants to support the provision of textbooks and reading materials; • Include the provision of teachers’ quarters in infrastructure provision, to encourage teachers to stay in the communities. 8.2.7 Key Conclusions. The spread of QUIPS best practices through the district was an improvement on the original concept for QUIPS. It enabled the districts to take more control over QUIPS interventions by equipping them to improve school quality and community participation. The limiting factor was that the approach did not convince district or national stakeholders that they were truly in the driver’s seat to effect educational change. Interviews across the country revealed that although QUIPS structures were put in place and training was delivered, the subtle human relations needed to ensure ownership by district and higher-level GES officers were lacking. This latter ingredient, although not quantifiable, is often the key to success over the long run. The fast pace and results-oriented work of the implementing partners were among the reasons why this was not achieved. 120 8.3 NATIONAL AND INTERNATIONAL PERSPECTIVES ON QUIPS OUTCOMES This section presents the main outcomes of the QUIPS program from a national perspective, especially in relation to the GES. 8.3.1 Recognition of the Need for Community Participation in Education. Those surveyed by the evaluation team recognized that QUIPS efforts generated community awareness that SMCs and PTAs made a renewed contribution to the performance of pupils, teachers, and the school as a whole. Activation and reinforcement of community leadership allowed rural high-performing schools to overcome severe structural and contextual constraints on pupil learning. The GES headquarters recognized the role of the community in the learning outcomes and academic performance of pupils by making the DEOs responsible for annual training of the SMCs/PTAs. During QUIPS, the Director General of the GES wrote a memo to all DEOs establishing the post of Community Participation Coordinator, the CPCs. A number of District Directors of Education have continued to support this position since QUIPS ended and District Assemblies are supplementing funds from the GES in an effort to support and train SMCs/PTAs. Additional structured support for SMC/PTA development is recommended to GES headquarters and to USAID. 8.3.2 Support for Decentralized Decision Making. The original QUIPS design included efforts to improve policies for quality primary education, such as increased non-salary recurrent expenditure and devolution of resources and responsibilities to the district level. But QUIPS technical assistance to headquarters was limited to M&E, and to curriculum development along with short-term institutional support through non￾project assistance.92 At the QUIPS mid–term review, NPA was reallocated to the DGM in an effort to demonstrate policy reform at the district level, from when it could be communicated to headquarters. But communication to headquarters about the new patterns of behavior - engineered through the DGM was not systematic and often was not acted upon. 8.3.3 Implementation Challenges of QUIPS and Lessons Learned. In many cases GES headquarters was not aware of the changes taking place in communities, schools, district offices, and training colleges across the country during the QUIPS era. The set-up, meager finances, and high personnel turnover at headquarters did not encourage quick decisions to reinforce the changes. Evaluation interviews suggest that USAID, the implementing partners, and MOE/GES did not fully collaborate or even reach effective agreements on the design, redesign, 92 NPA gave GES headquarters the opportunity to work on innovations in pursuit of a tranche of funding from USAID. The GES proposed and designed a personnel evaluation system, but set a level of usage that was not realistic. The result was that USAID could not accept the results or release the tranche, though ultimately a partial release was secured. 121 and implementation of QUIPS. Some interviewees were cynical about the USAID’s "re￾packaging" and project-led efforts in approaching the Government. They described USAID projects as standardized, without being adapted to variations across the country. It was said that “USAID project management systems were applied in a strict fashion, to the extent that the tools were more important than the process of working together as a team, which ultimately interfered with the pursuit of the project results.” QUIPS did succeed in getting support for the integration of some QUIPS curriculum materials into the pre-service teacher training colleges. ILP trained 320 tutors at the colleges as part of the field training program, which included mathematics, English, and child- centered methodology.93 8.3.4 The Sustainability of National Project Outcomes. Some aspects of the QUIPS program are being sustained within the national framework for educational implementation. They include the contribution QUIPS made in helping national policy makers recognize the role of the community in educational improvement and the harmonization of materials for teaching English as a second language and improving mathematics teaching. There were also some serious constraints that limited the sustainability of QUIPS at the national level and should be considered important for future USAID programming in Ghana. 8.3.4.1 Strengthening Materials Development and Access to Textbooks. The harmonization of materials produced by QUIPS and the responsibility given to the Teacher Education Division to coordinate their reproduction and distribution were important to sustaining the project outcomes of QUIPS across the nation. One of the major barriers to ensuring sustained change, particularly in relation to pupil learning outcomes, is lack of access to textbooks and other teaching aids. The evaluation made it clear that textbooks and reading materials are important in reinforcing and sustaining learning outcomes, yet not enough attention was placed on ensuring that GES performed its role in getting textbooks to the schools during QUIPS. Unfortunately, the inadequate attention given to teaching aids as a major school-based input reduced the potential learning outcomes in both QUIPS and control schools over the intervention period. GES/MOE did not accept the QUIPS recommendations to simplify the national primary school curriculum at the time; three years later, GES/MOE have not yet completed the curriculum and the schools remain without books and syllabuses. Moreover, it appears that, in anticipation of the new curriculum, old textbooks are not being reprinted. Meanwhile, new textbooks cannot be published because they should be written to the new curriculum. 93 In its review of the draft evaluation, Catholic Relief Services stated that: “the report seems to suggest that only the ILP program only organized trainings for tutors at Teacher Training Colleges in English and Mathematics, but the CRS program also conducted these trainings in the northern teacher training colleges.” 122 8.3.4.2 Community Participation Component. Sustainable changes that were observed to be influencing national GES operations include the following: • GES has recognized the role of SMCs/PTAs in influencing the academic performance of pupils; it has supported annual training of all SMCs and PTAs across the country and made the DEOs district education office responsible for it. SMC/PTA training is now one of the nine-benchmark budget categories in the annual district budget. • GES has also formally recognized the position of CPC by providing a budget line item for CPC activities in the annual district education budget estimates. GES continues to use the National Service Scheme to support community participation activities and the CPC position at the DEOs. 8.3.4.3 GES Support to QUIPS. The GES-USAID working group for QUIPS was not sustained during the life of the project for a number of reasons, such as changes in personnel and ineffective scheduling. The working group was intended to be a vehicle for face-to-face dialogue about the quality of QUIPS implementation in the field. In 2003-04, QUIPS interventions had covered much of the country, collecting considerable data relevant to national policy making, particularly as it relates to rural education quality. Because the QUIPS participatory planning and implementation approach was limited to QUIPS at national level, so was the agreement of the partners. The collaborative way the DGM was put together by all QUIPS partners and USAID did not adequately involve GES at either the district or the national levels. Greater coordination between the Ministry of Local Government and the MOE/GES could have encouraged greater local investment in education through the use of the Common Fund for sponsoring teacher trainees, teacher in-service, community awareness, and salaries for volunteer teachers. The economic opportunity costs of GES staff and teachers working on projects like QUIPS was not factored into the QUIPS program design and implementation. Another obstacle for sustainability of QUIPS at the district and community school levels was the lack of project termination planning by the GES and the implementing partners. Proper planning would have allowed for additional support for the primary school system when QUIPS was withdrawn. More consultation with and engagement of the District Assemblies during QUIPS would have eased the transition and increased ownership of QUIPS best practices at the district level. USAID, the Ministry of Finance and Economic Planning, and the GES did not follow standard procedures for closing down the QUIPS program in a way that would help maintain or reinforce the strengths of the effort. 123 8.3.5 Conclusion. The main obstacle for QUIPS, like other USAID-funded projects, was the implementation modalities that reinforced ownership by USAID and lessened the extent to which it became government- or district-owned. An approach to implementation that allows more diversity and flexibility in national and district decision making may make projects like QUIPS better received and sustained, particularly by governments that are moving toward sector-wide programming and encouraging their partners to help support a collective vision. CHAPTER 9: FACTORS THAT AFFECT CHILDREN’S LEARNING 9.1 INTRODUCTION One goal of the evaluation is to better understand the pupil and school factors that impact learning in Ghana’s primary schools. The quantitative analysis addressed two dimensions of achievement outcome.94 The first considers pupil and school-level factors related to learning growth during the two￾year period of active QUIPS training and support. In the first analysis, the outcome of interest was the estimated slope coefficient representing the learning curve of pupils and schools. Various pupil and home factors, general school factors, the targeted teacher instructional reforms, and community good practices targeted by QUIPS were entered into the analysis as predictors. The second dimension considers (1) pupil and school factors related to the early academic development of pupils and (2) effectiveness in Ghana’s primary schools in the absence of QUIPS interventions. The qualitative analysis component was an evaluative field study consisting of interviews with selected stakeholder groups and observations of school/communities categorized as “high” or “low” learning groups.95 Results from the evaluative field study were organized according to the following categories: • What children themselves bring to the learning situation; • How parents and communities support children’s learning; • The quality and extent of learning opportunities teachers and schools provide for children; and • Factors related to GES support for quality education at the district and from national levels. Qualitative findings are presented according to their frequency of occurrence in the results compiled by the four field teams. The absence of a factor emerging in the qualitative results does not necessarily mean that it is not important to learning among Ghanaian school children. Rather, it is simply not a factor that was reported or observed substantially during the evaluative field work. 9.2 QUANTITATIVE RESULTS The quantitative analyses used a variety of descriptive methods and HLM techniques (described in Chapter 5) applied to data from the following sources: 124 94 The nature of the achievement testing is discussed in Chapter 5. 95 The categorization of schools into high-and low-performing groups is discussed in Chapter 1. 125 • Pupil demographics collected at baseline during QUIPS; • Primary school statistics data collected the year immediately after the two- year intervention; • Data from the QUIPS Achievement Tests (Grades 3-4 and Grades 5-6 mathematics; Grades 3-4 and Grades 5-6 English reading); • Data from the QUIPS Classroom Observation Instrument; and • Data from the QUIPS Community Best Practices Instrument. As in any quantitative study, the predictors entered into the analysis are not exhaustive, so qualitative findings are necessary to give a comprehensive view of the most critical predictors of learning in Ghanaian schools today. Where possible, interpretation of the quantitative analyses is linked to these important qualitative findings. 9.2.1 Quantitative Analysis of Pupil and School Factors. The predictors considered in the quantitative analyses were the following: • Pupil Factors: (1) age; (2) sex; (3) mathematics text to use at home; (4) English text to use at home; (5) other children’s books in the home; (6) father that reads at home; (7) mother that reads at home; and (8) use of English in speaking to the child at home. • School/Community Factors: (1) region (north, middle, south); (2) school/ community location (urban, rural); (3) school type (QUIPS, control); (4) percent of trained teachers in school; (5) pupil-to-teacher ratio in school; (6) pupil-to￾classroom ratio; (7) text-to-pupil ratio, for English and math; (8) percentage of girls in the class; (9) percentage of girls in the school; (10) availability of water, electricity, roads; (11) school aggregates of texts for children to use at home; (12) school aggregates of other books available for children to use at home; (13) school aggregates of father reading at home; (14) school aggregates of mother reading at home; (15) school aggregates of a child’s exposure to English at home; (16) instruction practices as measured by the QUIPS Classroom Observation Instrument; and (17) community practices as measured by the QUIPS Community Best Practice Assessment Instrument. 9.2.2 Factors Related to Pupil Academic Development and School Effectiveness. Tables 9.1 and 9.2 present the results of analyses of factors related to the academic development of pupils and school performance based on static achievements in math and English collected before the QUIPS interventions. The achievement of pupils in Grades 3 and 5 pupils were measured at the beginning of the school year. The shaded areas in Table 9.1 show pupil and home factors that were identified as significantly related to early academic development. The shaded areas in Table 9.2 show general factors identified as significantly related to overall school effectiveness in Ghana’s primary schools, excluding QUIPS targeted school and community good practices. 126 Table 9.1: Pupil Factors Related to Achievement Measured at the Beginning of Grade 3 and Grade 5 Pupil’s Factors Grade 3 Math Grade 3 English Grade 5 Math Grade 5 English Gender Age at testing Text to use at home Other children’s books at home Father reads at home Mother reads at home English used in speaking to child at home Table 9.2: School Factors Related to Static Achievement Measured at the Beginning of Grade 3 and Grade 5 School Factors Grade 3 Math Grade 3 English Grade 5 Math Grade 5 English Region (north as compared to south and middle) School/community location (urban, rural) School type (QUIPS, control) Percentage of trained teachers in school Pupil-to-teacher ratio for school Pupil-to-classroom ratio for school Text-to-pupil ratio, for English and math Percentage of girls in the class Percentage of girls in the school Availability of water, electricity, and roads School aggregates of texts for use by children at home School aggregates of other books in the home School aggregates of father reading at home School aggregates of mother reading at home School aggregates of a child’s exposure to English at home The pupil and home factors most strongly associated with early academic development were: gender, availability of texts and other books at home, and exposure to spoken English at home. Gender was a strong predictor of performance in all primary school classes and subjects tested: Girls’ early academic development fell behind boys’ in math and English reading. Although this is not an uncommon finding in developing countries, what is interesting from these findings is the fact that encouragement of girls’ participation in school was identified throughout the study as a key factor in pupil learning and overall school effectiveness. 127 The results presented in Table 9.1 underscore the importance of home factors in a pupil’s academic development in Ghana. The fact that home factors make a difference is also a critical finding. The results of this study point to the importance in developing countries, as elsewhere, of emerging literacy factors in school performance, including early exposure to print by having reading material for children’s use at home; “reading role models” such as a parent reading at home; and for second language literacy exposure to the spoken second language. The importance of supporting a child’s study at home by providing materials, including a textbook, is also an important finding. Parent support of a child’s independent home study was underscored as critical; the theme was identified from both the qualitative and quantitative analyses. Table 9.2 presents information on school factors, some of which are school-level aggregates of pupil-level factors, because research has shown that there are situations where individual pupil characteristics massed at a higher level (classroom or school) can significantly impact the overall performance of pupils in the classroom or school.96 After controlling for some home factors, such as: “availability of books at home,” “exposure to print and the English language,” and “adult literacy in the home,” most school factors were only associated with some subjects and some classes.97 Urban and rural divisions and percentage of trained teachers were sited as a significant factor for at least two subject/grade combinations. 9.2.3 Pupil and School Factors Predicting Pupil Learning During the QUIPS Intervention Cycle. This second analysis studied pupil and school factors that influenced a child’s learning growth during the two-year period of QUIPS beyond those factors related to entry-level achievement. Gender was identified as a key pupil factor related to learning, although the strength of the association between learning growth and gender varied significantly. In other words, some schools are more egalitarian in their teaching than others.98 A second finding was predictable: the relation between exposure to spoken English at home and the learning of English reading in Grades 3-4 and Grades 5-6. Other pupil factors associated with achievement growth were age (younger children had steeper growth curves in Grade 3 math); availability of children’s books at home; and father’s literacy. The importance of a child’s exposure to the printed word and spoken English, 96 For example, the performance of lower-performing pupils is often enhanced if the school or classroom is generally high-performing. Where school contexts are not particularly egalitarian with regard to pupil home background, however, and pupils from higher socioeconomic backgrounds are favored, low performance is exacerbated by competition and favoritism. 97 Many of the school factors are based on national primary school statistics. Their reliability and validity in terms of individual schools is weaker than for district or regional aggregates. School-based information was required for this analysis. In the future, projects or programs should collect such statistics internally to strengthen the validity of evaluation findings. 98 This is an area in need of further quantitative study. 128 and to reading role models in the dynamic learning process cannot be overstated. It reinforces findings that point to home factors as one of the key ingredients of academic success for Ghanaian children. The most prominent school factor related to pupil learning measured over the course of the QUIPS intervention cycle was the “treatment” factor, i.e., the best predictor of pupil achievement growth during this two-year period was whether or not a pupil was attending a QUIPS or control school. Children participating in the QUIPS program consistently showed steeper growth curves than pupils in the matched control schools, even after controlling for a variety of other factors, such as urban/rural differences. In addition, the trained teacher/untrained-teacher ratio was identified as a factor influencing the learning of English reading: more trained teachers in a school and lower pupil/teacher ratios were associated with higher learning curves, on average, for English reading. Another noteworthy finding from the growth curve analysis was that schools in all regions of the country and in both urban and rural communities responded equally well to QUIPS training and support, regardless of entry-level performance differences. Both high and low-performing schools alike were impacted by the QUIPS program. This fact is reinforced by cross-tabulations of region and urban-rural school groupings by pupil learning groupings; these showed no significant differences in the distributions of high, medium, and low performing schools. 9.2.4 Classroom and Community Factors. The analyses of teacher and community factors in pupil learning are discussed in Chapter 6 and Chapter 7. These are summarized in Tables 9.3 and 9.4. These results are based on classroom and community data collected at the end of the QUIPS cycle, in July of the second academic year. Both statistically significant results and suggested trends99 given from the cross￾tabulations are presented. Shaded areas refer to significant associations between classroom practice and learning based on statistical findings. Cells marked with an “x” refer to noticeable trends in the expected direction identified from shifts in the distribution of scores given by the contingency table analyses. Overall, the findings about teacher practices presented in Table 9.3 were reinforced by results from the evaluative field work (discussed below). Both quantitative and qualitative study found that classrooms in schools where pupil learning during the “QUIPS era” was high had “pupil-friendly classrooms,” where teachers used good questioning techniques, provided feedback (e.g., exercise books), were open to pupil interactions in groups, and encouraged all children to participate. Pupil testimonials to good teaching underscored the importance of a friendly and “safe” learning 99 “Noteworthy trends” from descriptive data presented in table 7.6 are based on cell statistics that identified cell counts that were at least two standard deviations from the expected count for a balanced table (where there is no relation between the two variables). One factor that reduced the sensitivity of the chi-square test in these analyses was the existence of cells where the expected count was less than five. 129 environment, one where teachers facilitated interaction between pupils. They were not afraid to make mistakes and could learn from each other in small groups. Table 9.3: Classroom Factors Associated with Pupil Learning* Significance and Trends** School Factors Considered North South Teacher time spent on routine tasks Opportunities provided for children to use learning materials x Teacher use of questioning x Pupil questioning behavior Facilitation of creative and critical thinking Disciplinary behavior x x Use of strategic grouping Learning behavior of pupils within groups x Use of a variety of teaching methods Opportunities provided for pupils to demonstrate their abilities Interactions between pupils and teachers and between pupils Feedback provided by teachers Special encouragement of girls to participate Encouragement of all pupils to participate *The north and south were considered separately because of the differences in the distribution of teacher practices across regions. This result is discussed in Chapter 5. **Shaded areas refer to statistically significant associations between classroom practice and learning. Cells marked with an x refer to non-significant trends in the expected direction given descriptive results (e.g., high performance associated with schools in the high learning performance category). The most prominent classroom “good practice” where association with learning was statistically significant, in both the north and south, was the practice of encouraging girls to participate in class. This factor, “encouraging girls to participate,” also emerged consistently for all subjects at all grade levels as being significantly related to pupil learning in the HLM analyses, even after controlling for a variety of other factors. The two most prominent and statistically significant community factors, given in Table 9.4, were “supporting school quality” and “empowering local people to act.” The former was also found by the HLM analyses to be a predictor of school learning, even after controlling for a variety of pupil and school factors. Key dimensions of these two factors related to pupil learning as defined by the QUIPS Community Best Practice Instrument point to certain essentials in community-school relations that support learning: • Awareness and interest in pupil learning and the quality of the school environment; and • An active SMC/PTA where the views of all members are respected, as evidenced by group decision-making. 130 Table 9.4: Community Best Practices Associated with Primary School Learning* School Factors Signficant Factors and Trends** Trust in the school system X Supporting quality education Supporting girls’ education Empowering local people to act Strengthening school management X Partnering with external agencies X Use of participatory planning and design for school improvement Mobilization of resources for school improvement Use of a variety of teaching methods Monitoring and accountability of school finances and assets *The north and south were not considered separately because no differences were observed in the distribution of teacher practices across regions. (See chapter 5.) **Shaded areas refer to statistically significant associations between classroom practice and learning. Cells marked with an x refer to non-significant trends in the expected direction given descriptive results (e.g., high performance associated with schools in a high learning performance category). The community findings are reinforced by the evaluative field work, which pointed to the importance of empowering communities with knowledge and skills that help them support improved school quality in general and pupil learning in particular. The field work provided a rich source of information for understanding the basis for the relationship between these two community factors (support for quality education and empowering local people to act) and pupil learning. The qualitative findings suggested that basic knowledge and interest in pupil performance and the learning environment combined with basic information about the roles and responsibilities that communities have toward education can play an important role in improving school effectiveness and pupil learning. Communities in all high￾performing school/communities demonstrated this basic understanding. In summary, the behaviors emphasized in the QUIPS training and support programs had merit as important behavioral targets for supporting school effectiveness. It is not often the case that project interventions such as in-service training are associated with a “significant” pupil achievement outcome in the short term100 and rarely are community interventions directly related to pupil learning (based on quantitative analyses like these) in the short period of a development project. In the QUIPS program both school level factors and community factors were associated with children’s learning during the period when QUIPS was giving active training and support. 100 This is based on a number of factors, particularly the poor sensitivity of static achievement measures (versus dynamic pupil learning) to longitudinal change (e.g., due to random sampling error across different groups over time). 131 9.3 QUALITATIVE FINDINGS 9.3.1 Child Factors. The field teams interviewed boys and girls in separate groups and, where possible, these groups were disaggregated as “high achieving” and “struggling learners” children. The groups were interviewed separately to maximize their comfort level and facilitate honest discussions about the children’s own experience in the academic process. For the most part, the interviewees were taken from Grade 6 classes. The children were pleased to be included and most were very willing to talk about their school experiences, especially when assured that what they said would be held in confidence. The interviews shed light on what children themselves bring to the learning experience. Table 9.5 presents enabling and constraining factors mentioned by children across the 18 field sites, listed according to the frequency with which they identified. Table 9.5: Child Factors that Influence Learning Enabling Factors Constraining Factors Children desire to “be somebody” when they grow up and realize that education is the key to opening doors to their future. Learning is enhanced when children…. ƒ are not overburdened with household, farm, or other work ƒ have parents who can provide their basic school needs: exercise books, pens, uniforms, etc. ƒ are actively involved in learning, experience success, feel free to ask questions ƒ know that their parents and teachers have high expectations for them ƒ learn from each other Children like going to school and attend more often when they are confident that their classroom is a safe and friendly place Learning is enhanced when the environment is suitable for learning. Children are motivated to learn when they have educated role models. Learning is constrained when children… ƒ cannot concentrate effectively when they are hungry ƒ are too tired because they have too much physical work before even coming to school ƒ absent from class for too many days ƒ are afraid to ask questions ƒ are reluctant to become involved because they fear unpleasant consequences–being put down, caned, or laughed at by peers ƒ lack confidence and do not experience any success ƒ do not understand the teacher (because either the language is too advanced or they don’t speak or understand the language of instruction) ƒ lack basic needs ƒ have to walk very far to school Almost every child interviewed was very clear that he or she “wants to be somebody,” and they all know that education is the key to that future. One small boy, in torn uniform and bare foot, was adamant that he will be a doctor, and knows that to reach this goal, he must finish Grade 6, go on to JSS and SSS, and then to university. The fact that he struggles to read material at a much lower level than his grade placement plus that he 132 says that he must “go to farm to get money to pay his school fees,” suggest that he has not yet perceived these things as deterrents to his education. The two most mentioned constraints to learning the children identified were going to school hungry and having too much work outside of school. In combination they are an enormous impediment. Yet many children are still coming to school, especially when they view it as a place that is welcoming and where they can succeed. Children in the lower-achieving groups were more likely to identify these factors; their higher-achieving peers, especially the girls, were more likely to say that their parents did not overburden them. The less skilled children also appeared to have poorer attendance when the number of lessons in their exercise books was compared to those of their colleagues. The classroom experience was important to pupils. When they are actively involved in their learning, feel free to ask questions, experience success, have the materials they need, and the classroom and learning environment are welcoming and comfortable, children report that their “learning is better.” They appreciate their teachers and recognize when their needs are being addressed by teachers: “He knows we are not all equal but he does not mind. He puts us in groups and we can all learn.”101 When these circumstances are not the case they talk quite differently about school: “Some of the children reported that they are afraid to talk in the classroom of the non￾QUIPS teacher because if they are not able to answer the teacher’s question, they will be caned.” 102 9.3.2 Home and Community Factors. Of all the messages emerging from the QUIPS Final Evaluation, the clearest is about the role that communities are beginning to play. As they begin to experience the influence they can exert over the quality of their school, they become more confident and more involved in how the school operates. This is usually paralleled by the school becoming increasingly effective at educating their children. This was particularly true of the high-performing QUIPS schools, although community influence was an important factor in the high-performing control schools as well. All SMCs/PTAs in Ghana were influenced to a greater or lesser extent by the community mobilization component of QUIPS, so the finding that high-performing schools, QUIPS and control, share these factors is not surprising. (See Table 9.6; factors are again ordered by frequency of report.) 101 Grade 6 girl, Lonto Presby Primary, north. 102 Fieldwork report, Twifo Hemang Lower Denkyira District. 133 Table 9.6: Home and Community Factors that Influence Child Learning Enabling factors Constraining factors Home/Parents: ƒ feel free to visit the school ƒ make reasonable demands regarding children’s household chores ƒ supervise homework and provide structured opportunities for children to study at home, including light to study by ƒ provide adequate nourishment ƒ take steps to stay informed about their children’s progress – check exercise books for ‘the teacher’s mark,’ visit school regularly ƒ provide support including basic school needs and encouragement ƒ pay school fees ƒ participate in school-related community activities – meetings, communal labor ƒ prioritize meager resources to allow for school expenses ƒ buy books for their children Communities: ƒ have an SMC/PTA that understands its responsibilities and works to fulfill them, meets regularly, works to bring community and teachers together to effectively manage the school, maintains a cordial relationship between school and community, demonstrates a sense of ownership of the school ƒ complement/expand on what government has provided to improve quality learning, e.g., recruit and support volunteer teachers, raise funds to buy books ƒ develop and maintain school infrastructure, organize communal labor, ƒ support teachers; help with housing, food, farm ƒ do repeated sensitization activities to convince parents of the value of education for their children and ultimately for themselves, undertake drives to increase enrollment (in most communities many children are still not in school) ƒ visit the school to monitor teacher attendance, find out about school needs ƒ make and implement decisions about community activities that affect school attendance and learning, e.g. banning attendance at video shows for children during the school week, curtailing their involvement in funerals, etc. ƒ organize study groups, ensure there is light and supervision ƒ advocate for their school and take action with officials when necessary Home/Parents: ƒ do not provide an environment where children can study and do homework ƒ do not make lanterns available for children to study ƒ are unable to feed and clothe their children adequately ƒ overburden children with household/farming responsibilities ƒ do not pay school fees ƒ do not monitor children’s progress ƒ have no books at home ƒ harass or insult teachers ƒ do not visit the school, or organize night study groups, or supervise children at home ƒ show their lack of value for education by refusing to send their children to school ƒ do not see results from sending their children to school. Communities: ƒ have an SMC/PTA that does not meet regularly and fulfill its responsibilities ƒ do not cooperate with school and teachers to support the school ƒ expect the government to provide everything related to education ƒ has leadership (chiefs, elders) that do not invest time and attention in educational matters and advocate for support for their school ƒ puts up with disputes that interfere with effective management of the school 134 9.3.3 Parent Factors. Parents are the first line of community for children. The extent to which they value education is reflected in the way they support their own children’s education. This in turn influences how their children think about schooling. The comments about parental factors that affect children’s learning came from children, parents themselves, community leaders, head teachers, teachers, and DEO staff. Clearly their influence is critical. The expectations they hold for their children have a direct impact on what the children aspire to. This was clear in both child and adult reports. Something as simple as parents feeling welcome to visit the school and seeing it as part of their responsibility has struck a chord. Every community talked about this: “Before there were difficulties between parents and teachers, now that is not so. Now we visit and when my daughter sees me she points and tells the others that I am there and she is proud.”103 The issue of taking children out of school to help with farming or other income￾generating activities came up in every community. According to a public official in the East Gonja District, failure of parents to value education is at least in part a function of the poor school outcomes that they have witnessed from observing their own children. Parents want their children educated and able to gain employment. When schools have few teachers and inadequate resources, learning is impeded and parents are very aware of this. They drop out or come out unemployable – either way they become liabilities. The public official put it this way: “Parents argue that if you send your cattle to the bush and they do not come back you do not need to send some more.” A theme that reverberated across many of the communities, QUIPS and controls, high and low-performing schools, was light to allow children to study at night. In homes where there is only one lantern, as is often the case, many children simply do not have access to it. However, in communities where study groups are organized and light is available, not only do children get to finish homework and study but they feel that their parents are supporting and encouraging them. 9.3.4 Community Factors. An important shift over the course of the QUIPS implementation period was a growth in the role of communities in management and support of their schools. When stakeholders fully agree on their own responsibilities, things get done. Communal labor to build and then maintain school infrastructure has increased and there is evidence of an emerging culture of care for school infrastructure. People no longer expect the government to provide everything. Rather, many communities are becoming more active in supporting the school in a variety of ways, especially in the management of school affairs. The SMC of one northern village went 103 Mother, Nakpale Kworle L/A, QUIPS low. 135 directly to the District and Regional Education Offices to demands teachers at their school and were successful. Another village PTA wrote to the District Director of Education asking for deductions in the head teacher’s pay due to his irregular attendance. The request was investigated and acted upon. “Ownership by the community of the school assumed a different dimension. There was an increase in awareness of how the community had the potential to contribute to the learning of its children. Parents saw how they continued to be a major influence on the performance of their children while they were away from home and in the classroom” 104 9.3.5 Teacher and School Factors. How well teachers are able to meet the learning needs of children by creating classroom and school environments conducive to learning is probably the single most critical determinant of quality education delivery. However if teachers are to work effectively, schools must provide the materials and structures necessary to support their efforts. Table 9.7: Teacher and School Factors that Influence Child Learning Enabling factors Constraining factors Teachers ƒ provide appropriate, positive, and timely feedback ƒ use a variety of teaching and classroom management strategies ƒ use TLMs effectively to actively involve children in the learning ƒ are regular and punctual ƒ receive continuing supervision and support from the head teacher ƒ prepare good lesson notes and ensure that they cover the syllabus ƒ accommodate needs of pupils who are struggling, provide extra classes ƒ interact positively and regularly with parents ƒ create an open, friendly classroom where children are free to ask questions ƒ recognize that children can learn from each other ƒ are from or at least live in the community in which they are serving ƒ take advantage of professional training opportunities to improve teaching Teachers ƒ are not supervised and supported by head teachers ƒ are not regular and punctual because they live far away and have to travel distances to school ƒ do not plan lessons adequately ƒ do not use participatory, child-centered strategies ƒ have to pay for materials to make TLMs ƒ use inappropriate language (do not communicate in the child’s language, vocabulary is too advanced) ƒ do not cover enough material, do not assign homework ƒ are present but do not teach ƒ are unwilling to be posted to or stay in remote areas ƒ are not motivated 104 Fieldwork report, Tolon-Kumbungu, East Mamprusi, and Techiman. 136 Schools: ƒ have enough trained teachers, stable staffing ƒ have well-maintained infrastructure- good school blocks provide protected and comfortable places to learn, adequate furniture, adequate sanitary facilities mean children waste less time leaving class to attend to needs (especially for girls) ƒ have enough textbooks and other teaching and learning materials ƒ provide teacher training opportunities through regular in-service training, have a culture of good practices ƒ have good supervision and support ƒ are inviting to both children and parents ƒ have teacher accommodation which allows teachers to live close by and thus be regular and punctual Schools: ƒ do not have enough teachers ƒ do not have enough textbooks and other TLMs to allow teachers to teach effectively ƒ have overcrowded or combined classes ƒ do not provide effective supervision and in￾service training ƒ lack QUIPS teachers, who have left ƒ do not include parents in their planning and activities ƒ are in need of repairs to the extent that teaching and learning are jeopardized While teachers in high-performing QUIPS and control schools were more likely to demonstrate the enabling behaviors identified in Table 9.7, there was evidence that QUIPS schools that had been able to maintain supports and structures similar to those available during the project phase (full staffing, good head teacher supervision, support from the district, etc.) were more likely to have such characteristics. By the same token, low-performing schools often shared the same limitations. 9.3.6 Teacher Factors. Quality teacher feedback was the most often mentioned factor by children themselves, their parents, and school and district supervisors (head teachers and Circuit Supervisors). In several different ways pupils indicated that when teachers let them know how well they are doing and encouraged them to keep trying, they gain confidence, which in turn stimulates them to work harder. “Our teachers encourage us to ask questions and they ask us a lot of questions. If we get it wrong, they encourage us to try again.” (Field report, middle region.) “He asks questions a lot. When a child gets a question right, he praises him/her. He at times gives out money.”105 Child-centered teaching methods were identified as contributing to effective learning: “It is better now because he teaches us well. He doesn’t mind that we are not equal. He repeats if we don’t understand.” 106 “We sometimes do English, science, and religious and moral education in groups. In a large group some are not serious and you can’t learn. In small groups I can learn better. If you don’t know something, you can ask your friend. But sometimes people won’t share information so must ask the teacher.”107 In one school the children distinguished between the QUIPS trained and non-QUIPS teachers in their school, “Some of the children reported that they are afraid to talk in the 105 Field report, north. 106 Grade 6 girl about a QUIPS-trained pupil teacher. 107 Grade 6 girl, northern region. 137 classroom of the non-QUIPS teachers because if they are not able to respond to the teacher’s questions, they would be caned.”108 Parents notice feedback by teachers. While many parents cannot themselves read, they do understand the “teacher marks” – X or √—and reported that they regularly check their children’s exercise books. One mother reported that when she sees that her child has many checks, she rewards her by buying something small. The importance of child-centered teaching was identified in different ways by all stakeholders. Included in this is the notion that children can learn from each other, as teachers who have learned to use grouping have discovered. A teacher from Dunkwa Presby reported, “When teachers engage children in discussion, children are able to share ideas that sometimes teachers are not even aware of.” 9.3.7 School Factors. Availability of teachers is the single most common theme across many parts of the country. Not one of the schools visited in the north region had a full complement of teachers. In fact, most had only one trained and a couple of untrained teachers. What distinguished high- from low-performing schools was the extent to which the community responded to this issue. In both QUIPS and control high￾growth schools, volunteer teachers had been recruited by the community, trained by the head teacher, and were reported to be doing a satisfactory job. Teacher attendance and punctuality is an issue. All stakeholders recognize that when teachers are not in school teaching, children lose out. Some attendance problems were explained by the fact that teachers cannot or will not live in the community and have long distances to travel to get to school, may have to travel for a day or more to collect their salary, and sometimes are away for training. That this last is a two-sided issue is made clear by Grade 6 girls from Lonto Presby: “Our teachers absent themselves from school when they travel to attend courses, but after they come back they can teach us better.” Teachers from the community are reported to have a stronger sense of responsibility: “All of the teachers come from Kumdi and consider the school and its children their personal responsibility.”109 Sometimes the teachers report to school but do not teach and leave during the day. One SMC/PTA member attempted to have the district take action against such a person but to date nothing has been done. Lack of text books and other teaching materials was also mentioned often as critical. “Due to the lack of teachers and textbooks, few children can manage to just read and write. The effectiveness of the school is therefore bad.”110 108 Field Report. Middle region. 109 Elder, Kumdi, north. 110 SMC/PTA member, north. 138 When schools provide effective supervision and support, teachers are encouraged to perform more effectively: “The school has procedures in place that support the work of the new teachers. They do lesson planning, have their plans vetted, discuss methods with trained teachers, borrow materials for a lesson, create teaching aids, and create interesting quizzes and exams. In a short period the untrained teachers through the systems in the school learn to be proficient in the classroom.”111 9.3.8 Systemic Factors: District and Central Administrative Support. Ultimate responsibility for quality education in Ghana rests with the MOE through its implementing structure, the GES. GES at district, regional and national level has an important role to play in supporting schools and communities throughout the land. Stakeholders interviewed (see Table 9.8) were adamant that this vital link must play its role if children in Ghana are to receive the quality of education they deserve. Table 9.8: Factors that Influence Child Learning Enabling factors Constraining factors Districts: ƒ allocate teachers ƒ provide community sensitization activities to remind people about the importance of education ƒ are responsible for providing adequate supervision and support ƒ provide INSET to enhance professional capacity of teachers and head teachers ƒ identify, monitor, and evaluate indicators of learning GES Headquarters ƒ reviews and revises curricula to improve learning ƒ allocates newly graduated teachers to sponsoring districts as they become available to assign ƒ has structured the budget system to address critical benchmark areas ƒ has accepted community mobilization as a function of the education system by institutionalizing the position of Community Participation Coordinator Districts: ƒ do not provide enough teachers ƒ do not provide adequate support and supervision ƒ do not ensure that books get to schools in sufficient quantity ƒ do not respond to requests made by SMC/PTAs GES Headquarters ƒ creates bureaucratic delays, e.g., re-selection and appointment of untrained teachers, timely payment of expense claims, etc. ƒ fails to ensure adequate delivery of books to districts ƒ does not provide adequate funds to allow districts to carry out essential components of approved work plans 111 Field report, north. 139 9.3.9 District Factors. The District Education Office is the first, and for many the only, point of contact with GES. In fact very few of the stakeholders interviewed other than district officials even mentioned the other levels of GES. The district is viewed by communities and schools as the body that should be providing the essentials required to run a school and deliver quality education: teachers, books, supervision, and support. Many communities and schools feel that they are not well supported by their DEOs. “The education office is even taking away some of our teachers without replacing them.”112 Much of what district officials reported suggested that they are caught between schools and communities demanding support and a central authority unable to provide what they need to do their job. District officials say that they have no power over how many teachers they are allocated or over teacher movement resulting from transfers, study leaves and so on. Yet it is clear that communities do not understand this arrangement. Districts do not feel they can handle the pupil-teacher selection process on their own. The centralized system, which requires that teacher personnel matters “go through Accra,” results in long delays in posting teachers, delays that often extend until the end of the second term. The in-service training and supervision the district office provided to QUIPS schools during the “QUIPS era” is no longer available—and even then, this level of supervision and support was not provided to non-QUIPS schools. The importance of regular supervision was underlined by a district official from Twifo Praso (middle) who pointed out, “When you want the tortoise to pull out its head from the shell, you put fire in its shell but once you withdraw the fire the tortoise will crawl back in.” All district officials addressed the importance of supervision as an essential ingredient in quality schooling and pupil learning: “The regular visits of Circuits Supervisors help to put head teachers and teachers on their toes as regards sustaining best practices.”113 This issue was also cited by both head teachers and SMC/PTAs. District officials also pointed out the important role of monitoring and evaluation in improving learning in the schools. The M&E training provided to district officers through QUIPS has at least in some places enabled them to improve the way they go about gathering and using information to improve instruction. Another change associated with the QUIPS program is the role of the Circuit Supervisor as one of professional support. “What I like so much about QUIPS is the introduction of the ‘spirit of one never gives up’ to the people. . . the idea that there is always a way out whatever the problem, and one can always succeed.” 114 112 Elder, Lonto, north. 113 DMIT member, Zabzugu-Tatale District, north. 114 Government official, Northern Region. 140 9.3.10 Central Management. As the implementation arm of the MOE, GES is responsible for delivering education in Ghana. Much of the criticism aimed at GES has to do with control of resources and what are seen as onerous bureaucratic delays. When essential activities like curriculum reviews, which in turn delay development and availability of text books are seriously held up, the impact on how teachers teach and children learn is serious. Bureaucratic delays are also blamed for the difficulty in getting approval for untrained teachers’ assignments and for ensuring that funds reach the districts in time for them to implement the plans they have drawn up to address the needs they have identified. 9.4 CONCLUSIONS The result of these analyses underscore the importance of pupil and home factors in facilitating a child’s academic growth and development, and the critical importance of a child’s exposure to printed materials and spoken English in the home, as well as exposure to a parent reading in the home (a “reading role model”). In Ghana, as in most developing countries, gender is one strong predictor of school learning: The academic development of girls generally lags behind that of boys in all subject matters. A related and important finding is that special teacher encouragement of girls to participate in class and parental encouragement of girls were identified as key factors in learning in both quantitative and qualitative research. The importance of a child’s basic needs including sufficient nourishment before and during school was also underscored as underpinning learning. Among the critical needs was that families find ways of reducing or better distributing domestic chores so children have time to get to school and are rested when they do arrive and for them to have time to study at home. Critical teacher and school inputs to quality schooling identified as key to the academic success of primary school pupils in Ghana were: (1) teachers in classrooms teaching; (2) textbooks and other teaching aids available in the classroom and teachers who know how to use them; and (3) textbooks available for children to use at home and a light to study by. Head teacher leadership was also commonly mentioned. These very basic home and school factors “pupils encouraged by parents, rested and ready to learn at school and teachers in classrooms who care about and are aware of what pupils learn” together translate into more effective pupil learning. Quantitative and qualitative findings both suggest that not just mere presence but what the teacher does in the classroom is important: “Pupil-friendly classrooms” are linked to pupil learning. The findings underscore the importance of teachers being aware of children’s performance and encouraging all pupils to participate in class, facilitating 141 pupil interaction, providing opportunities for pupils to practice what they know, and giving meaningful feedback. Communities that support quality schooling and those where people are empowered to act in favor of quality schooling were associated with pupil learning in both quantitative and qualitative results. The evaluative fieldwork also identified the importance of communities being involved in school management, not just working on improving infrastructure and the school environment. The evaluative feedback also noted the critical need for DEOs to pay attention to and support their schools and the importance of external and supportive supervision through Circuit Supervisors as factors underpinning a school’s effectiveness. The support and supervision provided to QUIPS schools were identified by all stakeholders as among key factors in a teacher’s and head teacher’s ability to implement classroom reforms. Unfortunately, these were the very components that were the most difficult to sustain and mainstream. CHAPTER 10: LESSONS LEARNED FROM QUIPS Many of the lessons learned from the QUIPS program are not new. They are, rather, reminders of what is important, of what does or does not work and why. They are also reminders of how very complex the issue of effective schooling is and how interdependent all the critical components are. While an attempt has been made to isolate some of the more outstanding lessons, none stands alone. 10.1 CHILD-RELATED LESSONS All children want “to be somebody.” Building on their motivation and ambitions to help them realize their dreams underlies all school improvement efforts. However, high aspirations are not enough to empower many of Ghana’s children to become the best they can be. • Hungry children cannot learn well. In many parts of the country, children go to school without having eaten. The food programs that attempt to address this issue are seen mainly as strategies to increase enrollment. They must be recognized also as addressing hunger, an important factor that influences how well children can learn. • Child labor interferes with learning. Overburdened children cannot invest the time or energy required to take full advantage of education. This issue is of particular importance for girls, who are expected to perform household and family support tasks in addition to their school work. • Children know what they need. Including the perspective of children in designing programs to address their needs increases the likelihood that the intervention will succeed. 10.2 HOME- AND COMMUNITY-RELATED LESSONS Of all areas of QUIPS impact, community awareness and involvement showed the greatest growth. Since the community is by far the most stable of the beneficiary groups, this augers well for education in Ghana. More and more communities are demonstrating a sense of ownership and responsibility for the schools to which they entrust their children. • Increasing community awareness and empowering them to act increases their involvement. Community members can make a difference in how well schools respond to the challenges of educating children. While only a few public schools (367 of 12,451) benefited directly from QUIPS interventions, all 142 143 SMC/PTAs were trained through the program. Thus the evidence of greater change at the community level than in some QUIPS schools is understandable. The composition of the community and SMCs/PTAs has varied little since QUIPS, whereas there have been major changes in both school and district staffing. Besides providing tangible support to maintain school facilities, parents have learned that school visits keep teachers and children on their toes and are appreciated by their children. QUIPS school-communities are particularly forceful in proclaiming their disappointment about the diminishing staffing and resource support at their schools since the passing of the QUIPS era. • Cordial community-school relationships are critical to sustaining good practices at the school. Teachers need to realize that parents appreciate their efforts and they in turn need to live up to community expectations. When communities provide tangible support (teacher accommodation, maintaining school infrastructure) and teachers make efforts to communicate with parents about their children’s progress and offer extra help, the good relations that result contribute to school effectiveness. • Understanding the central role of chiefs and elders is basic to all community development activity. The role of the traditional leaders cannot be underestimated. They determine actions and attitudes in the community. Not only does the quality of leadership affect how the community operates, it has direct as well as indirect impacts on the school. Thus it is critical that all interventions work through this influential group to ensure the community support necessary to move forward. • The SMC/PTA has a central role in school improvement. The effectiveness of the SMC/PTA plays a large role in how well the school operates and, by extension, how well children learn. This is the group that builds and maintains the relationship between community members and teachers. • Children’s learning depends on adequate nourishment. Parents who understand that children need ‘fuel’ to learn effectively are likely to make a greater effort to ensure that their children get enough to eat if it is possible for them to do so. The role of programs like CRS school feeding is critical in contributing to children’s readiness to learn in areas where food sustainability is an issue. • Actions undertaken by communities demonstrate their commitment to school improvement. When the community takes an active role in addressing school-related problems, the school views it as a partner, as when it recruits and supports volunteer teachers to augment staff allocated by the GES. 144 • Positive local role models influence children’s and parents’ view of where education can lead. Community role models give children something to strive for, especially if the models are people they know. Strong teachers, especially when they were local people, provided excellent role models. The presence of role models also helps parents realize the value of educating children. • Monitoring progress and sharing results allows people to see their successes and to identify areas that need attention. Everyone—teachers, parents, children, community members, and GES officials—needs to see results if they are to stay engaged in development activity and planning. • Recognizing the value of education is important. Parents who understand the value of education are most likely to provide the basic school supplies their children need. They are more likely to provide what is needed to facilitate homework and study. However, when parents fail to see positive results from schooling, they are reluctant to send their children to school. 10.3 SCHOOL-RELATED LESSONS Schools are the delivery point for education. They are complex institutions too often viewed as simple and straightforward. There are important lessons to be learned from QUIPS about what enables a school in a developing country to effectively play its role. • Teacher availability is key. Lack of trained teachers in the schools has a major impact on sustaining the benefits of teacher training. When GES cannot provide teachers to a school, skills resulting training dissipate and disillusionment sets in. • Text books and other teaching/learning materials are essential tools of the trade for teachers and children. While committed teachers are critical to child learning, they need the tools with which to do their job. Without adequate textbooks and other teaching aids, teachers cannot teach effectively, especially if they have little formal education and training. • Local teachers play a special role. When teachers are members of the community in which they teach, their commitment is often greater and their impact is often heightened. Not only do they have a greater sense of responsibility for the children entrusted to them, they can serve also as powerful role models. • School leadership influences academic performance. Informed and effective school leadership has a direct influence on quality teaching and learning, which in turn stimulates academic achievement. In schools where the head teacher is providing regular support and supervision of teachers, learning is more effective. The opposite was observed also. 145 • Child-centered teaching and learning methodologies motivate children and influence performance. Instructional strategies that take into account individual learners, involve children actively in the lessons, and create a friendly classroom environment contribute to effective pupil learning. • Stable staffing enables schools to continue the good practices learned during in-service training. Change cannot be sustained if there is no corporate memory maintained in the school and classroom. The school culture in large part determines the norm for teacher behavior. When there is no critical mass to keep new ideas fresh and active, the tendency is to revert to old, familiar ways. Frequent staff changes limit the integration of new management and teaching strategies into practice. • Regular and supportive supervision is essential to maintaining and integrating new practices. The support and supervision role of the head teacher, supplemented by regular visits by the Circuit Supervisor, is a critical component of an effective school. When staff shortages require that the head teacher assume classroom responsibilities, quality teacher supervision is reduced. It is also at risk when Circuit Supervisors are unable to reach schools often due to lack of logistic support. 10.4 DISTRICT-RELATED LESSONS LEARNED The District Education Office is pivotal in the education system in Ghana. In collaboration with the District Assembly, it is responsible for ensuring that the schools within its jurisdiction comply with MOE/GES requirements. It is the intermediary between the school community and GES national headquarters. • Continuing supportive supervision is essential. Regular supervision and support from the district office is critical for both head teachers and teachers. Without regular GES visits, there is a risk that things will slow down. Moreover, such visits allow the DEO to stay aware of the needs of schools. The quality and quantity of regular supervision and support are critical to effective schools. During QUIPS the Circuit Supervisor visited schools often, sometimes to the detriment of non-project schools. That is no longer true. • Follow-up visits to schools between INSETS need to be built into any teacher improvement strategy. Reinforcement of training through follow-up visits to schools between training sessions is critical to the adoption of new teaching practices. While Circuit Supervisors have been trained to provide such support, there is concern that without the outside financial support, many will find it difficult to continue to do so. 146 • The District Teacher Support Team (DTST) is important if changes emanating from QUIPS are to be sustained. The inclusion of DTST members in QUIPS in-service training built the capacity of district staff. They became strong supporters of expanding QUIPS methods to other schools and an important source at the district level for spreading and sustaining initiatives introduced by QUIPS. • Training of district personnel increases sustainability. Providing training that enhances potential for people to do a better job has long-term benefits. Providing training and instrumentation for management and M&E enables the DEO to more effectively support all schools in the district and extend some of the benefits accrued from QUIPS. • Project requirements drain existing district human resources. When project requirements place heavy demands on district personnel, as was the QUIPS case, the district’s capacity to respond to other needs is compromised. The professional time and energy required to support QUIPS innovations in three schools left as many as a hundred or more others with radically reduced support. 10.5 INSTITUTIONAL LESSONS LEARNED When institutions work together to reach a common goal, it is essential that they clearly understand the nature of the relations and the roles and responsibilities of each partner. This is particularly challenging in a cross-cultural environment. • Institutional relationships matter. Trust and understanding among partners are critical. It is important to invest time and resources to ensure that all involved are expressing their views and listening to each other. For example, while QUIPS program personnel reported excellent working relationships with GES, GES officers did not always report so. An officer in the Basic Education Division described it this way: “The relationship was often difficult. QUIPS was in the driver’s seat.” Another officer suggested that towards the end many people did not want to be associated with QUIPS because they thought it was not sensitive to their needs. The issue of money was one aspect. For example, when QUIPS would not pay GES officers the same daily subsistence allowance (DSA) rates that USAID paid even their drivers, they felt that they were not being respected. Also, “QUIPS officials behaved as though they had authority over GES personnel. QUIPS was looking for results. If they saw someone not pulling, they tended to make sure that they were doing what they (QUIPS) thought they should be doing. This worried some officials, because they were made to look helpless.” Sometimes district officials and even teachers were not up to the demands and resented that they were forced to be in the project. 147 • Ensuring mutually respectful relationships between implementing partners is important. The involvement of several implementing contractors—each with its own standards, requirements, and contractual obligations—creates a situation that can affect collaboration and jeopardize maximum effectiveness. Potential for success increases to the extent that project objectives and implementing strategies are clear and agreed on by all parties. 10.6 PROJECT MANAGEMENT LESSONS LEARNED Effective management of a project is critical from the very beginning of the design phase through to the final evaluation. Iterative planning allows for responsiveness to the changes in circumstances that are likely to occur over the course of several years, as was the QUIPS time frame. However, changes must be effected in the context of the needs of the beneficiary, not just the implementer’s frame of reference. • The integrity of project design must be respected. The design phase of the project is critical to effective realization of results. When important design considerations are compromised (in this case reduction of intervention length from 36 to 24 months), the possibility for reaching objectives is compromised.115 • Sustainability depends on being realistic about available resources. If programs like QUIPS are to be sustained, they must make sure the essentials for implementation will remain in place when program support is withdrawn. This may require setting lower goals than is considered ideal, or even desirable. QUIPS set higher levels of practice than could be sustained given available GoG resources. This led to disintegration and discouragement on the part of many stakeholders—children, community members, teachers, and district personnel. • Focusing on possibilities instead of problems changes the way challenges are perceived. When people are given a new set of lenses through which to view old problems, they can start fresh. The adoption of appreciative inquiry (AI) as an underlying premise of QUIPS gave many stakeholders a chance to reexamine education in Ghana and think about different ways to respond. • Truly effective in-service training meets the learning needs of the teachers. The effectiveness of INSET is related to both the relevance and coherence of the training and to the readiness of the trainees. Teacher in-service training that does not provide enough time and support to ensure that new approaches can be 115 In its review of the draft evaluation, the Academy for Educational Development stated: “[We were] not aware that the original project design called for a 36 month intervention at each school. We proposed that at least two years was needed, and would have preferred longer, but the structure of the design to reach three schools in every district, combined with the realities of the project life time frame and limited funding, prevented this from being an option.” 148 integrated into the repertoire of the teachers risks being ineffective in the long run. The in-service component of QUIPS was ambitious. While INSET addressed strategies and skills necessary for child-centered learning, it did not provide comprehensive methodologies, especially for teaching reading. Perhaps the training assumed too much entry-level skill, especially since so many participants were completely untrained teachers with low levels of basic education. • Development takes time. All adult stakeholders were adamant that the duration of QUIPS in each school-community was too short. Behavior change is a slow process. As one person said, “You can’t tell a seed to grow faster; you can only give it plenty of sun and water and tie the young plant up until it is strong enough to survive on its own.” The original design of the project called for a 36-month treatment at each site: a year to learn, a year to practice with support, and a year to take ownership of new behaviors and integrate them into regular practice. • Operating on the basis of assumptions leads to difficulties. A key assumption of QUIPS was that intrinsic motivation related to a desire for professional growth, and improved performance of duties is enough to motivate teachers and other staff to invest time and energy to learn and then adopt new approaches to their work. This assumption indeed held for some. But there remained a considerable number of teachers and other GES officers who were not willing to make such a commitment without more tangible financial inducements. These could include higher per diem allowances, in-kind benefits like study tours, opportunities for further formal education, or credit for project participation that could lead to promotions or be applied to further studies.116 • Behavior change is difficult and expensive. Learning and then adopting different professional behavior is hard work. Old ways have to be discarded and the new understood, practiced, mastered, and then integrated into the learner’s everyday practices. For many teachers and district personnel, the gap between their entry level and the new learning was too great. Moreover, the task was seen sometimes as an imposed burden for which there was no tangible reward. Many teachers had to decrease the income-generating activities they needed to supplement their earnings as teachers in order to participate in QUIPS schools. • Final evaluation and feedback to stakeholders brings closure. Stakeholders of a program like QUIPS want to know how they contributed to the final results 116 In its review of the draft evaluation, the Academy for Educational Development noted that: “It could be more strongly pointed out that QUIPS did, in fact, include some attention to human resource issues (e.g. review of personnel practices, salary/career progression in relation to performance which took place even before the school and community interventions started. Thus, QUIPS did assume and recognize that more was needed than desire for professional growth in order for teachers to continue to apply new teaching methods and practices. However, MOE/GES was constrained in various ways in its ability to reform its systems to address these concerns, even with NPA support from USAID.” 149 and what those results are. It is important to ensure that they receive feedback from this study. • Final evaluation results will inform the donor community and contribute to future development plans. It will be some time before education in Ghana, and indeed across the developing world, will meet the ambitious goals set by local, national, and international bodies. To the extent that lessons about what worked and just as critically, what did not work, are shared, the more likely it is that lasting solutions will be found. Surely the children of Ghana and the world deserve no less. CHAPTER 11: KEY CONCLUSIONS AND RECOMMENDATIONS The purpose of the QUIPS Final Evaluation was to: • Assess the impact of the program on pupil learning and investigate factors that explain: (1) differences between high- and low-performing schools, districts- and regions; (2) the relatively modest learning achievement gains by pupils in QUIPS schools compared to matched controls, especially in English and (3) the narrowing of the performance gap between QUIPS and control schools during the last three years of the program; • Understand better the factors that are associated with improved pupil learning in Ghana; • Inform USAID and implementing partners about what worked and did not work under the QUIPS program; and • Establish whether QUIPS-targeted good practices have been sustained and spread in Ghana primary schools, communities, and district education offices. 11.1 IMPACT OF QUIPS ON PUPIL LEARNING One unusual practice of the QUIPS program was to track longitudinally individual pupil learning in QUIPS and control schools. Ultimately, it allowed this Final Evaluation to study more closely and comment with confidence about the impact of the program on pupil learning. QUIPS did make a difference in pupil learning. It positively shifted the learning curves of primary school pupils participating in QUIPS compared to pupils in the matched control schools. Given that the interventions were active for a relatively short time, two years, the outcome on learning is remarkable. Moreover, the longitudinal nature of the data allowed the evaluation to predict pupil learning over time. Follow-on analyses indicated that if the school improvements introduced under QUIPS were to be sustained, the achievement advantage of QUIPS pupils would increase over time. For these long-term learning outcomes to be realized, however, the classroom reforms, teacher supervision, and district and community support must be maintained also. If that does not happen, there is likely to be a return to the status quo. Although the impact on pupil learning was statistically significant, the gains were not large. Descriptive analyses of pupil performance pointed to some possible explanations for the limited gains. Children, it has been found, develop basic literacy and numeracy by the third or fourth year of formal schooling. Yet most public school pupils in Ghana are not reading with meaning until the fifth or sixth year. Math skills are also one and a 150 151 half to two years below grade level. For example, most children cannot compute simple subtraction problems until the end of Grade 5. Given such delays in academic development, program planners and educators must not expect substantial achievement gains, especially for the upper primary grades, without a specific literacy and numeracy enhancement effort. Such was not part of the QUIPS intervention. The lack of critical inputs (such as textbooks and the teachers who know how to use them and supplementary materials for children to gain reading practice) further attenuated the potential gains delivered by the QUIPS interventions. 11.2 BETTER UNDERSTANDING OF THE FACTORS RELATED TO PUPIL LEARNING The following conclusions are drawn from the findings of the study, which took into account quantitative data collected during and up to a year after all program implementation ended and also qualitative information gathered through interviews among sample QUIPS and control groups in northern, middle and southern Ghana. The conclusions start with those directly affecting children and move on to project implementation issues. 11.2.1 Child-Related Conclusions • The evaluation teams found that many children started the school day hungry and tired, thus undermining their ability to learn. All the groups with whom the evaluation teams interacted with spoke of how children’s readiness to learn was affected by hunger. In QUIPS low-performing communities, parents often were neglecting the basic needs of their children. Interviews with children revealed that the vast majority of parents were not ensuring that their children’s basic food needs were met before and during school. • Pupils can not be expected to make appreciable gains within a two-year intervention period in the absence of accelerated programs in literacy and numeracy. Pupils throughout Ghana were performing far below development expectations in English reading and mathematics. Two years of QUIPS interventions was simply not enough time to remedy this deficiency. 11.2.2 Community-Related Conclusions • QUIPS-fostered community involvement in school management has had a lasting impact on the quality of education and resultant learning. Strengthening the role of SMCs/PTAs in primary education was the broadest QUIPS intervention, and because communities are far more stable than teachers or district personnel, the impact is more likely to last. QUIPS also strengthened the internal leadership structures in communities and encouraged communities to contribute and plan towards school improvement. The community focused 152 interventions had a positive correlation with high performing schools and enhanced learning outcomes. 11.2.3 School-Related Conclusions • During each two-year intervention cycle, the QUIPS program achieved the planned results identified by the SO2 Results Framework for each cohort. By the second year of the QUIPS intervention, pupils in QUIPS schools were able to read with meaning more than control schools. Further, as a consequence of the perceived reputation of the QUIPS schools as being “good,” enrollments, particularly of girls, were increased in these schools. Nonetheless, in many schools these gains have not been sustained, in large part because the basic conditions that applied during QUIPS, especially staffing and supervision, have not been maintained. • QUIPS infrastructure development was successful in facilitating community involvement. Teachers’ work conditions in the QUIPS schools improved and schools stayed open during the rainy season. QUIPS micro-grants to promote school infrastructure development built positive relations between schools and communities. • The QUIPS Program achieved an immediate return on training in that teachers were responsive and implemented most of the reforms in QUIPS classrooms. Specific targeted QUIPS teacher training and support activities were associated with high pupil learning during the two-year intervention cycle, particularly encouragement of pupils, especially girls, to participate; teacher questioning and feedback to pupils; and facilitation of pupil interaction and creative thinking. However, there is little evidence that these classroom reforms were sustained. The evaluation identified a number of factors impeding sustainability; the most serious was teacher mobility. Given the diffuse distribution of QUIPS schools across the nation, and an in-service training program targeting teachers in these schools, there was insufficient support for teachers to carry on the new practices over the long run. • Education reforms targeting improvements in teacher instructional practices that fail to address systemic issues related to teaching (recruitment, training, remuneration, conditions of service) cannot be sustained. The QUIPS assumptions that desire for professional growth, improvement in the performance of duties, or commitment to the teaching profession would be sufficient to motivate teachers were misplaced. • The positive impact of effective head teachers on learning was clear in high-performing schools, whether of whether the school received QUIPS assistance or whether it was a “control” school. Enhancing school leadership 153 pays off by increasing school effectiveness. Further, the scaling up and spread of QUIPS influenced the practices of head teachers across districts observed by the evaluation teams and revealed that QUIPS was successful in spreading good practices in non-QUIPS schools. However, better head teachers must have better material support if they are to do effective training in their schools on their own. • High mobility of teachers and district personnel trained by QUIPS undermined the long-term effectiveness of in-service training for teachers. • While QUIPS in-service training exposed teachers to a variety of specific child-centered instructional and classroom management strategies, few of these were observed in practice during the final evaluation. The teachers who provided regular positive reinforcement and involved children actively in a friendly learning environment were seen most often in high-performing schools. • Absence of teaching aids, especially textbooks and supplementary reading materials continued to be a serious impediment to effective learning among children at all grade levels, but especially at the primary level, Grades 1-6. The evaluation team’s quantitative results from the analysis of achievement testing speak very well to the fact that children may have some vocabulary at grade level, but they are about two years behind in reading sentences with meaning. • The relevance in the learning context is supremely important. The team’s studies of what pupils were and were not able to do showed that mathematics story problems using common experiences of children were completed at higher grade levels (even to a smaller extent, problems at the same grade level of the pupil) than basic mathematics operations. • Teachers are encouraged to teach to the class level syllabus in spite of the fact that children are functioning two or more classes below class level. Even though teachers in QUIPS were encouraged to help students lagging behind and were exposed to continuous assessment, there was little evidence that these particular skills were perfected and are not being applied today. • High teacher absenteeism continued to undermine pupil learning in spite of QUIPS interventions. 11.2.4 District-Related Conclusions • The training provided by QUIPS for DEO personnel enhanced their capacity to operate more effectively in management, supervision, planning, and M&E. Including district officers in training activities for teachers and SMC/PTA 154 executives gave them learning opportunities and modeled effective ways to operate. Many of the activities that the district officers undertook as part of the QUIPS experience are no longer being done because funds to support them are not yet forthcoming. • Some districts reported increased capacity to access funds from other agencies as a result of their QUIPS experience, permitting them to spread good practices emanating from QUIPS to other schools. This created a realization that being proactive with well-grounded initiatives can increase a district’s ability to address locally identified needs. • Since all districts were involved in the QUIPS experience, the impact of the program on local management has the potential to influence delivery of primary education everywhere in Ghana. As decentralization moves more and more responsibility to the districts, this core of expertise will become better used. • Under QUIPS, district support and attention to school demands and supervision was found to be a major contributor to a school's success in teaching. However, the amount of district support provided to the QUIPS schools served to marginalize other schools in the district. Furthermore, after the QUIPS intervention ended, district supervision of the QUIPS schools deteriorated. 11.2.5 Institutional and Project-Related Conclusions • Programs that attempt to address such a far-reaching phenomenon as national primary education require sensitive planning and greater coordination with the agencies of government. Eventual success depends upon common understanding, agreement, and commitment, both at policy decision-making and program implementing levels. • When implementing partners have different understandings of the time required to accomplish goals or their capacity to act within the agreed timeframes, dissatisfaction can result. • When design considerations were compromised during initial (1997) negotiations between USAID and the GoG, the potential for realizing QUIPS objectives also was compromised. All stakeholders must understand and agree to the ramifications of altering the design before implementation begins. • In a program of the scope of QUIPS, flexibility to meet emerging challenges is critical. While the midterm review led to significant changes in how QUIPS unfolded, nevertheless structural stringencies precluded responsiveness to locally identified needs. A case in point was in-service training for teachers. 155 Directed to enhancing the capacity of trained teachers, in remote areas it was delivered also to many untrained teachers whose ability to take advantage of what was being offered was limited by their lack of academic and professional expertise. • Implementation of QUIPS by implementing partners under different contracts led to areas of competition and overlap that contributed to inconsistencies and biases in how the program was implemented and evaluated. Projects of this scope need to clarify objectives and establish effective procedures and relationships before implementation begins. 11.3 WHAT WORKED AND WHAT DID NOT WORK 11.3.1 Head Teacher Management. Head teacher management was identified as an important factor in pupils’ learning, but QUIPS fell short of meeting the goals related to it. Head teachers were included as trainees in the on-site and residential teacher training by QUIPS. They were not groomed to be master trainers or managers of training who were responsible to sustain the in-service process that QUIPS started. Rather, it was assumed that the district office would initiate regular teacher upgrading each year and that they would, as before, hire specialist to do the training. The idea that head teachers themselves would prepare and conduct training for the six other class teachers at the school was seen as unrealistic and unacceptable, except in the north where clear status differences existed between trained head teachers and contract untrained teachers.117 QUIPS management training of head teachers focused upon office management, data collection, school-community relations and other topics found in the Head Teachers Handbook. Upgrading of instructional skills was given less priority within the scope of work of the head teacher. For example, two staff meetings per term was the target set by the GES for the head teacher. 11.3.2 District Management. District management was another area where the expected results were not achieved. With Ghana now moving more and more toward decentralized management of education and considering a national capitation grant 117 In its review of the draft evaluation, the Academy for Educational Development stated: “Head teachers together with Circuit Supervisors did indeed get special training in coaching of teachers. However, implementation was compromised by the fact that many head teachers were not “detached” and thus did not have adequate time to provide coaching and mentoring. Moreover they were often pulled off by the DEO to do other tasks away from the school. In addition, some head teachers were promoted to these positions without having the strong teaching skills as part of their repertoire. However, having said this, there were real concerns within the QUIPS/ILP project team that the amount of training we provided, given the skill level of the head teachers, was probably not adequate. Resources and time were indeed limitations in overcoming this problem.” 156 scheme,118 the need for capacity development in planning and management is even more important. As with much of the QUIPS program, the attempt to reach every district in the nation compromised the impact and impeded sustainability. 11.3.3 District Grants. The district grant mechanism introduced at midterm enhanced district management capacity, particularly capacity to use its own staff to deliver in￾service training. The continuance of district training in the QUIPS approaches has helped spread the ideas to a larger number of classrooms in Ghana. Still, it seems teachers are not able to implement the classroom reforms without more widespread systemic change in the structure of the profession. Through the DCM, certain management and financial systems were put into place that improved district operations, but there is little evidence of sustained impact. 11.3.4 District Training in Monitoring and Evaluation. The M&E training provided through the QUIPS district grants led to a shift in thinking about district responsibility and accountability. The skills developed in collecting and, most important, using data to inform management decisions are remarkable, although their sustainability is challenged by shifting district priorities and funding limitations. 11.3.5 Community Initiatives. Building national awareness about the responsibilities of communities to support schooling was one of the major accomplishments of QUIPS. The importance of empowered parents and local authorities working with school personnel to improve instructional quality was emphasized by stakeholders throughout the evaluative fieldwork. Mainstreaming of SMC/PTA training across the nation has built awareness of the need to involve communities and share responsibility for primary education. 11.3.6 Project Design and Management. With teacher mobility within and exiting the system as widespread as it is in Ghana, it cannot be expected that classroom reforms will be sustained without systemic change at least within the district. The “diffuse” approach of working with only three schools per district also undermined the district management support program. Because personnel was not concentrated in any one district, the advisory support and mentoring needed when instituting shifts in management process and practices was diffused. An alternative approach would have been to concentrate the program in larger clusters of schools within only a few districts. 118 The capitation grant system is a five-year experiment by the Ghana Education Service supported by the World Bank. It is expected to be launched in the 2005-06 academic year. Capitation grants are like impress funds that are allocated through the district education offices to head teachers for the day-to-day operations of schools. The amount of funding is based upon enrollment figures taken in October. Schools receive more grant money for each girl student enrolled than for each boy student enrolled. At present, the grants are 25,000 cedis per year per boy student and 35,000 cedis per girl student. Assuming a primary school enrollment of 200 pupils with equal numbers of girls and boys, the head teacher will have a capitation grant of 6.0 million cedis or $667 per year. 157 One of the positive aspects of the design of the QUIPS program was its integrated approach—but the integration often was difficult to achieve. One reason was a lack of shared vision and teaming needed to ensure that all implementing agencies worked together coherently. The QUIPS program was implemented through four separate agreements. Among the problems this situation created were duplication of effort, competition, fragmentation, and lack of continuity between the approaches in the north and elsewhere. Without a deliberate and strategic effort to encourage cooperation, integrative approaches rarely realize their potential value. QUIPS provided an exemplary model for data collection and M&E. The use of longitudinal measures of change, both in pupil achievement and targeted behavioral change (instruction, head teacher support, parent involvement, community practices, and so on) gave implementers a powerful tool for monitoring pupil learning outcomes and understanding the basis for change. Also exemplary was the QUIPS investment in collecting data after the period of active QUIPS interventions as well as throughout the program in matched control schools. Maintaining such data in an integrated database is essential for efficient access, analysis, and use of information. The importance of using external data collectors and of coherency in data collection processes should not be overlooked. According to the results from the evaluative field work, the involvement of the MOE and GES in implementation planning was far from adequate. This truly had implications for the ultimate spread and sustainability of QUIPS good practices. 11.4 THE SPREAD AND SUSTAINABILITY OF QUIPS GOOD PRACTICES 11.4.1 Schools. The mobility of teachers and the diffuse distribution of QUIPS meant there was insufficient support for sustaining QUIPS good practices over the long term. Sustained reform can only be achieved when new skills are reinforced by fellow teachers and head teachers, communities, and particularly national systems and policies. To enhance the sustainability of any school reform, development programs must work within existing systems. The QUIPS-targeted classroom reforms were not sufficiently mainstreamed to provide the reinforcement needed to sustain them. The attempt to introduce QUIPS in every district in the nation, rather than concentrating the approach in a few districts, meant that it was impossible to build the critical mass needed for systemic change and support for teachers to sustain classroom reforms. Only where a strong head teacher and most of the QUIPS-trained teachers were still together and supported by the community were the classroom reforms sustained. 158 Furthermore, reforms related to teacher instructional behavior need to address more issues directly related to the integrity of the teaching profession, including recruitment and deployment, remuneration, incentives for teaching in remote regions, and other conditions of service. QUIPS attempted to address some of these basic issues using NPA, but the benchmarks were not achieved and the NPA was moved to project assistance in the form of the district grant mechanism strategy for spread and sustainability. The District Grant Mechanism did spread the new teaching practices but, as before, the approaches were not mainstreamed into the pre-service education program. Without sufficient support for teachers trying to implement new methods, they are likely to regress to the conventional approaches. 11.4.2 Districts. The evaluative field work identified some sustainability in the individual human resource capacities built through the district grant mechanism, particularly the M&E training, but this is challenged in the districts by shifting priorities, limited opportunities to practice new skills, and limited funding. 11.4.3 Communities. Although there was evidence that some good community practices had waned, there remained widespread recognition by parents, community leaders, and others of the responsibilities they have in supporting their education programs, even where targeted good practices were on the decline. The importance of a strong SMC/PTA and of parents in promoting pupil learning is recognized not only locally but also at the district and national level. The evaluation team considers the spread and sustainability of community good practices to have been successful. 11.4.4 Management Implications for Spread and Sustainability. The QUIPS Final Evaluation provided an opportunity to review the wealth of data reported by the implementing partners. On-site observations and interviews with stakeholders communicated clearly the complexities of reform efforts on a national scale. It was dramatic to witness the state of beneficiary stakeholder groups once external change agents and financing were withdrawn. To the extent that changes could be supported by current resources, they have lasted, but that is not true in far too many schools and districts. Without the support available during the program, most QUIPS schools have begun to revert to their former states. 11.5 RECOMMENDATIONS These recommendations flow from the conclusions of this evaluation. They identify potential actions USAID might consider. 159 11.5.1 Child-focused Recommendations • Current USAID-funded school feeding programs in areas where food security is an issue should be continued. A reassessment should be undertaken to ensure that coverage in fact addresses areas in need to ensure that children’s learning is not jeopardized by malnutrition. • Children’s voices should be included in the design of programs addressing educational reform. 11.5.2 Community-related Recommendations • Future education reform activities should include modules for parents and communities on the importance of feeding to child learning. They should include activities to help parents prioritize spending to allow for support for their children’s education. • The stability of the community makes it a good center for activities designed to foster involvement in school management. • Investment in the SMCs/PTAs by government and its development partners should be continued through the District Assemblies as well as the GES. Integrated planning and budgeting in financially viable assemblies has funded and made use of DEO technical expertise to implement innovations in primary schools as well as the initiatives from SMCs/PTAs. 11.5.3 School-related Recommendations • The demonstrated effectiveness of strong head teachers points to this group as an ideal target for continued intervention to improve primary schools, though interventions that enhance the head teachers’ capacity to deliver and support teacher training also must ensure adequate support for such activity. • Infrastructure projects for rural schools should be continued as vehicles for improving community management capacity and teacher retention and for coping with increasing enrollment. Construction of classrooms, latrines, and teacher accommodation should remain an integral part of USAID education projects. 11.5.4 District-related Recommendations • As decentralization proceeds in Ghana the capacity of both District Assemblies and DEOs to manage change and finances will grow. This is an 160 area that will become increasingly important in new education development programs. 11.5.5 Institutional and Project-related Recommendations • Longitudinal measures of change in pupil achievement and targeted behavioral change in instruction, head teacher support, parent involvement, and community practices are critical to measuring learning outcomes and their source. The model of an integrated database that QUIPS provided is exceptional and is recommended for any school quality reform program. • Donors and other stakeholders need to influence systemic reforms by working with both the formal teacher training system and the Ghana National Association of Teachers. The latter has considerable influence over teachers and would be an ally for reform. If clear concerns of GNAT members, such as allowances, were negotiated at this level, disagreements about them could cease to be impeding factors. Donor projects need to consider allowances for personnel who are assigned activities that are beyond the scope of normal classroom and office practice. • Both pre-service and in-service teacher training are critical to build a teaching force capable of meeting the learning needs of Ghana’s children. Project support must address both, and thought should be given to how to integrate the two in order to maximize use of resources. • USAID requires implementing partners to measure and report results. At times the reporting requirements distract from the delivery of the technical assistance and training that is the focus of the project. The system of deliverables, data, and deadlines needs to be adjusted to allow for greater flexibility. The intent to measure results should also encourage implementers to seek opportunities to better serve clients, including being on-call for unforeseen requests for assistance and support. REFERENCES Academy for Educational Development. (AED) (2004). Quality Improvement in Primary Schools: Improving Learning through Partnerships. A Report Submitted to USAID, Accra, Ghana. Adams, D. (1993). “Defining Educational Quality.” Educational Planning 9:3-18. Addae-Mensah, Ivan (2002). Schooling and Social Mobility in Ghana, (lecture series at the University of Ghana). Angmore, C Jakalia, A., Dzaka.D., & Asante, K. (1997). “Literature in the JSS Language Classroom.” In M.E.K Dakubu (ed.), English in Ghana, 157–172. Accra: Black Mask Publishers. Bacchus, K. et al. (1991). Curriculum Reform. London: Commonwealth Secretariat. Boardman, G.R. and R.L. Evans. Community Support for Education: Local Change Leaders. Education Development Center. Undated. Boardman, G.R., R.L. Evans and K. Leherr. Local School Partners: Training, Development and School-Community Change. Education Development Center. Undated. Bernbaum, M. et al. (1999). Promoting Primary Education for Girls in Malawi. Impact Evaluation, USAID. Bourdieu, P. (1977). Outline of a Theory of Practice. Cambridge: Cambridge University Press. Casely-Hayford, L. (2004). A Social Appraisal of the Education Strategic Plan in Ghana. Associates for Change Working Paper 1, November. Casely-Hayford, L. (2003). Complementary Education Systems in Ghana. Oxford Education Conference Paper, September 2003. Casely-Hayford, L. (2001). “How the Poor Get Poorer: An exploration into the participation, quality and needs of female teachers in rural deprived areas of Ghana.” Accra: Ministry of Education. Catholic Relief Services (2004). Quality Improvement in Primary Schools. CRS/QUIPS End-of-Project Report. 161 162 _____________, CRIQPEG (1996). Fostering English Language Learning in Ghana. Centre for Research on Improving the Quality of Primary Education in Ghana (CRIQPEG) (1996). A Report on 1996 Performance Assessment of Primary Six Pupils. Christensen, P, et al.(1997). Kids, Schools & Learning: A Retrospective Study of USAID Support to Basic Education in Sub-Saharan Africa. Technical Paper No. 56. Creemers, B., Peters, T. & Reynolds, D. (eds, 1989). School Effectiveness and School Improvement. Rockland, Mass: Swets & Zeitlinger. Delors, J., et al. (1996). Learning: The Treasure Within: Report to UNESCO of the International Commission of Education for the Twenty-first Century. Paris: UNESCO (www.unesco.org/delors/). Dzameshie, A. (1997). “Toward a Communicative Approach to Teaching English as a Second Language in Ghana.” In M.E.K. Dakubu (ed.), English in Ghana 173- 194. Accra, Ghana: Black Mask Publishers. Dzinyela, J.M (2001). Transforming Language Policy through Dialogue and School Based Research. Improving Educational Quality (IEQ) Project. Education Sector Review, Ministry of Education, Accra, Ghana (2004). Educational Assessment and Research Centre (EARC) (2002). A Comparative Study on Private and Public Primary Schools in Ghana. Study conducted for USAID/Ghana/EDU. EFA Global Monitoring Report. (2004). Understanding Education Quality. EFA Report. UNESCO.org.portal.unesco.org/education/en/ev.php￾url_id=35924&url_do=do_topic&url_section=201.html. Etsey, K.F (2004). The Effects of Comprehension Strategy Instruction on Ghanaian English Language Learners’ Comprehension Processes and Text Understanding. Dissertation. University of Iowa. Fianu, S.K (1993). PREP Evaluation Procedures for E.I.P Projects. Accra: Ministry of Education (MOE). Foucault, M. (1977). Discipline and Punish: The Birth of a Prison. Translated by A. Sheridan. Harmondsworth: Penguin. 163 Freire, P. (1985). The Politics of Education: Culture, Power and Liberation. Translated by D. Macedo. London: Macmillan. Fuller, B.; Clarke, P. (1994). “Raising School Effects While Ignoring Culture? Local conditions and the influence of classrooms, tools, rules and pedagogy.” Review of Educational Research 64:119-57. Giroux, H. (1993). Living Dangerously. New York: Peter Lang. Giroux, H. & McLaren, P. (1993). Between Borders: Pedagogy and the Politics of Cultural Studies. New York: P. Lang. Goody, E. & Bennett, J. (2001). “Literacy for Gonja and Birifor Children in Northern Ghana.” In D.R. Olson & N. Torrance (eds.), The Making of Literate Societies. Oxford: Blackwell Publishers. Government of Ghana (2003). White Paper on the Report of the Education Reform Review Committee. Haddad, W.D. et al. (1990). Education and Development: Evidence for New Priorities. World Bank Discussion Paper 95. Hartwell, A & Vargas-Baron, E. (1998). Learning for All : Policy Dialogue of Achieving Educational Quality. Prepared for the International Working Group on Education in Munich, Germany, June 23-26 1998. ( http://21learn.org.) Heyneman, S.P. (1989). “Third World School Quality: Current Collapse, Future Potential.” Educational Researcher. 18:12-19. Leherr, K., Arkorful, K., Boardman, J. & Israel, R. (2004). Community School Alliances Project. Report submitted to USAID, Accra, Ghana. Klauss, R. et al. (2004) A Qualitative Study of Factors Related to Pupil Achievement in QUIPS/ILP Schools, Accra: USAID/Ghana/EDU. Kraft, R. (1994). Teaching and Learning in Ghana: A Curriculum, Textbook, Syllabus and Handbook Analysis. Report submitted to USAID, Accra, Ghana. Kraft, R. (1995). A Tale of Two Ghanas: The View from the Classroom. Report submitted to USAID, Accra, Ghana. Kraft, R. (2003). Primary Education in Ghana. Report submitted to USAID, Accra, Ghana. 164 Lockheed, M.E. & Hanushek, E. (1988). “Improving Educational Efficiency in Developing Countries: What Do We Know? Compare 18:21-38. Lockheed, M.E., & Verspoor, A. (1991). Improving Primary Education in Developing Countries. World Bank/OUP. Manu, S.Y (1993). In-Service Education for Teachers in Ghana. Accra: PREP/MOE/Literacy House. Manu, S.Y. (1993). Report of PREP Mid-Term Evaluation. PREP/MOE. McEwan, P.J. (1999). “Recruitment of Rural Teachers in Developing Countries: An Economic Analysis. Teaching and Teacher Education 15:849-59. McLaren, P. (1994). Life in Schools: An Introduction to Critical Pedagogy in the Foundations of Education. 2nd ed. New York: Longman. Moulton, J., Mundy, K., Welmond, M., & Williams, J. (2001). “Paradigm Lost: The Implementation of Basic Education Reforms in Sub-Saharan Africa.” Technical Paper No. 109 published by Greenwood Press, Westport, Conn., 2002 (www.greenwood.com) Obeng, E.A.; Opare, J.A.,and Dzinyela, J.M (2003). The Role and Contribution of Higher Education to Improving the Quality of Basic Education. A case study of CRIQPEG. Raudenbush, S.W., & Willms, J.D. (eds, 1991). Schools, Classrooms & Pupils: International Studies from a Multilevel Perspective. New York: Academic Press. Review Committee (1990). White Paper on the Report of the Education Reform.The World Declaration of Education for All. (1990). http://www.ilo.org/public/english/employment/skills/recomm/publ/036.htm. School Effectiveness in Developing Countries – A Summary of the Research Evidence. (1993) Education Research Paper No. 01, p.25 Peter et al (2000) Schools that Learn, Doubleday, New York Skinner, B.F. (1968). The Technology of Teaching. Englewood Cliffs, N.J/London: Prentice-Hall. Vygotsky, L.S. (1978). Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press. 165 Yakubu, J.M. (1993). Executive Summary of Research on the Availability, Source and Utilization of Materials in Six Primary Schools in Ghana. Cape Coast: CRIQPEG. 166 ANNEX 1: LIST OF INTERVIEWS AT NATIONAL, DISTRICT AND COMMUNITY LEVELS Group or Person Interviewed (if no name put the position of the person) Location (Country, District and Village ) Organisation/Institution National Stakeholders Interviewed Mr. J.O. Afrani Accra PBME Sarah Agyeman Duah Accra CRDD Mrs. Nancy Opoku Head Accra GEU GES Mr. Chris Hammond, (Equal Liaison Officer) Accra Basic Education GES Capt. (Retired) Simon Tengabo, (Decentralization Officer) Accra Basic Education, GES Ash Hartwell USA University of Massachusetts Rudi Klauss USA AED Mr.Steven Manu, former Director, ILP Accra AED-CEDEM Kay Leherr, former CSA Chief of Party Accra EDC – EQUAL Elsie Menorkpor Accra USAID Daniel Ayugane Tamale CRS Adama Jehanfo Tamale CRS Deputy Director of Education Tamale GES. Tamale Rose Gobibilla Tamale GES, Regional Officer Zabzugu Tatale District Stakeholders M.S. Abdul-Rahman- District Director of Education Zabzugu District Education Office V.K. Nkansah - A/D Supervision Zabzugu District Education Office J.Y Awuah - A/D Finance and Administration Zabzugu District Education Office B. Seidu Alhassaan – A/D Statistics Zabzugu District Education Office Nyognu K. Isaac- Budget Officer Zabzugu District Education Office Awak Attah - ICA Ghana Project Officer Zabzugu District Education Office Sheini Paul Alhassan - Guidance/Counselling Coordinator Zabzugu District Education Office Issah Iddrisu - PRO/Community Participation Coordinator Zabzugu District Education Office Margaret Haruna - Primary School Head Teacher Zabzugu District Education Office Alhassam A. Adam, A/D Human Resource Management and Development/DHA Zabzugu District Education Office Margaret Haruna - Head Teacher Zabzugu District Education Office Abdul Rahman A. Razau, Head Teacher Zabzugu District Education Office Awaki Attah ICA Project Officer Zabzugu District Education Office Ivan Gam District Coordinating Director Zabzugu District Education Office Isahaku Yakubu District Finance Officer Zabzugu District Education Office Abukaru Inusa, District Planning Officer Zabzugu District Education Office School/ Community Stakeholders Mohammed Shaibu, Head Teacher Nakpale Kworle Nakpale Kworle Primary School Ziblim Alhassan Teacher Nakpale Kworle Nakpale Kworle Primary School Issahaku Zakari SMC Chairman Nakpale Kworle Community member Iddisi Cherifir, PTA Chairman Nakpale Kworle Community member Issahaku Osuman, Assemblyman, SMC Nakpale Kworle District Assembly Gani Tiyawumya SMC Secretary Nakpale Kworle Community member 167 Saaka Fuseini SMC (Old Pupil) Nakpale Kworle Community member Mariama Alidu SMC Member Nakpale Kworle Community member Mohammed Mahama PTA Second Trustee Nakpale Kworle Community member Haruna Sulemana, Unit Committee Member Nakpale Kworle District Assembly Saaka Aminu, Unit Committee Member Nakpale Kworle District Assembly Sana Imoro PTA Member Nakpale Kworle Community member Head Teacher Tusundo Tasundo L/A Primary School Hardi Gomba , Teacher Tusundo Tasundo L/A Primary School Third teacher on first aid training (UNICEF) Tusundo Tasundo L/A Primary School Balimbu Tasun, Chief Tusundo Chief Bachabuli Tabori - Elder – linguist Tusundo Traditional Council Bipom Tasun Elder Tusundo Traditional Council Libak Jagir, Elder Tusundo Traditional Council Likipinyanli Lappe, Elder Tusundo Traditional Council Kadin Tasun, Elder Tusundo Traditional Council Wumborben Jagir, Elder Tusundo Traditional Council Magambe Bamondow, Elder Tusundo Traditional Council Nibon Kabuja Tusundo Traditional Council Kitondo Bondan Tusundo Traditional Council Nbigna Lapee Tusundo Traditional Council Yilyire Jagir Tusundo Traditional Council Blishin Nditim Tusundo Traditional Council Langbo Tabori Tusundo Traditional Council Nlambe Tasun Tusundo Community member Domoi Lapee Tusundo Community member Tinyibor Gmanyim Tusundo Community member Tingeri Ndetin Tusundo Community member Bachatob Nigarn Tusundo Community member Gmayasam Ndetin Tusundo Community member Kwame Balinbu Tusundo Community member Balimbo Kwame, PTA Chairman Tusundo Community member Jofi Korath, PTA Organizer Tusundo Community member Kunjabu Nkunda, SMC Executive Tusundo Community member Hardi Gumba, PTA Secretary Tusundo Community member Baluma Kuma, Unit Committee Member Tusundo District Assembly Kunjab Nkunba, Unit Committee Member Tusundo District Assembly Rachia Kapinya, SMC Member Tusundo Community member Bibanlib Benan, PTA Member Tusundo Community member East Mamprusi District Stakeholders G.B.C Libiedem Gambaga District Director of Education A/D, Human Resource Management and Development Gambaga District Education Office A/D, Supervision Gambaga District Education Office A/D, Finance and Administration Gambaga District Education Office A/D Inspectorate Gambaga District Education Office M&E Co-ordinator Gambaga District Education Office School/Community Stakeholders Dauda Yaatun Kpanlore Community member Chief Representative. SMC Kpanlore Traditional Council Elders Kpanlore Community member Yayee Tiikam, SMC Kpanlore Community member Ibrahim Adam Kpanlore Community member Sule Ibrahim Kpanlore Parent James Lambongu Kpanlore Parent Adam Abu Kpanlore Parent 168 Dahamani B., Head Teacher, Assemblyman Kpanlore Kpanlori Primary School Samuel Gumah, Teacher Kpanlore Kpanlori Primary School Pupils – Grade 6 girls, Grade 6 boys Kpanlore Kpanlori Primary School Harold T. Yamusan Gbangu - Bomgbani Gbangu Primary School Elders Gbangu Chief Yidana Baako Gbangu Community member Issah Daamaka Gbangu SMC, Chairman Women Gbangu PTA Chairman Safura Tanko Gbangu Community member Samuel Kombat D. Gbangu Mother James K. Bayensi, Head Teacher Gbangu Gbangu Primary School Anthony Bawa, Untrained Teacher Gbangu Gbangu Primary School Alidu Mahama, Volunteer Teacher Gbangu Gbangu Primary School Chairman, PTA Gambaga District Assembly, S.S.C Grade 6 girl pupils, Grade 6 boy pupils Gbangu Gbangu Primary School Tolon Kumbungo District Stakeholders Yakubu Assani Tolon – Kumbungu District Director of Education John K. Duncan Tolon – Kumbungu District Education Office Dauda Abagri Tolon – Kumbungu District Education Office Adams Felix Tolon – Kumbungu District Education Office E.S. Alhassan Tolon – Kumbungu District Education Office Aboulai E. Issifu Tolon – Kumbungu District Education Office Sulemana S. Alhassan Tolon – Kumbungu District Education Office School/Community Stakeholders Yabdulai Seebaway, Head Teacher Kpalgun Kpalgun Primary School Ben, Teacher Nawuni Nawuni Primary School Mohammed Abdul Nawuni Nawuni Primary School Zacharia I. Nawuni Nawuni Primary School Abugri J.A. Nawuni Community member David Dotoe Nawuni School Pupil Amo Christian Nawuni School Pupil Rita Anaba Nawuni School Pupil Damata Bugri Nawuni School Pupil Zakaria Azara Nawuni School Pupil Chief Elders Nawuni Community member Women Nawuni Community member Assemblyman Nawuni Community member Co-ordinating Director Tolon-Kumbungu District Assembly East Gonja District Stakeholders J.A Zakaria, A/D Supervision Salaga District Education Office J.J., AD Statistics Salaga District Education Office E.K. Nyonyone, A/D Human Resource Management and Development Salaga District Education Office Saaka A. Mustapha, PRO Salaga District Education Office Mumuni Mahada, A/D-C/S Salaga District Education Office Edward Nyonyoni, A/D Human Resource Management and Development Salaga District Education Office I.M Harwa, CPC Salaga District Education Office J.J. Dramanu, A/D Statistics Salaga District Education Office Nyuma Charles, Training Officer Salaga District Education Office 169 Danso Bamapong, DMIT Coordinator Salaga District Education Office A.A. Ziblim, Partner Supervisor Salaga District Education Office Sikena Haruna, District Girls Education Officer Salaga District Education Office David Ninkabs, Circuit Supervisor Salaga District Education Office Mr. Ngumah Charles, District Training Officer Salaga District Education Office Tanko Iddrisu, Circuit Supervisor Salaga District Education Office Kofi Obakoasem, Circuit Supervisor Salaga District Education Office Somo, Veronica, Teacher Salaga District Education Office Mohammed Jakpa. Circuit Supervisor Salaga District Education Office Mr. J.B. Atogiba, District Coordinating Director Salaga District Education Office Mr. Owusu Frempong Peter, District Accountant Salaga District Education Office School/ Community Stakeholders Christopher Anane, Head Teacher Lonto Lonto Presby Primary Nyame Emmanuel Assistant Head, G4 Teacher Lonto Lonto Presby Primary Nana Wuran II – Chief Lonto Chief and Elders Nana Kofi Ntanu-Amankrado, Elder Lonto Traditional Council Amidu Bonosem- Elder Lonto Traditional Council Jacob Boti – Elder Lonto Traditional Council Kwadwo Mpani – Chief Linguist Lonto Traditional Council Kifi Bad, Chief’s Secretary Lonto Traditional Council Opoku Nyami Assemblyman Lonto District Assembly Kofi Kabole Elder Lonto Community member Kofi Donkor Elder Lonto Community member David Donkor, SMC Chairman Lonto Community member Daniel Anafo PTA Chairman Lonto Community member Opoku Nami Assemblyman Lonto District Assembly Alice Yaa Donkor, PTA Member Lonto Community member Mallam Fokuo SMC Member Lonto Community member Nsenyi David PTA Vice Chairman Lonto Community member Christopher Anane, SMC Vice Chairman Lonto Community member David Asempa, Unit Committee Member Lonto District Assembly Helina Anane Unit Committee Member Lonto District Assembly Kofi OkulemanyPTA Vice Secretary Lonto Community member Lydia Adjua Fordour PTA Member Lonto Community member Samuel Chamba, Head Teacher Ekumpe Ekumdi L/A Primary School Boakye Yaw, Grade 1B Teacher Ekumpe Ekumdi L/A Primary School Gyingyi Theresa, Grade 2 Teacher Ekumpe Ekumdi L/A Primary School Mboko Yaw, Grade 2A Teacher Ekumpe Ekumdi L/A Primary School Magana, Yaw, Grade 2B Teacher Ekumpe Ekumdi L/A Primary School Rebimatu Amadu, Grade 3 Teacher Ekumpe Ekumdi L/A Primary School Chilipa Yaw, Grade 4A Teacher Ekumpe Ekumdi L/A Primary School Dogyi A.K, Grade 4B Teacher Ekumpe Ekumdi L/A Primary School Chakuo Lazarus, Grade 5 Teacher Ekumpe Ekumdi L/A Primary School Nuamekye Daniel, Grade 6 Teacher Ekumpe Ekumdi L/A Primary School Nana Akwedi Ndes II –Odikono Ekumpe Traditional Council Nana Benye Kwadjo II Paramount Chief Ekumpe Chief Nana Asuu Dumoyeaker – Asafotse Ekumpe Traditional Council Nana Monymodoy Kojo, Elder Ekumpe Traditional Council Nana Yaw Saimbel, Basare Chief Ekumpe Traditional Council Hon. Emmuil Mbo, Assemblyman Ekumpe District Assembly Yaw Beyewa, Elder Ekumpe Community member Yaw Bawako, Elder Ekumpe Community member Obimyire Akusi, SMC Member Ekumpe Community member Mageri Mobolaa Konkomber, Asafotse Ekumpe Community member Hon B.H. Libi Kiomboi, Assemblyman Ekumpe District Assembly 170 Negulse – West Bank, Kumdi Ekumpe Community member Yamba Nsefo, Woman Ekumpe Community member Ama Nyando, Woman’s group Ekumpe Community member Abena Boa, Woman’s group Ekumpe Community member Kofi Bogusi, Konkomba Member Ekumpe Community member Dwepari Salweh, PTA Secretary Ekumpe Community member Mbo Emmanuek, Assemblyman Ekumpe District Assembly Yaw Asare, Unit Committee Member Ekumpe District Assembly Ama Ntoseh PTA Executive Ekumpe Community member Madam Gyingyl Teacher Ekumpe Community member Asun Lente Chief’s Representative Ekumpe Community member Eric Mboako Teacher Ekumpe Community member Oman Komante Yawa Teacher Ekumpe Community member Jachal Nimorti Unit Committee Member Ekumpe District Assembly Kango Niwee, SMC Member Ekumpe Community member Ahafo Ano South District Stakeholders District Chief Executive Mankranso District Assembly District Coordinating Director Mankranso District Assembly District Director of Education Mankranso District Education Office A/D Statistics Mankranso District Education Office A/D Statistics Mankranso District Education Office Community Participation Coordinator Mankranso District Education Office DMIT Co-coordinator Mankranso District Education Office School/ Community Stakeholders Head Teacher Adukrom Adukrom L/A Primary School Assistant Head Teacher Adukrom Adukrom L/A Primary School QUIPS Trained Teachers (4) Adukrom Adukrom L/A Primary School QUIPS Trained Teachers (3) Adukrom Adukrom L/A Primary School non QUIPS Trained Teacher(1) Adukrom Traditional Council Chief Adukrom Traditional Council Linguist Adukrom Traditional Council Queenmother Adukrom Traditional Council Market Woman Adukrom Community Members Assembly Man Adukrom Community Members SMC Chairman Adukrom Community Members PTA Chairman Adukrom Community Members Head Teacher Kunsu Dotiem L/A Primary GES Assistant Head Teacher Kunsu Dotiem L/A Primary GES Chief Kunsu Dotiem Traditional Council Queenmother Kunsu Dotiem Traditional Council Linguist Kunsu Dotiem Community member PTA Chairman Kunsu Dotiem Community member SMC Chairman Kunsu Dotiem Community member Woman’s Group Leader Kunsu Dotiem Community member Kwabre District Stakeholders Deputy District Coordinator Mamponteng District Assembly A/D Supervisor Mamponteng District Education Office A/D Human Resource Management and Development Mamponteng District Education Office A/D Statistics Mamponteng District Education Office 171 School/ Community Stakeholders Retired Head Teacher Ntribuoho Ntribuoho L/A Primary School Head Teacher Ntribuoho Ntribuoho L/A Primary School QUIPS Trained Teachers (3) Ntribuoho Ntribuoho L/A Primary School Chief Ntribuoho Traditional Council Otumfuo’s Bamuhene Ntribuoho Traditional Council Elders Ntribuoho Traditional Council Assembly Members Ntribuoho District Assembly Queenmother Ntribuoho Traditional Council SMC/PTA Chairman Ntribuoho Community members PTA Members Ntribuoho Community members Head Teacher Ntonso Ntonso SDA Primary School Assistant Head Teacher Ntonso Ntonso SDA Primary School Gyaasehene (Sub Chief) Ntonso Traditional Council Women’s Group Ntonso Community member Assembly Members Ntonso District Assembly SMC/PTA Chairman Ntonso Community member PTA Members Ntonso Community member Twifo Praso District Stakeholders District Coordinating Director Twifo Praso District District Education Office A/D Supervisor Twifo Praso District District Education Office A/D Human Resource Management and Development Twifo Praso District District Education Office A/D Statistics Twifo Praso District District Education Office Girl Child Coordinator Twifo Praso District District Education Office District Management Implementation Team Twifo Praso District District Education Office District Teacher Support Team Twifo Praso District District Education Office School/Community Stakeholders Head Teacher Nyame Bekyere Nyame Bekyere Primary School QUIPS Trained Teachers Nyame Bekyere Nyame Bekyere Primary School Interviewed with selected children Nyame Bekyere Nyame Bekyere Primary School Assemblyman Nyame Bekyere District Assembly Co-ordinating Director Nyame Bekyere District Assembly Chief and Elders Nyame Bekyere Traditional Council Queenmother Nyame Bekyere Traditional Council Interview with Head Teacher Watereso Watreso Primary School Interview with one Trained and one Untrained Teacher Watereso Watreso Primary School Chief and Elders Watereso Traditional Council Assemblyman Watereso District Assembly PTA and SMC Watereso Community member Unit Committee Member Watereso District Assembly Dunkwa On Offin District Stakeholders District Coordinating Director Dunkwa On Offin District Education Office A/D Supervision Dunkwa On Offin District Education Office A/D Human Resource Management and Development Dunkwa On Offin District Education Office A/D Statistics Dunkwa On Offin District Education Office Girl Child Coordinator Dunkwa On Offin District Education Office District Management Implementation Team Dunkwa On Offin District Education Office District Teacher Support Team Dunkwa On Offin District Education Office School/Community Stakeholders 172 Head Teacher Dunkwa Dunkwa Presby Primary School QUIPS Trained Teachers Dunkwa Dunkwa Presby Primary School Interview with selected children Dunkwa Dunkwa Presby Primary School Assemblyman Dunkwa District Assembly Co-ordinating Director Dunkwa District Assembly Chief and Elders Dunkwa Traditional Council Queenmother Dunkwa Traditional Council Interview with Head Teacher Achiase Achiase Primary School Interview with one Trained and one Untrained Teacher Achiase Achiase Primary School Chief and Elders Achiase Traditional Council Assemblyman Achiase District Assembly PTA and SMC Achiase Community member Unit Committee Member Achiase District Assembly 173 ANNEX 2: FIELD WORK PLAN, APRIL 2005 Team 1 1. Sunday April 3 Accra to Tamale 2. Monday April 4 Tamale CRS and GES 3. Tuesday April 5 Tolon-Kumbungu School No. 1 4. Wednesday April 6 Tolon-Kumbungu School No. 1 5. Thursday April 7 Tolon-Kumbungu School No. 2, TTC 6. Friday April 8 Tolon-Kumbungu School No. 2 7. Saturday April 9 Tamale-write-up 8. Sunday April 10 Tamale-write-up, move to Gambaga, East Mamprusi 9. Monday April 11 Bunkpurugu District Education Office 10. Tuesday April 12 Bunkpurugu – East Mamprusi school No. 1 11. Wednesday April 13 Bunkpurugu – East Mamprusi school No. 1 12 Thursday April 14 Bunkpurugu – East Mamprusi school No. 2 13. Friday April 15 Bunkpurugu – East Mamprusi school No. 2 14. Saturday April 16 Bunkpurugu – write-up 15. Sunday April 17 Bunkpurugu – move to Tamale 16. Monday April 18 Tamale-sorting, data collection, CRS, NGO, TTC 17. Tuesday April 19 return to Accra Team 2 1 Sunday April 3 Accra to Tamale 2. Monday April 4 Salaga –East Gonja District GES 3. Tuesday April 5 Salaga School No. 1 East Gonja 4. Wednesday April 6 Salaga School No. 1 East Gonja 5 Thursday April 7 Salaga School No. 2 East Gonja 6. Friday April 8 Salaga School No. 2 East Gonja 7. Saturday April 9 Salaga 8. Sunday April 10 Salaga 9. Monday April 11 Zabzugu – District Education Office 10. Tuesday April 12 Zabzugu-Zabzugu-Tatale school No. 1 11. Wednesday April 13 Zabzugu-Zabzugu-Tatale school No. 1 12. Thursday April 14 Zabzugu-Zabzugu-Tatale school No. 2 13. Friday April 15 Zabzugu-Zabzugu-Tatale school No. 2 move to Tamale in evening 14. Saturday April 16 Tamale-write up 15. Sunday April 17 Tamale-write up 16. Monday April 18 Tamale sorting, write up, TTC, NGO’s, CRS 17. Tuesday April 19 Return to Accra Team 3 1 Sunday April 3 2. Monday April 4 Accra to Kumasi 3. Tuesday April 5 Kumasi to Mampongteng DEO 4. Wednesday April 6 Ntribuoho school 1 174 5. Thursday April 7 Ntribuoho school 1 6. Friday April 8 Kumasi-write-up 7. Saturday April 9 Kumasi-write-up 8. Sunday April 10 Kumasi-write-up 9. Monday April 11 Ntonso school 2 10. Tuesday April 12 Ntonso school 2 11. Wednesday April 13 Mankranso DEO 12 Thursday April 14 Adukrom school 3 13. Friday April 15 Adukrom school 3 14. Saturday April 16 Kumasi write-up 15. Sunday April 17 Kumasi write-up 16. Monday April 18 Kunsu Dotiem school 4 17. Tuesday April 19 Kunsu Dotiem school 4 18. Wednesday April 20 Meeting with I.K. Gyasi (Kumasi) 19. Thursday April 21 Kumasi write-up 20. Friday April 22 Kumasi write-up 21. Saturday April 23 Kumasi to Accra Team 4 1 Wednesday April 6 Twifo Praso - District Education Office 2. Thursday April 7 Nyame Bekyere – School interview 3. Friday April 8 Nyame Bekyere – Community 4. Saturday April 9 Twifo Praso -write-up 5 Sunday April 10 Twifo Praso-write-up 6. Monday April 11 Watreso - School interview 7. Tuesday April 12 Watreso - Community interview 8. Wednesday April 13 Dunkwa On-Offin – District 9. Thursday April 14 Dunkwa District Office interview 10. Friday April 15 Dunkwa Presby – school interview 11. Saturday April 16 Dunkwa Presby-write-up 12. Sunday April 17 Dunkwa Presby -write-up 13. Monday April 18 Dunkwa Presby – Community interview 14. Tuesday April 19 Achiase –School interview 15. Wednesday April 20 Achiase - Community interview 16. Thursday April 21 Dunkwa -write-up and back to Accra on Friday 17. Tuesday April 19 Return to Accra 175 ANNEX 3: LIST OF QUIPS AND CONTROL SCHOOLS IN FIELD WORK SCHOOL GROUP TYPE COHORT REGION DISTRICT AVE ENG AVE MATH AGG ENG AGG MATH Ntiribuoho L/A HIGH QUIPS 4 Ashanti Kwabre 14.59 33.06 20.62 31.70 Ntonso SDA HIGH CONTROL 4 Ashanti Kwabre 18.84 26.12 15.89 21.43 Adukrom L/A LOW QUIPS 6 Ashanti Ahafo Ano South 7.27 20.89 6.28 16.94 Kunsu Datiem L/A LOW CONTROL 6 Ashanti Ahafo Ano South 10.10 19.83 6.28 16.94 Wamfie R/C LOW CONTROL 4 Brong Ahafo Dormaa 10.08 13.03 13.35 22.84 Lonto Presby HIGH QUIPS 4 Northern East Gonja 21.22 42.40 20.62 31.70 Ekumpe L/A MEDIUM CONTROL 4 Northern East Gonja 11.21 35.36 12.36 27.40 Nakpali Kworle LOW QUIPS 4 Northern Zabzugu Tatale 5.90 13.09 6.28 16.94 Tusundo L/A LOW CONTROL 4 Northern Zabzugu Tatale 1.95 7.92 6.28 16.94 Diare L/A LOW QUIPS 5 Northern Savelugu Nanton 16.74 30.93 13.35 22.84 Kpanlore E/A HIGH QUIPS 4 Northern East Mamprusi 18.36 32.60 20.62 31.70 Gbangu L/A HIGH CONTROL 4 Northern East Mamprusi 24.05 34.00 20.62 31.70 Nawuni R/C LOW QUIPS 6 Northern Tolon/Kum bungu 2.88 35.50 6.28 16.94 Kpalgun Zion LOW CONTROL 6 Northern Tolon/Kum bungu 2.35 12.22 6.28 16.94 Tuobodom Nuriya Islamic LOW QUIPS 5 Brong Ahafo Techiman 7.37 28.39 13.35 22.84 Watreso D/C HIGH QUIPS 4 Central Twifo - Herman 19.42 39.61 20.62 31.7 Nyamebek yere D/C MEDIUM CONTROL 4 Central Twifo - Herman 5.81 30.33 12.36 27.4 Dunkwa Presby HIGH QUIPS 6 Central Upper Denkyira 50.67 32.38 33.05 37.53 Achise D/C LOW CONTROL 6 Central Upper Denkyira 7.5 10.83 6.28 16.94 Kwaku Pamfo LOW QUIPS 5 Greater Accra GA 15.79 15.82 13.35 22.84 176 ANNEX 4: IMPROVING LEARNING THROUGH PARTNERSHIPS (ILP) TWO-YEAR INTERVENTION CYCLE IN THE SCHOOLS Key Activity Description Timing Stakeholder/Beneficiaries School selection Joint selection (GES/QUIPS) to identify schools that meet agreed criteria Before start of 2-years cycle – May-June each year School/community and pupils Launch workshop One-day meeting at district with 3 schools/communities and district stakeholders Start of 2 year cycle – July Community leaders, project school head teachers, circuit supervisors, District Education Office staff, DEOC, District Assembly representatives Baseline data collection Math and English achievement data collected in project schools Beginning of term 1 in first year of cycle Pupils in partnership schools Residential Training of Trainers (TOT) and instructional leadership and management TOT – to prepare district based education staff to be able to deliver INSET Before beginning of school year – August DTST members, head teachers of project schools, Circuit Supervisors, and selected District Education Office staff School-based INSET QUIPS/ILP trainer teams, together with some TOT graduates, conduct series of INSETs. After 3 INSETs, DTST/district team completes remaining 3 INSETs One per term (3 per year) for two years Head teachers, teachers, Circuit Supervisors Residential workshop on effective lesson planning & presentation (9 day) Develop skills of teachers in lesson note preparation, teaching methods, preparation of TLMs, etc December or April school holiday period Teachers, head teachers, Circuit Supervisors Residential management training TOT workshop Develop a core of district￾based management trainers (occurs with residential TOT for DTST staff – see above) August – after launch workshop District Education Office staff and others that may be identified by district education office from other agencies (e.g. Audit Service, District Assembly) First district management workshop Introduce stakeholders to effective management practices, including a review of SPIP and infrastructure process (4 days) First term of 2-year cycle (October-December) District Education Office staff, head teachers, Circuit Supervisors Final district management workshop Plan for continuing and spreading QUIPS best practices (1 day) End of two-year cycle District Education Staff, District Assembly staff/representatives, project school head teachers and circuit supervisors, DTST members Infrastructure Project Help strengthen physical environment of schools After first management workshop on receipt of community’s application Community, school staff, pupils Annual M&E data collection Achievement tests, COI, SCMI, SPIP, school statistics End of school year, years and 2 of cycle Pupils, teachers, parents 177 ANNEX 5: COMMUNITY SCHOOL ALLIANCES’ TWO-YEAR INTERVENTION CYCLE IN COMMUNITIES Key Activity Description Timing Stakeholder/Beneficiary School selection Joint selection (GES/QUIPS) to identify schools that meet agreed criteria Before start of 2- year cycle- May June each year School/community and pupils Launch workshop One-day meeting at district with 3 schools/communities and district stakeholders Start of 2-year cycle- July School leaders, project school head teachers, circuit supervisors, District Education Office staff, DEOC, District Assembly representatives Selection of district facilitators Up to one week training for national service personnel on community animation techniques . August National Service personnel from the University Residential PLA/PRA workshop One week training on PRA /PLA as a research tool to generate school discussions in partnership school communities September or October of each year National service personnel, selected District Monitoring Assistants (DMA), District training officers from partnership district Baseline data collection First start-up activity of CSA in partnership community. Data are collected on the current performance of the community on the 15 ‘best practice’ sub-objective in each partnership school. October to December of each year. The data is collected as part of the PLA/PRA activity Unit Committee members, Assembly persons, SMC/PTA executives, community members, teachers, pupils and chiefs and elders Participatory Rural Appraisal/Participatory Learning and Action (PRA/PLA) 4-6 days series of meetings held with community-based educational stakeholders to discuss school-related issues and also create awareness and mobilize resources for primary school October to December of each year Unit Committee members, Assembly persons, SMC/PTA executives, community members, teachers, pupils and chief and elders School Performance Improvement Plan (SPIP) Discussions and issues raised during PRA/PLA are synthesised into community and school improvement (action) plans (SPIP) on the last day of the PRA/PLA activity October to December Unit committee members, Assembly persons, SMC/PTA executives, Community members, teachers, pupils and chief and elders Information, Education and Communication (IEC) IEC campaign aimed at stepping up support and participation in the achievement of the ‘best practices’ sub￾objectives through the following strategies:1)Community performed drama and open forum 2) Story picture cards, calendars and newsletter 3 )Radio discussions February to May Unit committee members, Assembly persons, SMC/PTA executives, community members, teachers, pupils and chief and elders Residential SMC/PTA TOT workshop 5-day training to prepare CSA facilitators and selected district officers to be able to deliver SMC/PTA training March CSA facilitators, selected district officers 178 SMC/PTA training Institutional capacity-building provided to SMC/PTA executives to equip them with needed school planning and management skills Delivered in a 5- day, 3-day, 2-day format (10days) over the 2-year project cycle SMC/PTA executives, circuit supervisors for partnership circuits Micro-grant projects An amount of money which is provided to all partnership school communities to support community-initiated projects related to PLA/PRA and SPIP, which link the community to the school for improving children’s education Distributed in three tranches over the 2- year project cycle Unit committee members, Assembly persons, SMC/PTA executives, Community members, and chief and elders 179 Annex 6:School-Community Categorized by Teacher Performance and Community Participation (n = 18 ) COMMUNITY PARTICIPATION ( - ) ( + ) Kpanlori E/A Hi-Q Lonto Presby Hi-Q Achiase D/C Med-c Gbangu Bongbini Hi-c Ntonso SDA Hi-c Ekumpe L/A Med-c (+) Adukrom L/A Lo-Q Watereso D/C Hi-Q Dunkwa Presby Hi-Q TEACHER PERFOR- ________________________________________________ MANCE Nyamebekyere Hi-c Ntiribuoho Hi-Q Nakpali Kworle LO-Q Nawuni R/C Lo-Q ( - ) Kunsu Datiem Lo-c Tusundo L/A Lo-c Kpalgun Zion Lo-c Tuobodum Nuriya Lo-Q Islamic-urban urban school-communities in italics Q = QUIPS, c = control, Med = medium, Hi = High, Lo = Low teacher performance = classroom observations (use of lesson plans, use of TLMs, marking exercises, soliciting questions, appreciative inquiry) community participation = school visits by parents, basic needs of pupils provided by parents, supervision of pupils at home, meetings by PTA-SMC, parents reducing domestic chores, etc. 180 ANNEX 7: DESCRIPTION OF SCHOOL FACILITIES IN SAMPLED QUIPS AND NON-QUIPS SCHOOLS School District Type School Building Place of convenience – Urinal Before QUIPS After QUIPS Before QUIPS After QUIPS Lonto Presby East Gonja Hi QUIPS Uncladded pavilion in a small compound 7 classroom in one newly refurbished & older block Nil 4 seater KVIP & 2 landcrete urinals Ekundi L/A East Gonja Control Old leaky uncladded pavilion 6 unit classroom block with support from ADB Nil 6 seater KVIP, No Urinals Nakpale Kworle Zabzugu Tatalle QUIPS Old 7-unit classroom block with roof partly ripped off. Modern 6-unit classroom block with support from QUIPS Nil Nil Tasundo L/A Zabzugu Tatalle Control 3-unit classroom block for P1- P6 two set of 3- unit classroom block with support from ADB Nil 4-seater KVIP & 2 urinals Kpanlori E/A East Mamprusi QUIPS 4-unit mud brick classroom roofed with thatch 6-unit classroom block with support from QUIPS-CRS Nil Nil Gbangu L/A East Mamprusi Control 3-unit classroom block for KG1- P6 Same as before Impro￾vised Same Nawuni R/C East Mamprusi QUIPS 3-unit classroom block built by Catholic for P1-P6 Same, QUIPS classroom project not supported by community still at roofing stage Nil Nil Kpalgun Zion East Mamprusi Control Pavilion partitioned into 3-unit classroom for P1-P6 New 6-unit classroom block by District assembly yet to be completed Nil Nil Tuobodom Nuriya Islamic Techiman QUIPS Caddied Pavilion 6-unit imposing classroom block with QUIPS support Nil Nil 181 Ntribuoho L/A Kwabre QUIPS 6-unit dilapidated mud brick classroom 6-unit classroom, three in use other three incomplete intention is storey with QUIPS support Impro￾vised 3 urinal & 3-unit KVIP construc￾ted by commu￾nity Ntonso SDA Kwabre Control 6-unit leaky roofed with broken doors & windows Same Shared with cluster of schools same Adukrom L/A Ahafo Ano South QUIPS 6-unit dilapidated mud brick classroom Modern 6-unit classroom block with support from QUIPS Impro￾vised same, new one under construc￾tion Kunso Dotiem L/A Ahafo Ano South Control Dilapidated 3- unit classroom block Same & 3-unit cladded pavilion released by JSS Impro￾vised shared with JSS Watreso D/C Twifo￾Heman QUIPS Old 3-unit classroom block Modern 3-unit classroom block & old block Nil Nil Nyamebe￾kyere D/C Heman Twifo￾Heman Control Old 3-unit classroom block for P1- JSS3 Same Nil Nil Dunkwa Presby Upper Denkyira QUIPS 6-unit leaky roofed with broken doors & windows 4-unit classroom block being extended to a 2-storey Impro￾vised New sets of urinals & KVIP Achiase R/C Upper Denkyira Control 6-unit leaky roofed with broken doors & windows Renovated old building painted nicely Impro￾vised Impro￾vised Lonto Presby East Gonja QUIPS Uncladded pavilion in a small compound 7-classroom in one newly refurbished & older block Nil 4 seater KVIP & 2 landcrete urinals Ekundi L/A East Gonja Control Old leaky uncladded pavilion 6-unit classroom block with support from ADB Nil 6 seater KVIP, No urinals Nakpale Kworle Zabzugu Tatalle QUIPS Old 7-unit classroom block with roof partly ripped off. Modern 6-unit classroom block with support from QUIPS Nil Nil 182 ANNEX 8: TEACHER PROFILE, SAMPLED SCHOOLS PRIMARY J.S.S TRAINE D TEACHE RS TRAIN ED TEAC HERS No. SCHOOL NAME ENROLLMENT No. OF CLASSROOMS No. OF CLASSES No. OF TEACHERS Q UQ PUPIL TEACHER R.E.V COMMUNITY VOLUNTEER Q UQ PUPIL TEACHERS R.E.V COMM U NITY VOLUNTEER 1. NAWUNI R/C (Q) 68 3 6 3 2 1 - - - 2. KPALGUN ZION (C) 300 3 6 5 0 1 4 - - 3. KPANLORI E/A (Q) 237 6 6 7 2 1 1 - 3 SHARED TEACHERS 4. GBANGU L/A ( C) 281 3 6 4 0 1 2 - 1 5. LONTO PRESBY ( Q) 300 7 6 6 1 1 1 - 2 6. EKUMPE L/A ( C ) 338 6 10 10 0 1 5 - 4 7. NAKPALI KWORLE (Q 338 6 6 5 1 0 4 - 0 8. TUSUNDO L/A ( C) 149 6 4 3 0 1 2 - 0 9. NTIRIBUOHO L/A (Q ) 6 6 6 2 4 10 NTONSO L/A ( C) 6 6 6 6 11 ADUKROM L/A ( Q) 6 6 6 5 1 12. KUNSU DOTIEM ( C) 6 6 4 2 2 13. WATRESO D/C 6 3 14. NYAMEBEKYERE (Q) 3 9 6 - 3 3 - - SHARED TEACHERS 15. DUNKWA PRESBY (Q) 6 3 16. ACHIASE D/C ( C) 4 1 3 17. TUOBODOM NURI ( Q) 138 6 6 6 3 2 1 - - 18. KWAKU PAMFO ( Q) 6 6 6 - 6 - - - 183 ANNEX 9: INDEPENDENT REVIEW OF THE QUIPS ACHIEVEMENT TESTS – by Professor Wes Snyder In 1998-1999, The Mitchell Group developed a set of six English and Mathematics achievement tests designed to track Ghanaian pupils across four academic years. Three of the tests spanned Grades 3 and 4 for one group, and another three spanned Grades 5 and 6 for a cohort. The battery of tests consisted of Mathematics (at Grades 3-4 and Grades 5-6), English Language (at Grade 3-4 and Grades 5-6), and English performance tests (also, at Grades 3-4 and Grades 5-6). These tests were intended to enable evaluation of the USAID-funded Quality Improvement in Primary Schools Program (QUIPS), which comprised school improvement activities under the Improving Learning through Partnerships (ILP) and community involvement activities under the Community School Alliances (CSA).119 The sum of interventions was intended to enrich the schools and result in improved school performance. The tests, therefore had to honor the curriculum as prescribed and taught, as well as the principles of literacy development and Mathematics foundations. A.9.1 PRACTICE TO ADJUST FOR EXPERIENCE AND ADMINISTRATIVE COMPLICATIONS In the early 1970’s, Paul Schwarz and Bob Krug established the need for practice in modern tests for pupils facing these as novel experiences. The International Developed (ID) Tests were based upon the well-known Flanagan Aptitude Classification Tests (FACT), as well as aspects of the widely used Differential Aptitude Tests (DAT), but the adaptation required adjustments to the administration procedures as well as the item content to be relevant for West Africa and other contexts. Among pupils targeted for the tests, there was little exposure or experience with paper or written information, unless it was on a blackboard, and almost no experience with multiple-choice formatted questions and structured tests. The ID work demonstrated that more accurate information for selection decisions would be available if extensive practice were provided to even out administrative understanding and familiarize the pupils to modern testing procedures. Development of the QUIPS tests takes this advice to heart, even though the purpose is evaluative rather than selection. All the QUIPS tests have extensive practice sessions that tell pupils what to expect in approach, what item forms they will see, and how they are to respond to indicate what they know. The intent is to minimize the extraneous variance due to prior experience and individual quickness on situational adjustment. Also, this minimizes language problems in the Mathematics tests, or at least in their foundational elements for basic concepts and operations. 119 These programs correspond to four areas of intervention and results: improved teaching and supervision, improved educational management, increased community involvement, and improved learning environments in the schools. In total, they reflect “increased effectiveness of the primary education system.” 184 In addition to the practice tests, further direction and guiding examples are provided before each section of the QUIPS tests. All of this improves the validity of the measures, but has its costs in efficiency and administrative involvement. Within the context of the Ghanaian education system, this was a reasonable trade-off. The Ghanaian pupil has almost no experience with any testing in the early grades, and this entire experience is likely to be a grand event that must be controlled and elucidated so that the intent is not lost and pupils are able to perform their best in the assessment tasks, despite the unusual nature of the event. A.9.2 ADMINISTRATIVE DETAILS The QUIPS tests require trained test administrators. Two administrators are needed for each level test, Grades 3-4 and Grades 5-6, both trained over four days plus a refresher course just prior to the test administration. One member of the team must be fluent in the local language, especially to assure accurate and meaningful communication in the practice sessions and the Mathematics tests. Two teams of two administrators each traveled to each school for two days, administering the tests, and then spent another day reconciling the answers for transfer to scan forms for automatic scoring and analyses. Upon arrival at a school, the teams brief the head teacher of the schedule for each test day: introduction to the tests, practice sessions, snack and break to offset boredom and tiredness, classroom testing, another break, performance tests, and a debriefing at the end of the testing on day 2. Teachers of the classrooms to be tested are also briefed to secure their cooperation as well. The briefings included the purposes of the testing, the need for security, the reason for teachers to stay away during the testing, and the need for the head teacher and the teacher to keep other children away from the testing room and quiet in the vicinity. In addition to the practice sessions, test administrators were advised to use praise generously, make sure pupils are involved in the practice sessions, spot check initial work for understanding, provide special attention where needed, use local language translations generously, and teach processes but not content. Although considerable flexibility was permitted in the practice sessions, so that the administrators could be assured of pupil understanding, the test situations were strictly standardized. Administrators were particularly advised to maintain pupil attention to the tasks, keep pupils in a positive frame of mind, and make sure every pupil’s answers were secure. Because of the dependence on common instructions and translations, every pupil had the same form of the test and control over copying was necessary. Again, this is a decision to improve validity because the emphasis on practice and clear instructions assures that every pupil more likely understands all facets of the testing process and performs as well as they can. The risk is security and thus the emphasis on providing details in the administrator instructions. 185 Every effort was made to guarantee the standardization of the testing experience in each school and grade level. Additionally, every attempt was made to assure the motivation of pupils, even though these tests had no stakes for them and little for the schools in ways that were apparent to them. In low stakes testing, there is a need to elicit cooperation from the test-takers so that they are sufficiently motivated to try their best. The novelty of the testing probably helped make the event seem special and important, and the extra attention to detail in the administration of the tests helped to foster the pupils’ cooperation and continued engagement through the testing experience. The influence of copying is unknown. Pupils at these grade levels and without experience tend to look to others for confirmation of what they are doing, even if there’s no direct intent to “cheat.” Accordingly, it might have been useful to have multiple forms in each classroom and standardize instructions but not item particulars. This would have altered the administration dictation of the items and required the pupils to read their own items. In the case of mathematics, this might have increased the language load in the results. Experience in the Ghanaian Criterion Referenced Tests program, administered over a decade across the country at Grade 6, pupils do copy as much as they can. In that case, ten forms were prepared so that any copying led to random-in-effect performances. Although there is no evidence one-way or the other, this is an area that could have affected any results. A.9.3 STRUCTURE OF TESTS The QUIPS English tests contained items for matching words to pictures, letter and word recognition, word recognition from sentences read by the examiner, reading sentences themselves with understanding, and reading stories and interpreting meaning. Additional performance tests were given to a sample of pupils (recommended to be four females and four males from each class) in small groups and individual arrangements to supplement the more structured multiple-choice responses from the class tests, which had constructed responses and more complicated language situations. The mathematics tests contained items for basic number concepts, basic operations, and story problems. In both the English and mathematics test administrations, all reading was done in English as well as in one of the appropriate eight local languages (except in those cases where English was tested per se). The responses in the Mathematics tests were entirely constructed responses, although single answer. Although the tests were aimed at specific combinations of grades (3-4 or 5-6), the item content covered wide ranges of the curriculum from Grade 1 to Grade 5 in mathematics and Grade 1 to Grade 6 in English. Historically, performances in achievement tests in Ghana have been higher in English language than in Mathematics. Because these tests were designed to reflect changes in performance, the emphasis was on prerequisite skills and continuous knowledge domains that grow with 186 experience. Therefore, the ranges of the tests are purposively selected to be relevant to the Ghanaian context and sensitive to those knowledge and skill areas that might reflect improvements in the schools attributable to school and/or community interventions. The distribution of items for the practice tests, where Grade 3-4 and Grade 5-6 were identically structured, is given in Table 1. Although there are few items in each category, the emphasis is on the testing process and each item permits careful review of how the pupils are doing in the test taking. More practice was provided in English language items, rationalized mostly likely because the item types are more complicated in process. Table A16.1: Item Type by Subject in the Practice Tests Subject Basic Number Concepts Basic Operations Story Problems Mathematics 2 2 2 Matching Words to Pictures Letter and Word Recognition Word Recognition from Sentences Reading Sentences Reading Stories English 2 3 3 3 2 Stories (8 Items) The Mathematics classroom tests consisted of 40 items, and the English tests consisted of 56 items (58 in the Grades 3-4 test). Of course, in the reading stories there are really only 4 independent items. This is always a problem with reading passages. All the items for each passage are dependent upon the understanding of the same passage. If the pupil doesn’t understand the passage, then all the items will be affected. Therefore, there are really only 29 independent items in the English tests. Table A16.2: Item Type by Subject in the Classroom Tests Subject Basic Number Concepts Basic Operations Story Problems Mathematics 7 26 7 Matching Words to Pictures Letter and Word Recognition Word Recognition from Sentences Reading Sentences Reading Stories English 3 4 10 8 4 Stories (31 Items) 187 The English performance tests are administered in two sessions, small group and individual. There were five activities in the group work: writing English letters, writing as many words as they can in 10 minutes, taking turns in the group to tell familiar stories, telling a story (Kafui and his Bike for G3 and Hope Finds a Home for G5) and having the pupils retell the story, and writing words to dictation taken from the book stories for each grade levels. There were four sections to the individual assessments: writing English letters, writing words, writing to dictation, and story telling. The individual sessions were tape recorded in case there were scoring issues and to provide data, possibly, for later research. The group sessions provide some practice in the tasks, and the individual sessions are more structured in the interactions between the administrator and the pupils. Scores are counted for correct responses in word and dictation sessions, and rubrics for story responses, with more credit applied to pupil initiated responses, accuracy and exceptionality. The full rationale for the structure of the tests is unknown. Total scores will reflect those areas with greater variance, which may or may not correspond to the weights given by item assignment. In the Grade 3-4 Mathematics tests, 55% of the items came from the curriculum targeted for grades Grade 1 and Grade 2; and in the Grades 5-6 mathematics tests, 80% of the items came from Grades 1-4 curriculum, and no items came from Grade 6. In the Grades 3-4 English tests, about 32% of the items came from Grade 1 and Grade 2, but 7% come from Grade 5; and in the Grade 4/5 English tests, about 65% come from Grades 1-4 curriculum. Only in the English Grades 3-4 test do most of the items come from the intended grade levels. Most of the items in that case are for reading stories with understanding. Normally, the distribution of items should reflect the level of the test. In the Ghanaian case, because of consistently lower performances in the CRT program in both subjects, there is the belief that for rural schools (or more rural than others and those selected in this program) there is little chance for the pupils to be performing in general at grade level. That would account for some of the emphases on items selected from those “achieved” at the lower grades. Why the structures are so different from test to test is unknown, but at least this information was available and therefore, known to the test developers. Grades 3 and Grade 5 pupils were tested on three occasions beginning at the opening of G3 (or G5), the end of G3 (or G5), and the end of G4 (or G6). None of these tests systematically samples the curriculum, and as reported, their distribution of items across grade levels is also unsystematic. Accordingly, each test has to be interpreted very carefully, in terms of its particular content selection and not in general terms, such as English literacy. These are attempts at developing useful curriculum referenced tests that will reflect changes in areas of intervention. 188 To ascertain their validity for this evaluative purpose, information would be needed on the objectives sampled matched to intervention intentions from the various programs. This is not easily done during development because programs change and opportunities are exploited to enhance program effects. The problem is that these are not general tests, but curriculum tests. Therefore, they will only reflect changes relative to the particular items included. If the test structure does not reflect the program activities or intents, then the test is not valid for the evaluation of the interventions in that program. The assumption is, if the overall QUIPS program was influential, its effects would be generalized across the curriculum and any item reflecting continuous skills would be a valid indicator of program effectiveness. In the analyses, total scores are used, and these are not based upon unitary constructs (or may not be). A great deal may be hidden in any evaluative study using these tests in terms of total scores, where the total score was not established as a meaningful index of underlying construct(s). Caution is advised, and research should explore various structural analyses that might establish scores more sensitive to between-group differences (differences between QUIPS schools and others over time). CRT tests are not intended to be predictive in the same way as aptitude tests are. There has been concern expressed in Ghana about the relationships of tests and the information they convey about minimum competence. Scores in the Performance Monitoring Tests (PMT) given to schools to ascertain triage conditions for Inspectorate intervention do not match with more comprehensive Criterion Reference Tests (CRT) designed to provide a national report card on all aspects of the curriculum that can be assessed in multiple-choice format. The CRT agrees somewhat with the Basic Education Certificate Examination (BECE), but the PMT varies substantially from both. Matching the QUIPS tests with any of the other tests may not be that revealing. The QUIPS tests are structurally different and have a distinct evaluative purpose, which is not related to the other tests. Tests have to be considered in terms of their purpose. It would be a little like comparing a car built for general use with one built for performance, and there are many examples. Both might be fine cars in their category and for their purpose, but they are unlikely to be similar in any important aspects because their intents are very different. Tests cannot be easily compared unless they are doing the same thing. To some extent, the validity of the QUIPS tests is established by the sensitivity to program differences. Any match or mismatch with other tests would have to be interpreted in terms of the different purposes and test structures. A.9.4 TECHNICAL INFORMATION A key consideration to evaluate the assessments is the technical performances of the tests and their perceived effectiveness for the evaluative task. Alpha reliability indices 189 are available for draft versions of the tests, and their values range from 0.86 to 0.95 (where 1.00 is the higher direction). The English tests tend to be slightly higher in these values, and this is typical. These are very high reliabilities, and they help to support the use of the total score as generally indicative of performance across these constructs as operationalized. Of course, the construct structure is embedded within the alpha reliability so there is still no information available about possible underlying factors that might reflect changes over time more effectively than a summary index. The validity of these tests is argued on the basis of alignment with the curriculum for English literacy and basic Mathematics operations. Items were developed from the curriculum statements and content in textbooks used in the classrooms, when available. Content, presentation, and formatting were checked in field-testing to be sure pupils should know the material and be able to answer items in these formats. Three lines of argument are provided for the technical adequacy of the tests. All employ classical test indices, which are commonly used for checking item characteristics, even though they are not really designed for curriculum-based assessments. They are therefore not compelling evidence of adequacy but they are sensitive to many problems that a test may have. First, the pupils who perform high on the test as a whole are expected to score well on any item. The average discrimination index tells us if the set of items seems to internally differentiate between high scorers and low scorers. For the QUIPS test, the range of average discrimination indices is from 0.45 to 0.65, all high and indications of internal validity. Second, the difficulty of items on average should decrease from one grade to the next one. The ranges of average difficulty levels is 0.41 to 0.71, and in all cases, the average proportion of items correct from one grade to the next one increases by at least 0.11. In norm referenced tests it can be established that the maximum discrimination exists at 0.50 average difficulty level, so these indices mean that there is probably ample “floor” and “ceiling” to the tests so that changes either up or down could be reflected adequately. This speaks to the test structure again. The distribution of items selected from lower grades seems to be substantiated by these results. If items were selected only from the grade level, the difficulty indices would be much lower (which means more difficult). Thirdly, and lastly in the technical domain, average item (right or wrong) by total score Point-Biserial Correlations ranged from 0.42 to 0.55, all very high and indicative again of internal validity. On average the items are contributing to the total scores in ways that will effectively differentiate academic achievements as measured in these collections of items. These lines of argument indicate that the tests can indicate changes in achievement without obvious technical impediments. A.9.5 STRENGTHS AND WEAKNESSES The strengths of the QUIPS tests include: 190 • Extensive practice experiences that improve validity; • Attention to administrative details to assure standardization and pupil engagement; • Care of training that reduces extraneous variation of the testing context; • Curriculum alignment with both the written curriculum and the textbooks in use; • Use of local language in the instructional set to more effectively communicate to every pupil; and • Impressive classical internal technical indices that substantiate the care in test development and increase the likelihood of the usefulness of the tests for their purposes. The weaknesses are less clear because more detailed analyses would be required: • Total scores for curriculum-based tests are not always meaningful; • Interpretations of assessments used in evaluation must be made carefully to assure links between effect and attribution; • Rationale for item selection is general and not specific, where the specific selection of items is critical to the evaluative goals; and • Overall low scores on items predominantly from lower grade levels indicate that some items may reflect curricula never taught and present difficulties for assessing change in new grade levels that may not reach back to remediate gaps in foundational skills and knowledge. The tests were developed around classical procedures with the apparent aim of providing an adequate “range” of score possibilities to reflect changes across grade levels. Items were sampled from many grade levels, thus providing various levels of difficulty, but this does not necessarily assure correspondence to the evaluative goals. However, the sampling may be adequate. Differences between QUIPS schools and non-equivalent (but similar) Control Schools were detected by the tests, and this verifies the general assumption that intervention effects are sufficiently curriculum-wide that they are detected by these tests. The less-than-“expected” achievement differences, however, might reflect some inadequacies in the test. If the items were inadequately sampled from key areas of the effects of the interventions, then the assessments would not be sensitive to all of the evaluation needs. The problem with variation in test scores is that the cause is important. Some spread of scores may be due to different backgrounds and opportunities, rather than specific educational or schooling inputs. If test item selection is based solely on technical characteristics, then differences in socio-economic factors may loom within the relevant variance of the items. For example many pupils in Ghana are not taught the whole of the curriculum for their grade. Recent information from Kofi Mereku’s ‘Opportunity to Learn Study’ indicates that many pupils are taught about half the prescribed curriculum in Mathematics and English, and 191 most pupils fall far short of exposure to the entire official curriculum. Therefore, when pupils are not taught the material directly, their performance on test items is due to external experiences; frequently the opportunity to learn at home, from parents, books, siblings, or the community. This links item score variance directly to socio-economic factors and not to schooling influences. If QUIPS interventions are making a difference, that difference may be only in those areas actually covered by the involved teachers. The actual overlap of coverage of the curriculum and item distribution in the QUIPS tests could be small. This would make the observed differences favoring QUIPS even more significant than they would at first appear. Schooling differences are very difficult to detect, and changes due to innovations and interventions do not move distributions of scores much over a brief period of time. When tests are able to detect differences, they have succeeded, as long as the changes are consistent across indicators and their interpretation is meaningful. Given the technical credentials of the QUIPS tests, they appear to be valid for the evaluative purpose for which they were developed. But this very spread of scores is also problematic, indicating spread of accomplishment. One of the problems in educational assessment is that performance frequently lags behind practice. Assessment of performance, therefore, underestimates actual changes in practice. Coupled with the other positive influences of an intervention program beyond the immediate targeted schools, the purity of experimental design is rarely true for educational contexts. Therefore, even small gains or differences are greatly valued in reform evaluations. The key to test interpretation for reform is to understand the meaning of the items in the tests and their connections with intervention efforts. In evaluation terms, these are referred to as program rationales, or reform hypotheses, and they comprise logical statements that link probable effects attributable to specific or combinatorial interventions. This requires a great deal of information about the reform, and a theory of reform would be helpful, but is rarely available. But the process of developing such a rationale illuminates likely interpretive problems as well as meanings associated with the possible results. It is an essential process to interpret the testing results. Therefore, reform tests should probably: • Focus on those curricular or subject domains that comprise the specific targets for the reform (some additional items are also useful for the consideration of spread of effect and side effects). • Reform indices should not be solely single-shot, limited coverage assessments of the reform intents and there should be deeper investigation into the effects of schooling and community influences. 192 • Results in evaluation are not always conveyed in meaningful terms when total test scores and group statistics are the main communicative evidence, and greater attention should be paid to clear qualities of the assessment intent. • Because total test scores for curriculum-based tests are crude summaries of performance, reports on effects are better focused on the profile of achievement and its associated curricular intents. • Learning and test performance are not equivalent concepts, so in the evaluation every effort should be made to collect a program profile of accomplishments that crosschecks effects and attributions related to the evaluation questions. Tests are very useful when they are used in conjunction with a broader range of reform indices and reliance on their sensitivity and validity is not strained. Curriculum-based tests are more qualitative than is frequently acknowledged. Credibility is sought by the use of test scores when greater information is available in the connectivity of test. That is, the QUIPS tests were developed around curricula principles. However the Ghana primary school curriculum is not yet well articulated. It lacks horizontal and vertical integration, is not developmental by design, and it reflects popular and momentary decisions rather than concerted theoretical links that would give meaning to the instructional process within and over the 6 grade levels. Additionally, the Ghana curriculum as prescribed, is not necessarily taught as earlier indicated because teachers do not get to the prescribed material or because they rely heavily on textbooks and other instructional experiences. These textbooks are not well linked to the curriculum or are so mechanically linked that they are uninteresting and not likely to engage pupils in a deep learning process. The material is sometimes quickly presented, relies on lecture and rote memory, remains superficial, and is unrelated to the pupils’ lives or interests. The QUIPS tests meet most criteria for good indices for measuring educational reform. They do reflect some changes in performance that appear to be linked to the reform effort. Additional analyses would be useful, and greater attention to the full gamut of effects would provide a helpful context within which to interpret the necessarily brief and quick assessments provided by the QUIPS tests. Although many foreign experts were involved, including a linguist and some psychometricians, the tests were essentially locally developed under the guidance of the TMG team leader. The accomplishment of the testing effort within QUIPS is considerable and the time pressures to measure program effects precluded many other assessment possibilities. (These tests are not likely to be widely used because of their complicated administration procedures and their narrowed focus made necessary by the time and resource constraints.) All in all, these tests are remarkable and the principles (particularly the opportunities to practice testing processes for the pupils, attention to details of administration, provision 193 of language support, and careful training of administrators) that underlie their development should be more widely applied in other reform evaluations. 194 ANNEX 10: SCOPE OF WORK FOR THE QUIPS FINAL EVALUATION I. BACKGROUND USAID’s Strategic Objective 2 was designed to help Ghana’s education system progressively address district and school-level constraints. Strategic Objective 2 (more commonly referred to as the Quality Improvement in Primary Schools [QUIPS] project) was initiated to support the implementation of government’s Free and Compulsory Basic Education (fCUBE) program. SO2 began in 1996 and program activities ended on September 30, 2004. (Strategic Objective 2 will not formally end until June 30, 2005 in order to provide time to undertake this evaluation and related closeout activities). The overarching objective of SO2 was: “Increased effectiveness of the primary education system.” The strategic objective had the following Intermediate Results:120 IR 2.1: Improved environment for learning IR 2.2: More effective teaching IR 2.3: Greater community involvement IR 2.4: Primary education curriculum revised and instructional materials developed and tested along with assessment standards and procedures IR 2.5: Education personnel management system improved IR 2.6: Capacity and authority of districts and schools increased IR 2.7: School quality information and analysis improved. SO/QUIPS included interventions in community participation, classroom and school level activities, management of education at the district level, and infrastructure. The school program was implemented by two grantees. The Academy for Educational Development (AED) covered the South of Ghana (86 districts) while Catholic Relief Services (CRS) covered the Northern sector of the country (24 districts).121 CRS also implemented a school feeding program alongside the QUIPS program in the North. The school feeding program was aimed at promoting school enrollment and attendance especially for girls and it is believed that the feeding program has had some influence on pupils' achievement. support for primary education reform under 120 In practice these seven IR’s were consolidated into the following four IR’s: a) IR1, Improved teaching and supervision; b) IR2, Improved education management; c) Increased community involvement; and d) IR4, Improved learning environment. 121 Ghana had a total of 110 districts during the period of QUIPS implementation. Recently, the number of districts has been increased to 138. 195 A number of other partners were also involved in the implementation of the QUIPS program. Educational Development Center (EDC), with CARE as a sub-contractor, implemented the community-based activities in all districts in the Southern regions. EDC also implemented the district grant program beginning in 2002, which included a district training and support program in monitoring and evaluation capacity building. The Mitchell Group (TMG), with subcontractor Harvard, provided monitoring and evaluation oversight to QUIPS, including support in the development and implementation of achievement tests and a teacher assessment instrument. TMG also provided initial M&E training to the Planning Budgeting Monitoring and Evaluation Unit (PBME) and to District Monitoring Assistants (DMA’s) at the Ministry of Education. The achievement tests were conducted through a contract with the local NGO, Education Assessment and Research Center (EARC). In 2001 a mid-term assessment was conducted to inform USAID about the design and conduct of QUIPS activities for the remainder of the strategy period and to lay the basis for an extension of QUIPS activities through 2004. The assessment reviewed project activities in three broad areas: effectiveness of program at the community, school and district levels; sustainability systems put in place to ensure the mainstreaming of best practices; and replication of QUIPS approaches in a cost effective way, on a national scale. Two new components were introduced to the QUIPS program: the district grants mechanism and the monitoring and evaluation component. The program was scheduled to end in September 2004. After the mid-term evaluation, the QUIPS program oc e on five areas: (1) school and district management; (2) community participation; (3) district grant mechanism; (4) monitoring and evaluation; and (5) classroom instruction. Pupils in QUIPS schools were taken through achievement tests annually. The achievement tests assessed skills related to Mathematics, English literacy, and spoken English. Actual results for Mathematics and English literacy were given by compiled percentage correct scores obtained for pupils tested repeatedly from year to year. The repeated measures analysis provided an opportunity to compare the learning growth patterns of pupils participating in the QUIPS program and a set of matched Control Schools. Actual results for spoken English were given by the percentage of pupil responses in a story telling task. The test was administered in QUIPS and Control Schools. Schools receiving assistance from the district grant mechanism did not take the QUIPS achievement test. These interventions have led to pupil achievement and increased rates of learning growth in both Math and spoken English. They have also helped to significantly reduce dropout rates and increase community participation in QUIPS schools. Nonetheless, overall increases in pupil achievement have not been as impressive as expected and in one area, English literacy, the results have actually been disappointing, with the difference in English reading achievement scores between QUIPS and Control Schools narrowing over time. In 2002, the mean score spread between QUIPS and f us d 196 Control Schools in English reading was 14%. In 2003 it narrowed to 6% and in 2004 it further narrowed to 2%. The reasons are not necessarily attributable to weaknesses in the QUIPS program. The narrowing of the gap may be due to a variety of factors. For example, the QUIPS program has been developing the capacity of GES District Training Teams to support school improvement. These teams are supporting non-QUIPS schools, including the Control Schools. The results may also be due to: 1) teachers not properly utilizing the QUIPS approaches (they are more challenging to use than the older approaches); 2) QUIPS-trained teachers transferring out of QUIPS Schools; and 3) limited time on task due to teacher absenteeism. Whatever the cause or causes, it is important for USAID, its partners and the broader community involved in primary education in Ghana to develop a better understanding of the factors responsible for these results. II. PURPOSE The purpose of this evaluation is to assess the impact of the QUIPS program on pupil achievement and learning. Shifts in composite scores on pupil achievement as a direct consequence of education reforms are often difficult to obtain. However, much can be learned from impact assessments that go beyond analyses of composite outcomes by looking more closely at the range of factors that may influence positive and negative impacts. Specifically, the evaluation should investigate the range of factors that help explain: 1) differences between high performing and low performing schools, districts and regions; 2) the relatively modest learning achievement gains by QUIPS pupils; and 3) narrowing of the gap between the QUIPS partnership and Control Schools. The results of this evaluation should identify factors such as teaching methods and/or community practices that are associated with improved learning and identify challenges that need to be addressed in programs targeting improved pupil achievement. The results of this evaluation will inform USAID, the Ghana Education Service (GES), its implementing partners, the QUIPS recipients and the broader community concerned about primary education in Ghana and about what worked and what did not work under the QUIPS program. III. SERVICES REQUIRED The contractor shall provide professional and thorough evaluation services sufficient to achieve the purpose set forth above. Specifically, the evaluation should address the following questions: • What is the relationship between QUIPS interventions and pupil learning? • What factors, both within and outside of the QUIPS interventions, are associated with high and low achievement growth? That is, what are the common factors observed in partnership schools and in Control Schools that are associated with 197 high achievement gains and what factors appear to constrain learning even in the presence of school-based interventions such as QUIPS? • Why are some Control Schools appearing to reach levels of achievement comparable to QUIPS schools? • What factors are associated with fluctuations in pupil achievement outcome among the QUIPS schools? • Why did the QUIPS program appear to have a greater impact on Math and spoken English achievement and less on English literacy? • Has the increase in the achievement scores at QUIPS schools been satisfactory given the level of investment? If not, what factors account for the less than satisfactory results? • What QUIPS activities (e.g., workshops and training sessions) have been most responsible for increases in learning achievements? • Was the approach used to provide in-service training effective? • Was the approach used to assess pupil achievement effective? • What district grant related activities have been most useful in improving district education office capacity in planning, implementation, and M&E? • Were there any major areas not supported by QUIPS that should have been? • What role did the effectiveness of head teacher supervision play in QUIPS pupil achievement? • Did teachers use QUIPS methods and manuals as trained or did they alter or even stop using the practices? • When applied properly, were QUIPS practices sufficient for increasing pupil achievement? • What does the QUIPS experience tell us about the role of District Education Offices in pupil learning? • What was the relationship between strong community involvement and pupil achievement and was there an interaction effect of community and school-based interventions? • To what extent were QUIPS achievements attributable to QUIPS practices, approaches, methods and manuals as compared with achievements simply attributable to the QUIPS schools receiving greater attention and monitoring (i.e., impacts attributable to the Hawthorne effect)? • To what extent did district officials provide support to teachers and head teachers under QUIPS? What were the major successes and weaknesses in this area? • What was the relationship between high district performance (related to QUIPS interventions) and pupil performance and was there an interaction effect of district, school, and community factors? • Finally, was the scale of the QUIPS program (assisting 1,650 primary schools in all 110 districts) of sufficient magnitude to have a significant and lasting impact on the way primary schools operate in Ghana? 198 These questions are particularly relevant to the Education Office because the new education SO, SO8, again seeks to improve pupil achievement in schools. Thus, any lessons that can be gleaned from the existing program could be applied to the future program in order to enhance its chances for success. The results of this evaluation will inform USAID, the Ghana Education Service (GES), its implementing partners, the QUIPS recipients and the broader community concerned about primary education in Ghana about what worked and what did not work under the QUIPS program. It is envisioned that two separate reports will need to be prepared for these different constituencies. The first is a traditional evaluation for USAID, the GES, and its implementing partners. The second is a report, aimed at the QUIPS recipients and the broader education community that: a) is mindful of its audience; b) provides very practical information (in the sense that it provides “news that can be used”) from the QUIPS program; and c) is both easy and entertaining to read. This report should also lend itself to discussion in the general media in Ghana. IV. APPROACH The implementing partner is welcome to develop its own approach for conducting the evaluation. Below is an illustrative approach which may be modified as desired by the implementing organization. A. Conduct interviews with USAID Education Team Members, GES staff and relevant implementing partners (most of who remain in Ghana even though SO2 has ended). B. Conduct field visits to speak with and observe QUIPS classroom teachers, head teachers and GES district personnel. C. Review and analyze QUIPS Achievement Test Results and other QUIPS data. USAID has a comprehensive database that contains data on achievement as well as data on teacher, head teacher and SMC/PTA practices. This data is located at the EARC offices in Accra. Analysis of the data will require the application of longitudinal multi-level analysis techniques such as hierarchical linear modeling. (The final evaluation should include an appendix that describes the database in its entirety—including both analyzed and unanalyzed datasets). D. Identify possible explanations for the observed findings and trends. E. Develop an evaluation approach to assess whether any of the explanations are valid and explore unidentified explanations. F. Write draft report. G. Present draft report with initial findings to USAID, the GES and SO8 implementing partners. H. After receipt of initial feedback from USAID and the GES on the main evaluation report, draft the "lessons learned" report for the primary education community in Ghana. I. Present the lessons learned report to USAID and the GES. 199 J. After receipt of feedback from USAID and the GES on the lessons learned, report, prepare and circulate news releases on the report. Also, take other appropriate steps, including a dissemination workshop, to widely publicize the reports. K. Print copies of the lessons learned reports and distribute multiple copies to GES headquarters staff, all 138 District Education Offices and all head teachers of primary schools in districts where the QUIPS program was active. V. DELIVERABLES The contractor shall produce the following materials: • Final main evaluation report on QUIPS/SO2 activities; • Final "lessons learned" report on QUIPS/SO2 activities; • Preparation of complete electronic copies of the QUIPS databases stored at EARC, suitable for transfer to another institution; • At least one press release for local newspapers publicizing key findings in the lessons learned report; and • Contract may propose development of additional material to publicize the findings in the lessons learned report. • In addition to the submission requirements above, electronic copies of the main evaluation report and lessons learned report should be submitted to docsubmit@dec.cdie.org. For information on formatting requirements for submission, please see the webpage at http://www.dec.org.submit.cfm. 200 ANNEX 11: SUMMARY OF THE TWENTY QUESTIONS ADDRESSED IN THE FINAL EVALUATION 1. What is the relationship between QUIPS interventions and pupil learning? Analysis results indicated that, on average, pupils participating in QUIPS schools had steeper learning curves in mathematics, English reading, and in comprehending Grades 3-4 spoken English narrative than the pupils in matched control schools. That is, comparative analyses showed that during the period of active QUIPS interventions, pupils learned at an increased rate in QUIPS schools compared to the non-QUIPS schools. This finding was constant, even after controlling for individual and home, regional and urban/rural factors and the result is noteworthy. QUIPS produced learning achievement although to make a visible impact is especially difficult in a short intervention period of 24 months and when the interventions targeted general pedagogical reforms rather than intensive, subject-based treatments, such as literacy and numeracy enhancement. Follow-on analyses suggested that if the QUIPS quality improvements had been sustained, the achievement advantage to pupils in the QUIPS schools actually would have increased over time. For long term learning outcomes to be realized, however, the classroom reforms such as teacher supervision and the district and community support would need to be sustained. The evaluation fieldwork suggested that sustainability of the QUIPS program was limited and therefore there would be little achievement advantage to pupils expected over time. Chapter 5 is dedicated explicitly to analyses of pupil learning outcomes. 2. What factors, both within and outside of the QUIPS interventions, are associated with high and low achievement growth? That is, what are the common factors observed in partnership schools and in comparison schools that are associated with high achievement gains? What factors appear to constrain learning even in the presence of school-based interventions such as QUIPS? Qualitative and quantitative results pointed to a variety of factors that influenced pupil learning in Ghana’s public schools. Chapter 9 is dedicated to this question. One noteworthy finding from both the quantitative analyses and evaluative field work was the importance of home factors. Providing the basic needs of children was underscored as the most vital element to pupil learning. Pupils who were sufficiently nourished and rested for school and pupils coming from homes where parents were interested in their performance, provided a time and place for study at home, and encouraged their wards to do well in school were the pupils who performed higher. Early exposure to print through learning material available to read in the home, especially children’s reading 201 material coupled with a “reading role” model such as a mother or father visibly reading at home were found to be positively linked to a pupil’s academic development in both mathematics and English reading. Exposure to spoken English in the home also predicted higher scores in both English reading and mathematics. Sex was an additional factor related to learning, with females performing lower than males. A variety school level factors was tied to higher pupil performance. One of the findings emerging from the evaluative field work was the role of head teacher leadership. In addition, and not surprisingly, was the importance of a teachers’ commitment and availability to children in the classroom or “time-on-task.” The fieldwork pointed to a number of factors that were important in securing and sustaining teacher commitment, including: head teacher leadership and availability for supervising and supporting teachers, interest and support from district officials, enhanced integrity of the teaching profession, conditions of service including remuneration, the school environment and the availability of teaching aids. In addition, the quantitative analysis indicated that what the teacher did in the classroom was a factor in pupil learning, not simply their availability. When teachers established “pupil friendly” classrooms, encouraged and facilitated interaction and critical thinking, provided opportunities for children to demonstrate their work and gave regular feedback to pupils, learning was enhanced. Special encouragement to girls to participate in class was found to be particularly important. Finally, according to statistical results, community support for quality education where persons were empowered to act in support of quality schooling were linked to pupil learning. Results of the evaluation field work confirmed these findings that linked active communities with pupil performance. 3. Why are some comparison schools appearing to reach levels of achievement comparable to QUIPS schools? It is not surprising that some comparison schools demonstrated pupil learning that was categorized as “high.” Some of the QUIPS schools selected were relatively high performing schools in the first place and the match criteria were rigorous. Therefore, the matched control schools for these high performing QUIPS schools were high performing from the beginning. This is the principal explanation for this finding. Although the QUIPS schools outperformed the control schools in pupil learning, the differences are based on the differences in the degree of pupil learning as well as based on group differences. Importantly, the evaluative field work noted that the factors differentiating high and low performing schools were more prominent than the factors differentiating QUIPS and non-QUIPS schools. That is, many of the important factors associated with high performing schools were present in both the QUIPS and non￾QUIPS schools. 202 One follow-on analysis that would help further our understanding of this would be to rank the schools, both QUIPS and non-QUIPS, on the estimated school learning curves (i.e., measured by their respective slope coefficients) for the various subjects tested and then compare the coefficients based on actual school matches. 4. What factors are associated with fluctuations in pupil achievement outcome among the QUIPS schools? The individual factors (i.e., home and pupil) discussed in Question 2 above and in Chapter 9 accounted for the variation in pupil performance both within the QUIPS school sample and when the QUIPS and control schools were combined. Data was collected longitudinally on targeted “good” teacher instructional practices and the community (school involvement) “best” practices. These data were collected solely in the QUIPS schools. They were not collected in the control schools. This allowed for quantitative investigations about fluctuations in performance within the QUIPS schools relative to the set of targeted teacher and community practices. It precluded any comparative analyses about the targeted school and community “good” practices. Quantitative analyses about the relationship between the targeted school and community practices and the pupil achievement growth identified seven teacher practices and two community practices that were (statistically) associated with high achievement growth for a school. For the targeted teaching practices, the factors were different for northern as opposed to southern Ghana, but not for the targeted community practices. The most prominent teacher factor merging from the quantitative analysis was “encouraging girls to participate.” Other factors included, for southern Ghana: teacher use of questioning techniques; providing feedback to pupils; facilitating critical thinking; facilitating interaction; and encouraging all pupils to participate. For northern Ghana, in addition to “encouraging girls” the use of quality materials also was identified as a significant factor related to learning. The school factors differentiating between high and low performance are discussed in Chapter 6. Two community “best” practices were identified from the quantitative analysis as being associated with pupil learning within the QUIPS school population. These were “supporting school quality” and “empowering people to act in support of schools.” This is discussed in greater length in Chapter 7. 5. Why did the QUIPS program appear to have a greater impact on math and spoken English achievement and less on English literacy? This question emphasizes “QUIPS impact” on the different subject matter and therefore it is assumed that the question is concerned with the relative difference in the performance of QUIPS and control schools or the achievement advantage or “gap” 203 given by QUIPS, and the variability in that “gap” across the three subject areas. First of all, it is important to point out that the tests themselves are not directly comparable.122 Conceptually, however, comparisons of pupil achievement growth patterns (i.e., as measured by the group aggregates of the estimated slopes for schools) across the subjects are conceptually more meaningful, especially comparisons of mathematics and English reading. These comparisons should be considered, however, with caution. Analysis of results found that differences in the QUIPS/Control gap (i.e., based on learning growth, not static achievement gains) across the subject areas were dependent on grade level. For Grades 3-4, the relative performance gap (i.e., difference between QUIPS and control schools) favored English reading and spoken English narrative with learning growth curves slightly higher for these subjects than for mathematics. For Grades 5-6, as predicted by the question, the relative performance gap favored mathematics, with the learning growth curves higher for mathematics than for English reading or spoken English narrative. Importantly, it is noted that, overall, the gap between the QUIPS and Control schools was greater for Grade 3-4 than for Grade 5-6, especially for English reading and spoken English narrative. This is not surprising. Pupils in the early grades are still in the formative stages, with known “developmental readiness” for learning. Findings from this evaluation pointed to the significant delays in pupil performance that exist among upper primary pupils in Ghana. With these types of delays it is not surprising that the impact was slightly attenuated for the upper grades, especially considering that QUIPS did not focus specifically on literacy and numeracy enhancement and did not introduce any accelerated learning approaches to address the severe academic development delays. These results underscore the importance of: 122 The English narrative test is a different type of test altogether emphasizing the development of a child’s use of spoken English in a somewhat contrived context. The QUIPS team felt it important to track pupil learning in spoken English as well as English reading for two reasons, the development of spoken English is part of the curriculum and the QUIPS school level interventions did include a component of second language learning. Direct measures of English in a “test format” often fail to capture spontaneous use of the second language. For example, very often a question and answer format is used, often assessing a child’s use of rote memorized forms such as greetings. However, the evaluation of spontaneous discourse was just prohibitive due to the time and skill needed to assess discourse with any validity and reliability. Therefore, the spoken English narrative task was chosen for the purpose of tracking a child’s second language development, fully acknowledging the narrative provided a somewhat narrative view of the child’s overall language development. The mathematics and English reading tests are more similar in structure, though still not directly comparable. The tests are tied to essential skills in mathematics and English reading and cover the entire primary school curriculum. This situation confounds the ability to compare performance across tests, particularly with regard to static achievement scores and especially comparisons of mathematics or English reading to English narrative. It would be possible, with further psychometric study, to calibrate the mathematics and English test instruments so that these tests would be directly comparable. 204 • focusing on the lower primary grades to support the establishment of a good foundation for literacy and numeracy attainment (and because there is clearly greater impact at the lower levels) • focusing on enhancing learning in the subject area where there are the greatest overall deficiencies • ensuring that there are sufficient textbooks available in the classroom and teachers that know how to use them as part of instruction • ensuring that there are sufficient opportunities for children to practice what they know by providing supplemental readers in the classroom and loaning texts to pupils for use at home • for older children (i.e., pupils in upper primary or junior secondary grades), there is the need to include a component of accelerated or remedial learning 6. Has the increase in the achievement scores at QUIPS schools been satisfactory given the level of investment? If not, what factors account for the less than satisfactory results? The limitations in “return on investment” with regard to the achievement advantage given by QUIPS are related to the seemingly limited sustainability of the quality improvements that led to those achievement gains than the achievement gains themselves. It was clear from the results of this evaluation that QUIPS successfully increased the rate of learning for pupils participating in the QUIPS schools. This result was statistically significant even when controlling for a number of variables known to effect learning, including regional and urban/rural differences. Although the magnitude of the achievement gains measured at the end of the two-year intervention was not great, it is noteworthy that such a shift was made at all given the type, intensity, and length of the intervention. Results suggest that it was a combination of school, community and district effects that were responsible for the achievement results. The inter-workings of these factors all served to support child learning. If the inter-connectedness and positive social change for quality schooling would have been sustained and made widespread, the return on the QUIPS investment could be said to have been satisfactory. The results of this evaluation indicate that this was not the case and therefore, the return on the investment cannot be considered “satisfactory.” The diffuse design with 3 schools in all districts precluded the ability to work closely within the system at a district level or to build networks of support that teachers need to implement and sustain classroom reforms. In Ghana where there is considerable teacher mobility within a district, networking of innovative teachers is important for innovation. In addition, the QUIPS program design for a few model schools limited the level of support and guidance possible for any teacher group. The alternative of a district focus would have allowed for considerably more teacher-guided practice to support and validate sustained teacher change. 205 Ghana’s pre-service teacher training programs were not impacted so that new teachers could benefit from the instructional techniques given by the QUIPS program. District supervision was concentrated in the QUIPS schools during the two-year intervention at the expense of attention to non-QUIPS schools in the district. This level of supervision and support played a role, collectively with other inputs, in influencing pupil learning. But this level of effort could not be expected to be sustained, nor should it be when other schools/communities were (reportedly) marginalized as a result. There is evidence that communities across the nation benefited from the QUIPS program and that there is some sustainability of the community “good” practices. Nevertheless, community change in and of itself is not sufficient to promote marked and sustained shifts in pupil learning. As discussed above, it appears to be the QUIPS’ integrated approach that produced the “achievement advantage” for pupils. Thus it is the inter￾connected inputs that must be sustained in order to expect a sustained advantage. 7. What QUIPS activities (e.g., workshops and training sessions) have been most responsible for increases in learning achievements? As discussed above in Question 6, the learning gains were attributed to the inter￾workings of all components of the QUIPS program, including: classroom, school and district administration plus the community/home factors. In this way the QUIPS design served to promote learning through combined effects. The inter-connected inputs facilitated teacher commitment and confidence. During the two-year period of active QUIPS interventions, teachers were supported in a positive way by both the QUIPS master trainer team and the Circuit Supervisors in the district. Their confidence was further strengthened by the attention given to the children’s learning and the operations of the school by the community in general, and parents in particular. QUIPS served to build an inter-connectedness among various components related to schooling: the teacher, head teacher, district administration, the pupil and the community. Committed, confident teachers and pupils who feel encouraged and supported by both teachers and parents translate into more quality time-on-task. In sum, confident teachers and encouraged pupils in the classroom… at the same time… produced more teaching and learning during the class time. If you add to this process the improved instructional techniques and community practices known to promote school effectiveness, one may predict achievement advantages for pupils benefiting from the QUIPS set of interventions. It would be unfair and inaccurate to point to any one or more approaches being the most responsible for increases in learning achievements. Rather, it appears to be the result of a combined effect. Having said that, the design of QUIPS failed to provide an opportunity to address this question from a research perspective. It is difficult to compare the approaches within QUIPS in a valid way. To do so would have required that a sub-set of school/communities receive only a few but not all of the inputs. For 206 example, some schools could have been assigned only the school-level inputs while another set of schools could have been assigned only the community interventions. This was not done. Indeed, had QUIPS decided on this design, it would have contributed in a more meaningful way to understanding the relative impact of different interventions as well as their collective outcome. In the future, it would be wise to invest in a design that provided an opportunity (micro-study) for this sort of comparative research. 8. Was the approach used to provide in-service training effective? This question was investigated by a leading teacher trainer in Ghana, Dr. Felicia Etsay, Faculty of Education, University of Cape Coast. Her analysis is summarized briefly in the following: The QUIPS program embraced important elements of school-based training; however, some of the important assumptions of this approach were not met by QUIPS. The school-based training provided opportunities for ‘novice’ teachers to work side-by￾side with ‘expert’ master trainers in the classroom to address the challenges of teaching in real classrooms in Ghana. QUIPS used the following strategies in the training, all aligned with the “Cognitive Apprenticeship” model in teaching novice learners: demonstration; self-evaluation; appreciative inquiry; and journal writing. Some of the assumptions of the school-based model were not met in the QUIPS teacher-training program. First, demonstration lessons need sufficient guided participation and practice and the master trainer’s models must be explicit in order for the novice to understand the instructional information. Findings from the evaluation field work suggested that supervised practice by classroom teachers was limited during the QUIPS in-school program. The demonstrations by master trainers were considered to be insufficiently explicit and redundant according to interviewees. The amount of on￾site training provided by QUIPS (i.e., one week per term) combined with a large number of targeted instructional methods designed into the training compromised a teachers’ ability to assimilate the new techniques. It was more difficult for the teachers to apply them appropriately and “naturally” in the classroom. Attaining independence and confidence in performing a task does not come easily. It comes with prolonged practice and feed back from a teacher-mentor. The amount of practice delivery of lessons in the QUIPS teacher training needed to be increased. To confirm this, the field observations of the classroom practice of QUIPS-trained teachers showed that there was limited sustainability of the instructional methods targeted in the six in-school teacher training sessions. Second, although the QUIPS ‘master’ trainers were supposed to be well trained in all of the targeted practices, field results provided evidence that this was not always the case, 207 resulting in unexpected negative outcomes. For example, although QUIPS-trained teachers were observed to apply strategic grouping in the classroom, the teachers did little to encourage pupil interactions within these groups. The lack of pupil participation defeated the purpose of the strategic grouping in the first place. Furthermore, responses from QUIPS-trained teachers in the field suggested that there were unfortunate misconceptions about what are good “low-cost no-cost, teacher-made” teaching and teaching aids. Considering the exceptional models presented to the trainees by QUIPS (i.e., very pretty materials made with strong poster board, colored markers, and other costly purchased materials), teachers and district officials alike inaccurately learned that it is impossible for teachers themselves to prepare their own classroom materials without considerable external monies. The field studies found that the teachers do not prepare their own classroom materials. A number of teachers and officers expect them to be purchased. Where TLM’s were available, rarely did the QUIPS evaluation team see evidence of use of functional TLM’s. Finally, the teacher manuals and handbooks were judged to be too technical for promoting the widespread reading and use by teachers and head teachers.123 The QUIPS residential workshops were held for eight days and involved head teachers and circuit supervisors as well as regular teaching staff. Considering the large number of people (e.g. 600 at a time) who attended these workshops, it would have been more effective if the group had been divided into two or more groups and the number of days of training extended to at least two weeks. This alteration would have allowed the participants to be more thorough in going through all the course topics.124 9. Was the approach used to assess pupil achievement effective? An external analysis of the test instruments was obtained and reference is made to this report in Annex 16. The final summative paragraph taken from this reports suggests that the approach was effective: “All in all, the QUIPS tests are remarkable and the principles (particularly the opportunities to practice testing processes for the pupils, attention to details of administration, provision of language support, and careful training 123 In its review of the draft evaluation, the Academy for Educational Development stated: “Regarding the TLMs it is the case that the teachers themselves made the TLMs at a QUIPS sponsored workshop with materials provided by QUIPS - but these materials were very basic (poster board, marking pens). They were not expensive per se, but the headteachers and teachers did indeed complain that they received little support from the DEO and community to cover the costs of purchasing such materials once QUIPS phased out in their schools.” 124 The Academy for Educational Development also noted that: “Regarding the residential workshops, the largest training program was one that approached 300 participants at a training site at one time, but there were over twenty master trainers at this training site and there were numerous classrooms at the site for small group work. The written reports of these workshops may have given the erroneous impression that 600 were at the training site at the same time. This was not the case. Instead, the 600 participants or so were split into two groups, with each group of about 300 participants taking part in the training with some 20-25 trainers. These workshops were conducted twice, sequentially, thus reaching a total of 600 participants.” 208 of administrators) that underlie their development should be more widely applied in other reform evaluations. They were excellent tests for their role at the time, and their approach provides a model for evaluative assessment development.” 10. What district grant related activities have been most useful in improving district education office capacity in planning, implementation, and M&E? This question is addressed in Chapter 7 of this report. Overall, the management training, including the training associated with the District Grant Mechanism and related Monitoring and Evaluation training was viewed as useful and relevant. It is noteworthy that the compiled result of the grant mechanism and training efforts served to introduce district education offices to the processes of open discussion and group decision￾making. This is an important first step in the context of a nation that is beginning to realize the challenges of decentralized education management. Although this shift in district management from a vertical to a more open and “horizontal” management approach has been difficult to sustain, districts still recognize the value of the approach. As Ghana moves into a capitation grant scheme in support of quality schooling, school and district capacities to practice group decision-making, participatory approaches with wide stakeholder input, and data-driven decision-making will become ever so more critical. Although the District Grants Program was valuable in its own right and successful in spreading certain aspects of the QUIPS program across the nation, there were limitations with regard to the extent and time given for the spread activity. These are discussed in more detail in Chapter 7. One element of the spread activity was that districts were encouraged to “choose” the aspect of QUIPS that they wanted to spread. Yet, in fact, QUIPS impact is tied to the integrated approach that embraces the importance of “1+1 = 3” or “whole school reform.” The QUIPS integrated approach simultaneously helped to strengthen the effectiveness of teachers in classrooms, parents in communities, officers and systems in the districts as well as school infrastructure. The district officials interviewed recognized the limitations in a spread activity that involved selecting only a few of the elements of the integrated whole. 11. Were there any major areas not supported by QUIPS that should have been? Four areas stand out as lacking in the QUIPS program. The first is the limited engagement of the government in the design and implementation of QUIPS, particularly the GES Headquarters and Ghana’s teacher education program. During the second half of QUIPS under the district grants program, the GES at the district level became more involved in implementation. At the national level, GES participation was viewed as minimal, which precluded substantive systemic change, particularly with regard to institutionalizing the best of the school and district level initiatives. District and national 209 education officials characterized QUIPS as a donor-driven activity and viewed the arrangement as problematic. A second area is related to the effort of QUIPS to work through the teacher training program in addition to the in-service training at schools. There was a deliberate attempt to share materials and techniques with Ghana’s national teacher training program. Moreover, the techniques emphasized in the QUIPS in-service training were integrated into the national curriculum. Nevertheless, the focus of action by QUIPS was to target training to the teachers working in QUIPS schools and this outreach was not conducted within existing systems of in-service operating from the district offices. The evaluation team all agreed that working within the teacher education system may have led to a more sustainable and systemic change in teacher upgrading. Another lack in the program was the absence of focus on some of the critical non￾teacher factors related to pupil learning. More action during QUIPS needed to be taken to ensure that textbooks were available, that teachers had access to them, and that teachers were sufficiently skilled in using them. Given what we have learned about the importance of having a text and supplementary reading material in the home, QUIPS could have supported the school system to develop a book loan program to mobilize schools and districts to help pupils use texts and other supplemental readers in their homes. Finally, although the community program worked toward building parent awareness about the importance of providing for their children’s basic needs such as nourishment, rest and encouragement and strived to help parents understand the link between these basic needs and pupil learning, a more focused approach in this area may have been useful. Both quantitative and qualitative results underscore the importance of providing for a child’s basic needs not only during but before entering formal primary school. 12. What role did the effectiveness of head teacher supervision play in QUIPS pupil achievement? Results from the evaluative field work underscore the importance of head teacher supervision and leadership in the overall achievement of pupils. This is discussed at length in Chapter 6 and Chapter 9. 13. Did teachers use QUIPS methods and manuals as trained or did they alter or even stop using the practices? The evaluative field work and an external evaluation of the in-service training suggest that the school-level methods were altered. It was clear form the site visits that very few QUIPS classroom reforms were still apparent in the former QUIPS schools unless the school maintained a strong core of QUIPS-trained teachers and head teacher leadership. The commitment and classroom practices of QUIPS-trained teachers who 210 were no longer working side-by-side with other QUIPS teachers was not as strong as that of teachers who continued to work in an “intact” QUIPS school. Positive practices observed to be maintained by QUIPS-trained teachers in some schools included: positive attitude and encouragement of all pupils in the classroom; provision of feedback to pupils; teacher use of questioning; and teacher use of lesson plans. Rarely did the evaluation teams observe teaching and learning materials being used as effectively in the classroom as they had been reported to be during the time of QUIPS. Although teachers sometimes grouped children around a table, there was little to no encouragement or facilitation of pupil-to-pupil interaction within the groups. When communities were observed to sustain their involvement in schools, the classroom practices were more similar to those prescribed by the QUIPS program. The evaluation field work did observe that even when community participation in school improvement had deteriorated, there was universal awareness of and desire to “get back on track” and to upgrade their support for school quality. 14. When applied properly, were QUIPS practices sufficient for increasing pupil achievement? In the final analysis, it does appear that the QUIPS practices were not sufficient, particularly considering the delays in pupil achievement that exist in Ghana today. Question 5 above points to a number of areas of improvement that are needed in future programming. The following additions relate more specifically to Question 14. • Considering the limited literacy and numeracy attainment that exists in Ghana today, a strong focus on action in one or both of these subject areas is called for. QUIPS’ teacher training covered many areas but did not focus per se on literacy and numeracy enhancement. That is, there was great breadth in the QUIPS training program, but insufficient depth with regard to these two key primary school subject areas. Although pupil-centered instruction and an active learning environment are important, Ghanaian teachers, especially in the lower primary classes, need dedicated support and assistance in the teaching of reading and mathematics. Providing for the basic building blocks for literacy and/or numeracy attainment in the early classes is critical. Learning in the upper primary grades depends on certain essential skills being mastered in the early grades. • More attention could have been directed toward mobilizing the attention of parents toward child learning and helping them understand the need to provide, from the pre-primary ages and into primary school, certain basic physical and cognitive needs related to learning readiness, including: nourishment and rest, 211 encouragement, exposure to print and people reading in the home, and exposure to spoken English.125 • Given the severe academic delays of pupils in the upper primary grades, it is suggested that interventions targeting the upper primary and junior secondary school populations should include elements of accelerated learning. Without this, it is not likely that substantial gains could be expected relative to developmental and curriculum expectations. 15. What does the QUIPS experience tell us about the role of district education offices in pupil learning? Chapter 8 is dedicated to this question. To be sure, the evaluation results pointed to the district involvement as important for school performance. First of all, the report underscores the need for districts to develop awareness about the status of schools in the district and the status of pupil performance in those schools. Interviews suggested that district monitoring and evaluation training program did much to build this awareness and laid a foundation for district decisions to be more information-based than they were in the past. The role of the district in providing positive, supportive supervision through the circuit supervisors was underscored as a factor in school effectiveness and pupil achievement as well as district led teacher professional development. Districts too have accepted the responsibility to maintain the level of community involvement in schools. Unfortunately, there were obstacles to the districts providing the level of supervision, training and support that was provided to QUIPS schools. These obstacles are related to funding, especially the inefficiency in the flow of funds from central to district level. 16. What was the relationship between strong community involvement and pupil achievement and was there an interaction effect of community and school￾based interventions? The answer is to this question is yes. Indeed, in the final analysis, it was the combined inputs that appeared to have made a difference. The answer to this question is discussed in Chapter 7 in the report and also Question 7 above. 17. To what extent were QUIPS achievements attributable to QUIPS practices, approaches, methods and manuals as compared with achievements simply attributable to the QUIPS schools receiving greater attention and monitoring (i.e., impacts attributable to the Hawthorne Effect)? The quantitative results indicate that there were specific QUIPS interventions that accounted for learning outcomes, including certain targeted teacher behaviors and 125 Our pupil data showed that though there was someone located in the home who spoke to the child in English, English was not necessarily used when speaking to the child. Therefore, our results suggest that any person in the home who is able to speak English should make an effort to use English, at least some of time, when speaking to the child. 212 community best practices. These are discussed in Chapter 6, Chapter 7, and Chapter 9 and in Questions 1 and 2 above. Therefore, QUIPS effects would not be accurately attributed to the Hawthorne effect. The combined school, community and district interventions did serve to shift pupil learning in a positive way. It must be said; however, that the high visibility of the QUIPS schools in the district resulted in unexpected and negative outcomes, two of which were related to the overall attention given to the three QUIPS schools (and this could be construed as a “negative” Hawthorne effect, if you will). First, parents with children in near-by schools transferred their children to QUIPS schools. This resulted in reduced enrollments and less attention being given to the nearby schools. Second, because districts wanting to “do well by QUIPS” turned all their attention to providing the requested inputs (i.e., trained teachers, teacher support and supervision) to the three QUIPS schools, district office attention to other schools in the districts was compromised. This is discussed in the next Question (18.) below. 18. To what extent did district officials provide support to teachers and head teachers under QUIPS? What were the major successes and weaknesses in this area? Clearly, during the QUIPS period, district officials supported teachers and head teachers in the three QUIPS schools. The evaluative fieldwork suggested that because of all the “eyes on QUIPS” teachers and head teachers, their performance was sustained at a higher level during the two-year intervention cycle. However, once QUIPS ended, the higher level of support and supervision by the district returned to normal levels. This return is understandable because district education officers reported that during the QUIPS period, they gave other schools in the district less attention. The local view was that the imbalances in the district brought on as a result of the QUIPS program needed to be equalized. 19. What was the relationship between high district performance (related to QUIPS interventions) and pupil performance? Was there an interaction effect of district, school, and community factors? As mentioned in Question 7 above, the learning outcomes of QUIPS are best attributed to the combined inputs and related changes on the part of: districts in their enhanced quality of support and supervision; teachers and head teachers; and parents and community members. No single intervention was more or less effective than the other, although there were observed differences in the sustainability of the reforms at the different levels. The community changes appeared to have been better sustained and more widely spread than the school level or district reforms. The limitations in school￾level changes have much to do with the mobility of teachers and the isolation inherent in the selection of three schools per district. The district level support and supervision was 213 not maintained at the level given during QUIPS because of constraints in finance and staffing and the general belief that it was not fair. 20. Finally, was the scale of the QUIPS program (assisting 367 primary schools in all 110 districts) of sufficient magnitude to have a significant and lasting impact on the way primary schools operate in Ghana? The answer to this question is decidedly “no.” This is discussed in length throughout the report from a review of the program design to the details of QUIPS implementation as well as the findings on sustainability from the evaluation field trip in April, 2005.