USAID/Zambia Assistance to Strengthen Ministry of Education Support to Community Schools May 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by DevTech Systems, Inc. USAID/Zambia Assistance to Strengthen Ministry of Education Support to Community Schools Authors: Peggy Chakufyali Jane Chinombwe Judith Oki, Team Leader May 2008 Prepared by DevTech Systems, Inc. under MOBIS Contract #GS-10F-0048L, Purchase Order # 611-O-00-08-00003. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Acronyms ............................................................................................................. iii I. Executive Summary ........................................................................................... 1 II. Overview ........................................................................................................... 9 III. Background and Context: USAID Support to Community Schools ................ 10 Background ................................................................................................................. 10 Historical Overview of Community Schools ................................................................ 10 Collapse of the Zambia Community Schools Secretariat ............................................ 11 Current Status of Community Schools ........................................................................ 12 Overview of USAID Programs Supporting Basic Education ....................................... 13 Summary ..................................................................................................................... 15 IV. Evaluation Methodology ................................................................................ 16 Evaluation Approach ................................................................................................... 16 Evaluation Activities .................................................................................................... 18 Data Analysis .............................................................................................................. 19 Limitation and challenges in the methodology ............................................................ 19 V. Evaluation Findings ........................................................................................ 21 USAID Program Level Results .................................................................................... 21 CHANGES2 ............................................................................................................ 22 EQUIP2 ................................................................................................................... 22 QUESTT .................................................................................................................. 22 MoE Institutional Level Results ................................................................................... 24 National Level ......................................................................................................... 24 Provincial/District Levels ......................................................................................... 30 Community and School Levels ................................................................................ 34 Classroom Level ..................................................................................................... 39 VI. Conclusions ................................................................................................... 42 Systemic Support to Policy Implementation ................................................................ 42 Perceptions of Quality ................................................................................................. 43 VII. Recommendations ........................................................................................ 46 Policy Implementation ................................................................................................. 46 Teacher Development and Support ............................................................................ 46 Teacher Placement ..................................................................................................... 47 Teaching and Learning Materials ................................................................................ 47 Infrastructure development ......................................................................................... 48 Expanding Available Resources ................................................................................. 49 Defining and Monitoring Quality .................................................................................. 49 i Annexes: Annex A: CHANGES2 Findings Annex B: EQUIP2 Findings Annex C: QUESTT Findings Annex D: Sites Visited Annex E: Individuals Contacted Annex F: Interview Schedule and Questionnaire Annex G: Integrated Information Gathering Tools Annex H: Questions for Ministry of Education and Cooperating Partners Annex I: References Annex J: Scope of Work ii ACRONYMS BESSIP Basic Education Sub Sector Improvement Program CA Continuous Assessment CBO Community Based Organization CDC Curriculum Development Center CHANGES2 Community Health and Nutrition, Gender and Education Support Program COE College of Education CS Community School CST Community School Teacher (volunteer) DEBS District Education Board Secretary DESO District Education Standards Officer DEST District Education Support Teams DODE Directorate of Open and Distance Education DRCC District Resource Centre Coordinator ECZ Examination Council of Zambia EMIS Educational Management and Information Systems EQUIP2 Educational Quality Improvement Program ESO Education Standards Officer FBO Faith-Based Organization FGDs Focus Group Discussions FNDP Fifth National Development Plan FTI Fast Track Initiative GC Guidance and Counseling GRACE Grade Meetings at Resource Center GRZ Government of Zambia ICT Information Communication Technology IGA Income Generating Activity IIM Integrated Information Management IM Institutional Management INGO International Non-Governmental Organization IRI Interactive Radio Instruction JSC Joint Steering Committee M&E Monitoring and Evaluation MoE Ministry of Education NGO Non-Governmental Organization NIF National Implementation Framework ODL Open and Distance Learning OVC Orphans and Vulnerable Children PAGE Program for the Advancement of Girls Education PAM Program Against Malnutrition PCSC Parent Community Schools Committee PEO Provincial Education Officer PESO Provincial Education Standards Officer iii iv PEST Provincial Education Support Teams POC Provincial Outreach Coordinator PRCC Provincial Research Centre Coordinators QUESTT Quality Education Services Through Technology SCP School-Community Partnership SEO Senior Education Officer SESO Senior Education Standards Officer SHN School, Health and Nutrition SIC School In-service Coordinator SPARK Skills, Participation, Access to Relevant Knowledge SPRINT School Program of In-service for the Team TA Technical Adviser TED Teacher Education Development TESS Teacher Education and Specialized Services TG Teacher’s Guide UNICEF United Nations Children’s Emergency Fund USAID United States Agency for International Development ZATEC Zambia Teacher Education Course ZCSS Zambia Community Schools Secretariat ZIC Zonal In-service Coordinator ZOCS Zambia Open Community Schools I. EXECUTIVE SUMMARY Introduction This report presents the results of an independent assessment of the contributions of three U.S. Agency for International Development Zambia Mission (USAID/Zambia) programs to strengthen the Ministry of Education (MoE) support to community schools. The report is intended to use the experience of the programs to assess changes in the overall landscape of community schools and to shed light on practices that promote equity and quality in community schools. Background and Context Community schools (CSs) in Zambia account for approximately 2,500 of the providers of basic education in the country, having grown from 55 in 1996 to 2456 in 2006. Between these years, registration, coordination and channeling of government resources was the function of the umbrella Zambia Community Schools Secretariat (ZCSS), supported by non-governmental organizations (NGOs) and international donors. CSs used a curriculum that condensed the basic education curriculum and integrated practical knowledge and skills. In 2006, the ZCSS collapsed, leaving support to CSs largely to NGOs and pro-active provincial and district education offices. The MoE, recognizing the need to regularize and embrace these critical partners in basic education provision, took on the task of building a comprehensive policy framework and implementation guidelines for CSs. At the same time, USAID/Zambia recognized the opportunity to assist the MoE in this effort and was among the select missions to receive USAID “Fast Track Initiative” (FTI) funding.1 Three projects received FTI funds to adapt or expand their programs to support CSs over a period of approximately 18 months that ended in December 2007. USAID Basic Education Portfolio Existing programs with the capacity to adapt their activities for CSs are: Community Health and Nutrition, Gender and Education Support Program (CHANGES2), the Educational Quality Improvement Project (EQUIP2) and the Quality Education Services Through Technology (QUESTT). Taken as a whole, these programs had the mandate, technical skills and established relationships with MoE structures to enable the MoE to advance a critical CSs agenda that embraced policy, capacity building, and improved quality of teaching and learning. EQUIP2, working at the national level and fully integrated into the MoE, was positioned to promote and assist a highly participatory process for developing the required policy shifts from ZCSS coordination and support to MoE-led coordination and support. CHANGES2 and QUESTT had mandates to develop and support school level quality, strengthening MoE structures beyond the national level to classrooms. CHANGES2 introduced fundamentals of teaching and school management, building on existing practices and strengthening in-service professional development structures; QUESTT expanded its Interactive Radio Instruction (IRI) 1 Though sharing a similar name and intent, this fund is distinct from the Fast Track Initiative of the Education for All Secretariat. 1 methodology and an innovative approach to formal teacher training. Both built human capacity at the community level for resource mobilization and use of improved infrastructure. Descriptions of each program can be found in Section III. Evaluation Purpose and Methodology This evaluation was commissioned by USAID to “assess the degree to which USAID supplementary support to the MoE through the CHANGES2, EQUIP2, and QUESTT projects strengthened the ability of the MoE to support CSs at the various institutional levels of the education delivery system.” The evaluation assesses program coverage, effectiveness and impact, documents lessons learned, and provides summary findings, conclusions and recommendations, including suggestions for continued support to CSs. The evaluation model was primarily qualitative, supported by the wealth of quantitative data collected by the Planning Directorate of the MoE and its USAID project partners. Most of the primary data collection came from observations, interviews and focus group discussions conducted at the provincial, district and school/community levels with education officers, teachers, parents, students and community members. The evaluation results, therefore, reflect perspectives of individuals at multiple levels of the educational delivery system. Program staff were invited to observe interviews, focus groups and other field-based activities, in addition to being interviewed for information-gathering purposes. The overall evaluation approach is illustrated on page 17. A three-person evaluation team visited 17 schools in 11 districts and five provinces. At each school, managers, teachers, pupils and community members were interviewed. At 11 schools teachers were observed in the classroom. Teachers at six schools were not observed for the following reasons: three schools in the Northwestern Province were visited to learn about the CHANGES2 small grants program and the schedule precluded classroom observations; three additional schools were conducting end of term exams. See Annexes D-H for data collection tools, sites visited and individuals contacted. Major Findings The report provides findings about each program at various institutional levels along the education chain of delivery, reflecting the breadth and depth of the three programs when taken together. The evaluation team found that USAID programs have substantially improved the overall landscape for CSs. Additionally, the programs have furthered several of the MoE’s broad objectives as articulated in the Education Sector National Implementation Framework (p. 19), including: • strengthening institutional frameworks to coordinate provision of education; • developing policy changes reflective of the current educational requirements; • promoting alternative modes of education provision; and • promoting innovative methodologies in learning institutions. The findings for each program evaluated are summarized in tabular form in the body of the report on page 21, and detailed in Annexes A-C. The findings by institutional level derived from the programs are described in detail on page 24. 2 Key findings include: 1. USAID programs have promoted critical changes since 2006 in the overall landscape of CSs in the following ways: • A policy framework with implementation guidelines has been established through a highly participatory, consensus-building process. CSs are included in the MoE’s Annual Work Plan and Budget and are represented in national school level pilot activities. (EQUIP2) • Pilot programs of the MoE in Recordkeeping for School Managers included 1,500 community school managers. The Examination Council of Zambia (ECZ) pilot in continuous assessment reached 180 community school managers. (EQUIP2) • 948 Community School Teachers (CSTs) and 412 Government of Zambia (GRZ) Basic School Teachers have been trained for 20 days in basic teaching skills; 948 managers trained for five days in fundamentals of school management. (CHANGES2) • 321 school/community committees have been trained and supported to work together to complete school infrastructure. (CHANGES2 and QUESTT) • Interactive Radio Instruction (IRI) has been extended to an additional 1,000 community school teachers and 3,000 GRZ teachers through zonal in-service training coordinators. (QUESTT) • Over 450 CSTs have been enrolled as distance students in the Zambia Teacher Education Course (ZATEC) in eight Colleges of Education. (QUESTT) • Tools, skills and approaches to monitoring and support to CSs have been practiced by MoE provincial, district and zonal level officers, including those responsible for planning, building, procurement and educational standards. (CHANGES2, EQUIP2, QUESTT) 2. At all levels of the MoE, from national headquarters through district offices, there is a shared sense that CSs are critical partners in the provision of basic education that create access for groups that have not been able to attend GRZ schools due to distance, stigma and/or poverty. 3. The new operating guidelines have been welcomed, albeit cautiously, throughout the system. Proactive districts, which have consistently supported CSs, find that they validate existing practices. Other districts and all provinces are concerned about securing the necessary resources to implement the guidelines and maintaining a balance between government support and community ownership. 4. The use of existing structures of the MoE system, especially district officers and sub￾district structures has strengthened the capacity for the delivery of continuous professional development, and is an effective means of reaching the school level. 5. The Colleges of Education have been trained to develop distance training modules and will be engaged in the EQUIP2-led development of a comprehensive training and qualification program for school managers. The Colleges of Education are an 3 effective arm of the MoE yet, by design, they are less flexible in responding to identified needs of schools. 6. Community perceptions of quality, as well as their use and support of schools are influenced by infrastructure development, staff and community development and pupil performance. Conclusions 1. The evaluation team found that USAID programs have substantially improved the overall landscape for CSs. They have furthered several of the broad objectives of the MoE as articulated in the Education Sector National Implementation Framework (p. 19). 2. USAID assistance to strengthen the MoE support to CSs has been thorough and timely. It has enabled the MoE to develop a comprehensive framework and operating guidelines for CSs. Its programs have provided models for delivering support to CSs in ways that build systemic capacity and enhance education quality. The programs have recognized and strengthened existing models of support provided by district offices and NGOs, and their experience can be used to shape a cohesive approach for continued support by government, cooperating partners, and international and national NGOs. The evaluation draws conclusions in terms of systemic support to policy implementation and of perceptions of quality. Elements of quality include development, placement and retention of teachers in CSs, the provision of infrastructure, and teaching and learning materials. Systemic Support to Policy Implementation The following conclusions have been drawn regarding systemic support to policy implementation: • Districts are apt and appropriate points of coordination and support to CSs. District-led support and coordination provide the necessary flexibility in responding to CSs and are in line with decentralization policy. QUESTT and CHANGES2 have demonstrated that working with and through district structures is an effective way to deliver services to CSs. • There is no need at present to “replace” ZCSS coordination and resourcing functions as the roles and responsibilities for civil society and governments are spelled out in the guidelines. There may be a role for a national umbrella organization to help monitor the implementation of the guidelines and related advocacy work. • USAID programs have access and influence at all levels of the education system, and the experiences and lessons of the programs can be better coordinated to 4 improve the quality of teaching and learning, including school management, materials provision and infrastructure development. • Communities and government at all institutional levels recognize the need to maintain balance between community leadership and initiative, and government support. • Small grants for infrastructure development, whether through government or external sources, promote continued community participation. The small grants programs required a community contribution, and have generated community skills and cooperation, connected communities with relevant district offices and completed standard infrastructure. Perceptions of Quality The strategic framework and policy guidelines of the MoE provide a vision for CSs as full and active partners in the provision of quality basic education. The following conclusions have been drawn regarding improved quality in CSs: • Perceptions of education quality increase when teachers, school managers and school committees are trained and feel competent; infrastructure is improved; and creative and innovative practices are used in the classroom. • Training and support have improved the basic operation of the CSs, teacher and pupil confidence, teaching practice and community relations. • Teaching practice is improved through training and follow-up support and through the availability of teaching and learning materials, including the syllabus. • CSTs have demonstrated their ability to use very basic teaching aids such as those provided in the teachers’ tool kit to make and use teaching aids such as the geometric shapes, and alphabet charts, and to use modern technology such as an MP3 player. • Where it has been done, placement of trained teachers at CSs has been achieved by seconding teachers from a GRZ school establishment. Although the seconded teachers are often placed as “teacher in charge,” no tensions between the two were identified in the schools visited. • Teacher morale and retention are improved by consistent (even when small) allowances/support from the community. • The Zambia Teacher Education Course (ZATEC) distance model has been a motivator for those enrolled and by design, has kept the CSTs in school for at least four years. However, the experience of other programs indicates that CSTs who complete ZATEC are often not returned to their communities or experience long delays in recruitment. • Improved infrastructure creates a sense of stability and permanence. Furniture, and teaching and learning materials, including MP3 players, have similar effects. • Maintenance of infrastructure and equipment is critical to sustainability. Infrastructure maintenance is included in the grants management training, and QUESTT has built a support network for MP3 players through Peace Corps Volunteers. The current system for returning IRI radios to Lusaka is, however, inefficient. 5 • Successful use of grant funds attracts additional resources. This practice has been encouraged during training of communities. About five schools have sought and/or received support from other sources based on the improvements made through the CHANGES2 and QUESTT grants. • Though communities articulate the need to make school improvements, they have not yet been directly involved in defining and monitoring the quality of education. Recommendations The report makes recommendations in several thematic areas including policy implementation; teacher development and support; teacher placement; teaching and learning materials; infrastructure development; expanding available resources; and defining and maintaining quality. The recommendations detailed in Section VII are summarized below: Policy Implementation 1. Continue program implementation through existing MoE structures, especially at the district and sub-district levels. 2. Manage misinterpretation of guidelines to avoid unrealistic community expectations or fears, and to define and maintain community ownership of the schools by leveraging existing networks, activities and skills of all three USAID programs to develop a standard package of messages. 3. Continue to expand capacity and resources to districts to retain flexible and comprehensive support to CSs. 4. Establish an approach to monitoring the implementation of the guidelines. Teacher Development and Support 1. Develop a pathway to professional recognition for teachers and school managers through a competency framework that links academic record, performance and short courses. Include issues related to status of trained government teachers who serve as “teachers in charge” of CSs. Define the roles and inputs of the district in service network and the Colleges of Education. 2. Promote regular financial and/or in-kind support to CSTs. 3. Ensure that district level staffing, including District Resource Center Coordinator (DRCC) and Zonal In-service Coordinator (ZIC) workloads are sufficient to serve the expanding requirements of district support to teacher professional development. 6 Teacher placement 1. Establish the legal status of CSs that will enable them to participate in government payroll. In the interim, districts should continue seconding trained and salaried teachers to CSs. 2. Include CSTs who finish ZATEC in the recruitment and posting cycles as soon as possible, returning them to their communities of origin for an agreed upon period of service. 3. Promote balance between the assignment of trained government teachers to government schools and the potential displacement of community volunteers. Teaching and Learning Materials 1. Maintain the benefits of IRI by establishing a system for locally-based troubleshooting and repair of radios, and expanding use of MP3s. 2. Identify private sector resources to sponsor additional MP3 players and upgrade the quality of the teachers’ toolkit. Infrastructure development 1. Maintain/expand the small grants program for infrastructure development by • increasing the number of small grants available to CSs with continued district involvement, support and capacity building. • ensuring that individual grants are small enough to require a cost share, community prioritizing and careful planning, yet large enough to make a tangible impact. The current range of US$2,500 to US$5,000 has had good results. 2. Consider conditions under which a school or community might be eligible for a second grant, while maintaining the concept of leverage, e.g., investment in an income generating activity that could be used for regular allowances to teachers, and/or have the school operate for profit. 3. Actively disseminate the small grants model, encouraging NGOs and local partners to work with district officers in school construction and rehabilitation (rather than outside the system). Expanding Available Resources 1. Expand participation of private resources (non-government, non-donor) by district/provincial and national level personnel. 2. Districts should pursue regional businesses and NGOs, including private sector industries (tourism, mining, retail, telecommunications, agricultural/food processing, financial institutions) as well as small NGOs, church groups, and organizations such as Rotary and Lions Club. 7 8 3. The MoE, with appropriate technical support, should develop guidelines for private sector financing of education, and develop the skills among district staff to solicit and build partnerships at the district level. 4. The national or international levels should explore a partnership that would provide MP3 players and accessories at a deeply discounted price. Possible partners include manufactures, retailers and NGOs. Defining and Monitoring Quality 1. Develop quality standards for CSs based on the operating guidelines and existing MoE standards. 2. Establish a developmental approach to standards monitoring that includes assessment, problem solving and action planning as essential steps to promote school level quality improvement. 3. Build on the success of community empowerment in the grants and school/community partnership activities of CHANGES2 and QUESTT to involve communities in monitoring of non-infrastructure standards of quality. II. OVERVIEW This report presents the results of an independent assessment of targeted basic education activities funded by the U.S. Agency for International Development (USAID) in Zambia. The overall purpose of these activities was to strengthen the Ministry of Education (MoE) support to CSs. The assessment was carried out by a three-person team between March 10 and April 25, 2008 by DevTech Systems, Inc. The evaluation reviewed the activities and achievements of three USAID-supported programs - CHANGES2 Program, the EQUIP2 Zambia project and the QUESTT project - using special, one-time Fast Track Initiative2 (FTI) funding made available to accelerate the achievement of universal primary education. The report draws lessons learned (identifying achievements, promising practices and potential gaps) from the programs and proposes recommendations for systemic support to CSs within and beyond the three individual projects. The statement of work for the evaluation presented questions for each project in broad and cross-cutting thematic areas. This framework provided an opportunity to examine the work of each program at various institutional levels of the basic education delivery system. The methodology and the presentation of the evaluation results reflect these dimensions. The report is organized to discuss first the relevant background and context of USAID support to basic education in Zambia and its focus on CSs, as well as an overview of the three projects through which funds were disbursed to “fast track” or enhance support to basic education. It describes the methodology and analytic approach that were designed to document lessons at the various institutional levels within which the three projects work. Findings and discussions of achievements, challenges, promising practices and gaps in support to CSs are presented in the body of the report by institutional level: national, provincial, district/zones, school/community, and classroom. Finally, the report presents a summary of recommendations for continued support to CSs. It is the hope of the evaluation team that this presentation of results will appeal to a broad audience, and will help to define ways to strengthen partnerships with the MoE to strengthen support CSs. 2 Although this funding was called “Fast Track Initiative” funding, it is not part of the Fast Track Initiative led by the Education for All Secretariat. 9 III. BACKGROUND AND CONTEXT: USAID SUPPORT TO COMMUNITY SCHOOLS Background Community schools (CSs) are a major provider of basic education in Zambia. Today, there are over 2,500 CSs in Zambia delivering education to approximately 500,000 students and comprising nearly 20 percent of the total basic education enrolment in Zambia. CSs were selected to benefit from FTI funding due to the convergence of the following four factors: • the collapse of the Zambia Community Schools Secretariat (ZCSS); • the ongoing desire of MoE officials, particularly at the district level, to recognize CSs as an integral part of the basic education delivery system; • the availability of supplemental, one-time FTI funding provided to selected USAID Missions to support basic education; and • the presence of three well-established USAID-funded projects with the capacity to rapidly expand and adapt their activities in the service of CSs. This section includes a brief historical overview of CSs and the ZCSS; a description of the MoE interest in CSs; a brief statement about the USAID initiative and funding through FTI to CSs; and an overview of the three projects assessed during this evaluation. Historical Overview of Community Schools The growth of CSs and their resulting contribution to access and equity in basic education has drawn increasing attention from non-governmental organizations (NGOs), the government and cooperating partners over the past several years. A brief discussion of this historical growth creates a shared context for the remainder of this study. CSs emerged in Zambia in the early to mid-1990s. Established through community initiatives and often supported by NGOs or faith-based organizations (FBOs), these schools filled a critical gap in the provision of basic education. They served pupils who were unable to access government schools because of distance, school fees and associated costs; and/or the age restrictions of government schools. These schools use a specialized curriculum called Skills, Participation, Access to Relevant Knowledge (SPARK), which compresses the basic curriculum and includes practical skills to enable overage pupils to join the economy. The first CS opened in 1992. By 1996 there were 55 CSs in the country (MoE Strategic Framework for Community Schools, December 2006). The GRZ recognized that CSs provided critical basic education opportunities for disadvantaged children and sought ways to complement community initiatives with government resources, including grants, teaching and learning materials and appointment of qualified teachers. CSs continued to grow exponentially, from an 10 estimated 833 registered in 2,000 to 2456 in 2006 (MoE Strategic Framework for Community Schools, December 2006). While this number represents schools that are officially counted, the actual number may exceed 3,000 schools with a total enrollment of more than 500,000 pupils, accounting for over 30 percent of the total number of schools offering basic education in Zambia. The MoE acknowledges that CSs are critical in providing school access particularly for marginalized groups. Moreover, the National Implementation Framework (NIF) commits to developing a flexible and inclusive education system that provides mechanisms for increasing equitable access to quality basic education for all (p. 32). Collapse of the Zambia Community Schools Secretariat In 1997, the Zambia Community School Secretariat (ZCSS) was established with support from UNICEF and other non-governmental partners to support CSs. The ZCSS became the sole established organization for advocacy, coordination, material support and facilitation to CSs. The ZCSS worked closely with the MoE to develop the Policy Guidelines for the Establishment of Community Schools, chairing a multi-stakeholder Task Force, whose report was published in 2001. A Memorandum of Agreement between the ZCSS and the MoE was signed in 2001, and revisited in 2005, positioning the ZCSS as the national umbrella organization dedicated to the development of the CSs. The memorandum formalized the arrangements through which a community or nongovernmental school initiative would be complemented by government endorsement, support, and resources. CSs registered with the ZCSS to access grants, teaching and learning materials, training opportunities for volunteer teachers and government-paid teachers. In effect, this rendered the ZCSS a parallel structure for coordination, management and resourcing of CSs. Under the Basic Education Sub-Sector Improvement Program (BESSIP), CSs were eligible for 30 percent of grant funding made available to districts. Up to 20 percent of the grants could be used to pay teacher allowances. In the 2005 updated Memorandum of Agreement, the MoE committed to ensuring the overall development of CSs on an equitable basis. At the national level, ZCSS was relatively successful in its advocacy and coordination roles. Its organizational structure, capacity and resources, however, were unable to respond to the increasing numbers and needs of the CSs. In spite of a renewed strategic focus on its organizational development and plans to create representative structures at provincial, district and sub-district levels, the ZCSS administrative challenges rendered the Secretariat defunct in 2006. 11 Current Status of Community Schools Decentralized Support to Community Schools Despite the collapse of the ZCSS, proactive provinces and districts continue to support CSs within the existing parameters. Southern and Central Provinces, for example, have successfully deployed trained and government-paid teachers to a large number of CSs. This arrangement provides professional strength to the schools, as well as makes them eligible to receive government grants through the District Education Board Secretary (DEBS) office. DEBS has also facilitated improvements in infrastructure and water supply at some CSs (See Box 1). USAID funding to “fast track” education support to community schools. At about the time that ZCSS efficacy was waning, USAID made supplemental funds available to selected missions to promote practices that would accelerate education support to increase equity for girls and vulnerable children. The purpose of the additional funds was to “achieve maximum and immediate results, increased equitable access to quality basic education through systematic inputs with sustained impacts.” The USAID/Zambia mission recognized an opportunity to assist the MoE in filling the void left by the collapse of the ZCSS by channeling “fast track” funding through the established projects in its basic education portfolio. Box 1: Hauma Community School: A Case Study in Decentralized Support to Community Schools In Gwembe district, Hauma Community School (HCS) was established in 2001 and today has permanent structures with boreholes. HCS is a microcosm of the many well￾established and government-supported community schools in Zambia. HCS was established because the nearest government school for children in Hauma Village was approximately 9 kilometers away. Today, HCS school, enrolls at age (vs. over age) school children and has exchanged its curriculum from one designed to accelerate older children through school (SPARK) for the national curriculum. HCS’s development has been consistently supported by the MoE and other donors. For example, HCS includes both government-hired and volunteer teachers, and is the recipient of MoE funds. Save the Children Norway supported efforts to build a classroom, install solar power and provide pre- and in-service teacher training. HCS was also a beneficiary of the MoE’s implementation of the Program for the Advancement of Girls Education (PAGE) program, which aimed to create conditions that encouraged the participation of girls in school, including the installation of a borehole and four pit￾latrines to improve sanitation and safety for girls. This support, targeted through districts, is illustrative of the decentralized support enjoyed by community schools. USAID’s Basic Education Portfolio and Role in Strengthening Landscape for Community Schools At the time of the ZCSS ‘s collapse, USAID support to the MoE included three projects designed to work at various institutional levels of education service delivery: 1) the Community Health and Nutrition, Gender and Education Support Program (CHANGES2), 2) the Educational Quality Improvement Project 2 (EQUIP2) and 3) the 12 Quality Education Services Through Technology (QUESTT). The three projects had the mandate, technical skills and established relationships with MoE structures to respond to the void in support to CSs and to the opportunity presented by the FTI funding. When focused on CSs, the projects enabled the MoE to move forward a critical CSs agenda, embracing policy, capacity building, and improved quality of teaching and learning, including materials, furniture and infrastructure. Each project is described briefly below. Overview of USAID Programs Supporting Basic Education The CHANGES2 Program and Community Schools The CHANGES2 program, implemented by the American Institutes for Research (AIR), is designed to strengthen the professional skills of basic education teachers, with a special focus on HIV/AIDS prevention and mitigation. The program also supports the expansion of the MoE School, Health and Nutrition (SHN) activities to new programmatic and geographical areas. CHANGES2 was initiated to empower pupils, teachers and community members to improve education, gender equity and health. The program works in all 38 districts of the Central, Copperbelt, Lusaka, and Southern provinces, reaching more than 1,200 CSs. Along the “line of rail”, these provinces have high HIV/AIDS prevalence rates, as well as high concentrations of school-aged children. The core program includes technical assistance to Zambia’s Ministries of Education, Health and Community Development and Social Services in the following program areas: a. HIV/AIDS Prevention and Mitigation; b. School Health and Nutrition; c. School–Community Partnership and Outreach; d. Teacher Education and Professional Development; and e. Scholarship for Orphans and Vulnerable Children (OVC). With the additional FTI funding, CHANGES2 assisted the MoE in strengthening teacher support mechanisms, such as standards monitoring and professional development, pedagogical processes and community-supported infrastructure at CSs. The operational strategy of CHANGES2 for CSs is built around the following outcomes: • Improved teaching-learning process through training of CSTs, formulation of CSTs’ training materials, provision of teachers’ tool kit of teaching materials, engaging CSTs in the MoE School Program of In-service for the Team (SPRINT) for on-going professional development, practical teaching in CSs for pre-service teachers to improve mentoring of untrained teachers, and a reduction in teacher/pupil ratio;. • Strengthened MoE system and structures for monitoring and supporting the quality of education in CSs: strengthening the role, skills and tools available to district education support teams and zonal education support teams for professional support to CSTs, including ongoing professional development; and, • Improved infrastructure and project management capacity of Parent Community Schools Committees (PCSCs), through a small grants program. 13 The EQUIP2 Program and Community Schools The overall goals of the EQUIP2 project, implemented by the Academy of Education Development (AED), are to cascade institutional capacity through all levels of the educational system, from the MoE headquarters to provincial offices to district offices to schools; to follow interventions from the center to the school and community levels; and to support policy implementation and improve information management in ways that increase the education system’s efficiency and effectiveness. The EQUIP2 project and staff are based within the MoE structure, and therefore are well-positioned to assist the MoE to take advantage of USAID supplemental funding to move the CSs agenda forward. This ensures that EQUIP2 achievements are part and parcel of MoE achievements. EQUIP2 strengthens MoE support to CSs by assisting the MoE to: • elevate attention to CSs by conducting a national stakeholder consultative process on CSs; • develop the Strategic Framework and Operating Guidelines for Community Schools; • convene a national dissemination workshop of the CS guidelines; • improve the quality of recordkeeping and continuous assessment at all schools, including CSs; • strengthen data collection and accuracy as it related to CSs; and • support institutional shifts to build a more inclusive basic education system. EQUIP2 advisors also work with the MoE to complete the School Health and Nutrition (SHN) Strategy 2008-2010 and conduct Teacher Health Days to promote voluntary counseling and testing among teachers and their families. The QUESTT Program and Community Schools The QUESTT program, implemented by the Education Development Center (EDC), has a natural affinity for CSs, as its flagship activity, the Interactive Radio Instruction (IRI) program, is designed to provide quality instruction delivered by radio to children who are unable to access government schools. The IRI relies on trained community-based volunteer mentors (uncertified teachers). The early program was established at IRI centers, many of which have expanded to become CSs. With the additional FTI funds, QUESTT scaled up the use of IRI in both government and CSs, to which it integrated the pre-service training program at the Colleges of Education. QUESTT also implemented a small grants program to build community capacity to support education improvement. The program included all 72 districts in Zambia, although infrastructure grants were provided only in the five provinces where CHANGES2 was not active. The project explored and piloted several initiatives to promote the sustainable development of CSs, including support to community radio 14 stations to produce education programs on the theme “Education For All”, sponsoring the Teachers in the Zambia Teacher Education Course (ZATEC) distance program through eight colleges of education, and piloting the use of MP3 players to deliver teacher professional development materials as well as IRI programs. Summary The complementary design of CHANGES2, EQUIP2 and QUESTT provided a unique opportunity to assess program support to CSs at each level along the chain of education delivery, from MoE Headquarters to provincial and district offices to communities and schools and finally to the classroom. The focus on institutional levels and specific lessons gathered from each project at the various levels are presented in subsequent sections. 15 IV. EVALUATION METHODOLOGY The overall purpose of the evaluation was to assess the degree to which USAID supplementary support to the MoE through the CHANGES2, EQUIP2, and QUESTT projects strengthened the ability of the MoE to support CSs at the various institutional levels of the education delivery system. The evaluation assesses program coverage, effectiveness and impact, documents lessons learned, and provides summary findings, conclusions and recommendations, including suggestions for continued support to CSs. Evaluation Approach The evaluation model was primarily qualitative, supported by the wealth of quantitative data collected by the Planning Directorate of the Ministry of Education and its USAID project partners. Most of the primary data collection came from observations, interviews and focus group discussions conducted at the provincial, district and school/community levels, with education officers, teachers, parents, students and community members. The evaluation results therefore reflect perspectives of individuals at multiple levels of the educational delivery system. Evaluation questions for each project, at each institutional level, were discussed and vetted with project staff and the USAID mission. Project staff also was involved in the identification of field sites and activities that could be accessed during the evaluation timeframe. Program staff was invited to observe interviews, focus groups and other field￾based activities and were also interviewed to gather information. Figure 1 illustrates the evaluation framework. 16 17 Evaluation Activities Document Review Team members reviewed documents that provided contextual information, including literature related to the historical context of CSs in Zambia; the structures and practices that supported their development; the MoE National Implementation Framework for the Fifth National Development Plan; and the recently released Operating Guidelines for Community Schools. Relevant plans, reports and materials produced by the three USAID/Zambia programs were also included. (See Annex VI for a list of documents reviewed.) Site Visits The team visited 17 schools in 11 districts of the Northwestern, Copperbelt, Central, Lusaka and Southern provinces where CHANGES2 or QUESTT had conducted teacher training and/or provided a small grant for infrastructure or income-generating activities. At each school, the team observed classroom teachers trained by the projects and/or untrained volunteer teachers; conducted interviews and/or focus group discussions with the school manager, parents/community members and pupils. Direct observation of the overall environment complemented the interviews. Additionally, the evaluation team visited the Mufulira College of Education, three District Resource Centers, one community radio station and one CHANGES2-supported teacher training program conducted by the District Resource Center Coordinators (DRCCs) of the Central Province. In each province and district, a courtesy call and detailed interview was conducted with Provincial Education Officers (PEOs) and at their discretion, additional officers familiar with the programs. A similar process was followed with District Education Board Secretaries (DEBS) and the district staff working most closely with the projects. Interviews and Focus Groups The largest pool of data came from interviews and focus groups conducted at the school and community levels with school managers, teachers, pupils and community representatives. The team interviewed MoE officers representing each of the five directorates in the national headquarters, as well as representatives of the NGO and donor communities, namely, the Zambia National Education Coalition (ZANEC), Zambia Open Community Schools (ZOCSs), Save the Children Norway and CARE International. The team also interviewed the Education Attaché and Program Officer of the Netherlands Embassy as a lead cooperating partner in the education sector. The team also conducted focus groups with community members and pupils who participated and benefited from small grants as well as other program activities. Interviews provided a chance for the team to learn about the contextual conditions, project successes and obstacles, views and experiences of stakeholders, and concrete examples of success that may not be found in project documents. Interviews with 18 personnel working at the multiple levels of the education delivery system enabled the team to identify patterns throughout the five provinces at various institutional levels, helping to shed light on some of the inherent structural issues in CSs in Zambia. Classroom Observations Classroom observations provided firsthand descriptions of how specific teaching and learning activities were carried out in the classroom following training. The Evaluation Team observed at least one teacher per school, which contributed to the overall assessment of quality at the classroom level, the use of skills acquired during training and teaching and learning materials provided to the school. Observations of Community Schools Teacher Training Observations of CSs teacher training at Kabwe provided direct experience with the training delivery model, as well as the opportunity to interview a group of DRCCs and teachers about the quality and application of the training. Data Analysis For each element of the overall framework of the evaluation, key data sources, interview items and/or observations were compiled to complete the presentation of program and institutional level findings described in the following sections. This analysis allowed the team to consider the perspectives of the multiple stakeholders of each project and institutional level. Limitation and challenges in the methodology Timing presented the major challenge in conducting the evaluation. The length of the 2008 rainy season and the end of the school term compressed field work and the selection of schools. The team was limited to 21 calendar days, including three public holidays, to visit schools before the end of school term. Project teams were tasked with developing a preliminary list of schools that would best illustrate the achievements and challenges of the schools, communities and MoE structures that they supported. This list was reduced to fit the available time, through a joint planning session with the three projects. During the planning session, the conditions of the roads, distances to the individual schools as well as the degree to which the schools were illustrative of project practices and achievements overall were considered in the final selection of schools. The potential bias in school selection was addressed, at least in part, by the finding that many schools that were recommended by one project had also participated in activities of another. It was not possible to quickly identify specific schools that had participated in EQUIP2 pilot activities in recordkeeping or continuous assessment. Questions were posed to school managers and DEBS staff regarding these pilots and two schools reported participating in them. Travel to the schools also highlighted the fact that road conditions and the distribution of settlements often meant traveling for 30 to 45 minutes 19 20 to cover as little as 10 to 15 kilometers from the town or the paved road to arrive at a very isolated setting. The intensity of the schedules of MoE headquarters’ staff presented a challenge to the team as well. While district and provincial staff were quite accessible to the team – even traveling with the team in several cases – it was difficult to secure time with headquarters staff. Though the team was able to meet the HIV/AIDS Advisor, they were not able to meet representatives of the Human Resources Management office. Additionally, the team had time to interview a finite number of cooperating partners, including international and national NGOs. The team is aware of the significant contribution and support provided by such organizations to community and infrastructure development, as well as in capacity building for CSTs and managers. Therefore, the team makes the assumption that the sample interviewed is representative of the many organizations that support CSs, including World Vision, World Food Program, Project Concern International, and several other USAID and non-USAID supported activities. Finally, the shift in focus to CSs and the limitations of the school calendar meant that the team did not adequately assess the impact of IRI in government schools. The team did visit one Government of Zambia (GRZ) school where IRI was practiced. Fortunately, the QUESTT program has substantial data comparing pupil performance in GRZ schools that do and do not use IRI. V. EVALUATION FINDINGS This section presents the overall findings of the FTI evaluation first by project- and institutional-level impact. The findings include achievements and challenges, promising practices and gaps identified. The institutional levels are national, provincial/district, community, school and classroom. The specific contributions of each USAID-funded program to these findings are attached as Annexes A-C. While the CHANGES2, EQUIP2 and QUESTT programs were the focus of this assessment, many schools also acknowledge similar and complementary support from other organizations, such as CARE International, Save the Children Norway and the Zambia Open Community Schools (ZOCS). CARE International promotes CSs as comprehensive centers of care, providing education, health and HIV/AIDS-related services. Save the Children Norway has provided support to infrastructure and non￾infrastructure quality improvements in schools, including support for the ZATEC by distance in the Southern Province. The ZOCS supports both comprehensive care and quality improvements, through a human rights advocacy approach to the delivery of education services to the underserved. The practices of these organizations also have informed our findings and recommendations. Finally, the decentralization process at the MoE generated promising practices at the field level that are also discussed in this section. USAID Program Level Results The scale of USAID support to CSs - the major focus of the FTI funding - was significant. EQUIP2’s assistance to the MoE to develop policies and guidelines provided a framework for delivering support to CSs, as did the pilot programs in continuous assessment and recordkeeping. CHANGES2 and QUESTT provided school level support and promoted quality through teacher development and small grants for infrastructure. Table 1 summarizes the major school level activities of all projects. 21 Table 1: Summary Results of FTI Programming by Activity (See Annexes A-C for complete program evaluations). Activity CHANGES2 EQUIP2 QUESTT Total Creating an enabling policy environment for support to CSs Strategic Framework; Implementation Guidelines; Legislation; SHN Strategy 4 major policy documents disseminated Teacher Development: Community School Teacher Training in Basic and Innovative Practices and Methods (short professional courses) 20 days Basic Skills Training: 948 CSTs 5 days Continuous Assessment 180 CSTs 3 days IRI 893 CSTs in 9 provinces 2,021 CSTs trained GRZ teacher training in IRI 3 days IRI 3,091 in 9 provinces 3,091 GRZ teachers trained Formal Teacher Training: Full time Volunteer CSTs enrolled in Zambia Teacher Education Course (ZATEC) 458 Volunteer CSTs with Grade 12 458 CSTs Community School Managers Training (short professional courses 5 days School Management: 948 CS Managers in 4 provinces 5 days Recordkeeping: 830 CS Managers in 7 Provinces 1,778 CS managers Small Infrastructure Grants 163 grants to CSs in 4 provinces. 158 grants to CSs in 5 provinces 321 Grants to CSs Teaching/Learning Materials Provided 948 Teachers’ Toolkits provided to CSs; SHN; Project Management; Teachers’ Basic Skills Training Manual; Grade 1-7 Syllabus; Head teachers Handbook; SPRINT Manual Modules, activity books, sample forms and tests. Mentors guides, radios, wall charts, MP3 players, bicycles for monitoring within a zone. 22 Table 1 summarizes the activities and results of the interventions implemented by the three USAID programs. Together, the activities covered all nine provinces in Zambia and all institutional levels of the education service delivery structures. As Table 1 demonstrates, EQUIP2 assisted the MoE to produce four policy frameworks. Key among these, are the implementation guidelines which have been developed through a highly participatory, consensus-building process. As a result, CSs are now included in the MoE’s Annual Work Plan and budget, and they are represented in national level pilot activities. All three programs trained a large number of both GRZ and community school teachers. EQUIP2’s support to a pilot program of the MoE in Recordkeeping for School Managers included 830 community school managers. The EQUIP2 program, through the Examination Council of Zambia (ECZ) implemented pilot continuous assessment pilot also reached 180 community school managers. CHANGES2 provided training in basic teaching skills to 948 CSTs and training in the fundamentals of school management was given to 948 CSTs. CHANGES2 and QUESTT together provided 321 grants to school/community committees. The committees have also been trained and supported to work together to complete school infrastructure. Altogether, the three programs trained approximately 3,800 CSTs. QUESTT also extended IRI to an additional 3,091 GRZ teachers through zonal in￾service training coordinators. QUESTT also enrolled 458 CSTs in ZATEC as distance students in eight Colleges of Education. Tools, skills and approaches to monitoring and support to CSs have been practiced by provincial, district and zonal level officers of the MoE, including those responsible for planning, building, procurement and educational standards. The policy guidelines and strategies have created coherence and consistency in approaches to coordination, management and resourcing of CSs. Efforts to improve and standardize approaches to recordkeeping for use at all levels have improved the management of education services. Critical innovations in continuous assessment have provided better quality and equity, as pupil performance is reviewed over time, not just on the day of the national examination. EQUIP2 has worked diligently in these areas, and has conscientiously included CSs in its pilot efforts in Continuous Assessment (CA) and recordkeeping. Teacher development, infrastructure, and teaching and learning materials are the most frequently mentioned concerns of education managers throughout the system in terms of challenges to supporting CSs. In addressing the lack of teaching materials, the CHANGES2 program provided tool kits to all 948 CSs that were targeted. Overall, the USAID program has succeeded in addressing many concerns albeit in a preliminary way. For provincial and district officials who have provided assistance, however, the MoE’s efforts to clarify support mechanisms has legitimized their initial efforts to channel resources to CSs. 23 MoE Institutional Level Results National Level The MoE at all levels has embraced CSs as full partners in the provision of basic education services. An overwhelming majority of provincial and district staff agreed that CSs made a significant contribution to the provision of basic education. The USAID support to CSs has contributed to several of the broad objectives of the Fifth National Development Plan, as reflected in the Education Sector National Implementation Framework (p.19) by: • strengthening institutional frameworks to coordinate provision of education; • developing policy changes reflective of the current educational requirements; • promoting alternative modes of education provision; and • promoting innovative methodologies in learning institutions. Strengthening institutional frameworks to coordinate provision of education The process of completing the Strategic Framework (MoE, December 2006) and the Operating Guidelines for Community Schools (MoE, November 2007) was as important a contribution to institutional strengthening as the products themselves. EQUIP2 provided support, including technical assistance and cost sharing to these efforts. The broad-based consultative process demonstrated a willingness on the part of MoE to learn from CS representatives, district and provincial level officers, and the experiences of national and international organizations supporting CSs. The roles of government and civil society, as well as external organizations are well described within the Operating Guidelines. Policy dialogue between EQUIP2 and relevant Ministry offices resulted in the inclusion of CSs in the National Implementation Framework (NIF) and resulting annual work plans and budgets. Prior to the completion of the Operating Guidelines, CSs were eligible for government support, but resources had not been specifically earmarked. This highly visible statement of support was made possible by the analytical work of EQUIP2, based on increasingly accurate and complete information about CSs. Policy changes reflective of current education requirements EQUIP2 contributed directly to this objective, by assisting the MoE to develop legislation, a strategic framework and operating guidelines. These policy documents acknowledge the significant role that CSs play in the provision of basic education and spell out procedures and requirements for accessing government support. The guidelines are comprehensive, for the first time addressing the roles and functions of government and civil society in the promotion of quality in CSs. The guidelines provide specific mechanisms for the development and deployment of teachers; for resource 24 allocation; and for standards monitoring for quality of learner performance, as well as infrastructure. While there may be a role for a national level organization to advocate for and to monitor progress in implementing the guidelines for CSs, there does not seem to be a significant need at this time for an NGO to “replace” ZCSS, as districts are now responsible for registration, coordination and support to CSs. Stakeholders generally applaud the formal recognition given to CSs as providers of critical services to children who would otherwise lack access to basic education. In some quarters, however, there is a lingering sense of caution, wanting to make sure that community initiative is retained and that the expectations of the government are not too high. Alternative modes of teacher education promoted CSs, due to the challenges they face regarding the quality of teachers, have helped generate, two practical alternative models of teacher qualification that utilize the Zambia Teacher Education Course (ZATEC). They have also been the beneficiaries of extensive professional development of teachers through short courses in basic teaching skills. Combining the short courses with the formal qualification course may be a next step in promoting the quality of teachers in CSs. The first alternative model is the development of a distance education program for qualified CSTs. QUESTT engaged staff members of all ten Colleges of Education to develop distance modules for ZATEC assignments, contact sessions and monitoring guides and to sponsor 458 students/CSTs in the program. The students and college lecturers, who were interviewed, reported that contact time was inadequate. This shortcoming can be attributed to lack of experience among the lecturers in delivering by distance and to the decision to structure contact time so that teachers are not taken out of the classroom. The second alternative is the assignment of campus-based ZATEC students to CSs for their teaching practice. CHANGES2 provided funds to the ten colleges of education to enable lecturers to monitor 189 students conducting teaching practice in CSs. This monitoring exercise showed that the ZATEC students appreciated practicing in the CSs. Two teachers interviewed by the team shared this sentiment but were also concerned that the CS to which they were posted did not have housing for teachers nor text books and other teaching and learning materials. College lecturers also reported finding it difficult to monitor students posted to CSs, citing remoteness, and lack of both transport and financial compensation as key constraints. While the ZATEC program is available to qualifying Grade 12s, there are many more CSTs whose Grade 12 certificate is not strong enough to merit a place in a College of Education or who have completed only Grade 9. These CSTs have participated in short courses in basic teaching skills offered by CHANGES2, QUESTT and other NGOs such as CARE International. Their practical experiences applying their training in the 25 classroom, documented through monitoring and support visits provide evidence of the knowledge, skills and professional attitudes they demonstrate. These can be linked to a formal qualification through the development of a competency framework which describes comprehensively the knowledge, skills and attitudes required of a basic education teacher. The skills and practices promoted in existing short courses provide a strong starting point for a competency framework, as do the teaching skills, attitudes and knowledge embedded in the outcomes based curriculum. A competency framework work would enable the development of new and complementary short courses, avoiding duplication of effort and unnecessary redundancies in content and focus. Figure 2 below describes a pathway to formal qualification that links current status of CSTs (Grade 9, Grade 12 not eligible for CE, Grade 12 eligible for CE) and specific in￾service programs, to entry to ZATEC. It considers the teaching experiences and short courses of Grade 9 and Grade 12 CSTs and suggests that upon successful demonstration of core competencies, they be granted entry to ZATEC (preferably by distance) at Colleges of Education. CSTs with a grade nine level of education can enroll in a distance course to complete grade twelve. While continuing to teach, this group of teachers also participate in short courses designed to improve basic teaching skills and can build a portfolio of lessons and reflections that are linked to the competency framework and can be used to gain entry to a formal qualifications program. Standards officers and DRCCs would not only validate the portfolio process and content but also provide professional mentoring needed to improve the quality of teaching. The strength of the in-service training network of zonal and district level resources is an asset to a competency based model for teacher development. These individuals currently conduct in-service training for CHANGES2 and QUESTT and are the front line of support for school based professional development of all teachers and especially CSTs. 26 Figure 2: Competency Based Pathway to Teacher Qualification Community Support CSTs who have completed Grade 12 may or may not meet the basic entry requirements for Colleges of Education. For those who do not, the competency framework would enable them to receive credits for preferred entry based on their teaching experience and short courses that they would attend. The evaluation team also learned that many CSTs who have completed Grade 12 and meet the entry standards do not always know how, when and where to apply for admission to college. Access to such information can be provided during short courses, as part of mentoring and support provided by district and zonal teams and at the DRCCs.. The competency based model for the development and qualification of volunteer CSTs is an alternative method of providing teacher education. Though not a current practice, it is discussed here because it is well within reach. Teaching competencies are implicit in the existing professional development packages that are being extended to CSTs by USAID and other partners and the strength of the MoE in-service training and support network at the district and zonal levels is an available asset in implementing a competency based model. Their support is valued by CSTs and in turn, the receptivity of CSTs to this support provides motivation and support to the district and zonal teams. A final note on teacher development: Community members, pupils, parents and teachers themselves note that teacher training, through short courses or qualification programs, influences community perceptions of quality. Community volunteer teachers acquire 27 confidence as well as knowledge and skills; pupils respond well to the improved teaching; and community members take pride in these teachers. Community investment in continued support to volunteer teachers is essential until they can be placed on government establishment. The community support dimension in the Figure 2 reflects this fundamental relationship. Innovative methodologies in learning institutions promoted IRI is an innovative methodology. It was enhanced by the introduction of MP3 players where radio reception is poor. QUESTT expanded the materials on the MP3 players to include lesson-specific teacher resources for Grade 6 teachers in the areas of English, Mathematics and Science. In subsequent expansion, the area of primary literacy has been targeted in the development of video resources for teachers. Peace Corps Volunteers often assist teachers in training and trouble-shooting on the MP3 players. The EQUIP2 program developed an internet-based resource, the Global Learning Portal (GLP), that shares information on education issues in Zambia and provides professional resources for educators. Connectivity and access to the GLP provides resources and online courses available for professional development and networking at Colleges of Education on the World Wide Web. The Continuous Assessment (CA) Research Pilot, designed to determine the extent to which CA improves pupil performance, is an innovation that follows the introduction of the outcomes-based curriculum. The pilot has developed a series of tools to monitor and record pupil achievement of learning outcomes. Teachers in nine CSs have been trained as part of the pilot and another 60 schools participate on a less formal basis. Promising practices The achievements described above at the national level and the contributions of USAID’s programs to the objectives of the MoE National Implementation Framework reflect several promising practices, including: • broad-based consultation to develop the Operating Guidelines. The process captured the prevailing practices and interests of multiple stakeholders and can also be used to disseminate the guidelines and to develop consensus among stakeholders on other critical issues at the policy level, including the proposed competency framework for teacher development; • staff of USAID-funded programs work with and through MoE structures to strengthen capacity as well as develop and deliver technical policies and interventions. This practice at all levels has resulted in the identification of issues on the ground that need to be addressed at national level as well as creating a network through which national initiatives can be implemented and reinforced; • linkage of in-service training with structured and related monitoring and support to apply learning on the job; • introduction of communication technologies at schools and colleges provides access to more resources than traditionally are available in print; and, 28 • use of distance education to train CSTs for certification addresses equity of access issues for rural teachers and maintains the classrooms being served by the CSTs while they are doing the training. It also extends the period of time that CSTs remain in their posts. Ongoing challenges Great strides have been made in defining the relationship between CSs and government. At the national level, several factors will need to be considered as the implementation of the Operating Guidelines moves forward. These include, but are not limited to: • Maintaining the essence of CSs. As the Guidelines are extended out to the school and community level is will be important that community ownership is maintained. The pace and extent of government support may not meet the expectations created by the Guidelines. On the other hand, in some settings there may be a fear of government takeover and a return to the conditions that gave birth to CSs in the first place; • Developing an approach to tracking the success and placement of ZATEC graduates through the distance learning program. This should include and assessment of the skills and experiences of college lecturers in distance education; the balance of entry level academic skills and teaching experience of the students; and the structure and duration of the contact sessions. • Building a coherent system for professional development, qualification and certification of teachers and school managers. The provision of short courses and ZATEC sponsorship, and related monitoring and support, have shed light on several issues that affect professional development for CSTs and GRZ teachers. These include: the need to: o integrate system for professional development of teachers and school managers that would link short courses, formal programs, academic qualifications and teaching experience; o clarify the roles and relationships between national and district level institutions for professional development and qualification of teachers and school managers; o explore attitudes of pre- and in-service training providers toward CSs and sensitize them to the new guidelines and the implications for their work.; and o standardize approaches, skills and tools for monitoring and support of schools that is focused not only on compliance but also on improving quality. The approach would include attention to non-infrastructure aspects of quality and include observation, feedback and action planning for improved performance. 29 The earlier discussion on a teacher competency framework and pathway to qualification provides an approach to addressing these concerns in an integrated manner. • Strengthening management of CSs. CHANGES2 has provided a short course that provides an overview of school management, or as one school manager noted, “taught us what happens in a school.” EQUIP 2’s experience in training CS managers in record keeping has led to the development of a formal, college based program for head teachers (GRZ and CS). The program is being designed to ensure that teachers who are promoted to head teacher positions acquire the necessary leadership, management and administrative skills to fulfill their new responsibilities. This program should be made available to qualified, trained GRZ teachers who are seconded to CSs as “teachers in charge”. In those provinces where there is a concerted effort to place at least one such teacher at each CS, this will provide a tremendous boost to the management of the schools, and will provide some motivation and recognition to the seconded teachers. • Two issues will remain with respect to CS management. The first is how to provide skills upgrading to school managers who are not qualified teachers rather are community leaders, retired professionals and/or those in the forefront of establishing the school They have responded positively to the CHANGES2 training and are likely to continue to benefit from short courses offered by similar school focused programs; they could also benefit from an expanded offering of short courses; and may or may not be qualified, eligible or interested in the more formal program. The second is how “teacher in charge” of a CS translates into a position on the career ladder of a primary school teacher. • Training and qualification of school managers might also follow a pathway similar to that for teachers, i.e., develop a competency framework of critical knowledge, skills and attitudes required of an effective community school manager (or basic school manager). Link the certificate course that EQUIP 2 is developing as well as short courses to these competencies. Create pathways for qualification for interested school managers who may not be qualified teachers, but who have strong leadership and managerial skills and who want to pursue a certificate in school management. Provincial/District Levels MoE decentralized roles and responsibilities are very well established at the provincial level. Provincial Education Offices (PEOs) are able to articulate the significance of CSs in the provision of basic education and share a vision of equity in government support to all providers of basic education. At the same time, they realize that districts are best positioned to coordinate CSs and channel government resources to them. PEOs, which monitor all types of schools, are beginning to include monitoring visits to CSs in their programs. Provinces have been disbursing grant funds and teaching and learning materials to districts for allocation to schools, including CSs. 30 The districts have been the frontline of support to CSs although the assistance given has been uneven across the country. Pro-active districts have overcome barriers to the provision of teachers and small grants. These districts have provided teachers, grant funds for infrastructure, and learning materials in practical ways that include: • seconding teachers from nearby GRZ schools; • sharing grant funds with CSs by direct disbursement to GRZ-salaried teachers or through direct payment to select vendors willing to provide goods to CSs while billing the district; and, • sharing available teaching and learning materials with CSs. All three of the USAID programs have supported the improvement in the focus, quality and frequency of provincial and district level support to CSs. The work done by the programs has expanded support to CSs. The work with various MoE structures (ZICs, DRCCs and Colleges) has built relationships between communities and government bureaucratic structures. Efforts to institutionalize support to CSs have also included ensuring the availability of reliable data. EQUIP2 recently collaborated with UNICEF and provincial/district teams to conduct a data verification exercise for school census data. In addition to the data verification, the teams were able to collect Global Positioning System (GPS) coordinates for CSs. CHANGES2 and QUESTT have supported monitoring visits and capacity-building of district staff focused on teaching, school management and community development of infrastructure. Their interventions were delivered through the established structure of District Resource Center Coordinators, and Zonal In-Service Coordinators (ZICs). DRCCs reported that their own training skills had improved with time and practice. They also note that the interest and response of CSTs to training also boosts their morale and performance. CSTs trained by DRCCs feel freer to use DRCs, when they have time and transportation. DRCCs and ZICs were also participated in the MP3 player training to ensure that there was district capacity in their use and to encourage the use of teacher training videos beyond the community of MP3-equipped schools. During the field visits, the team stopped unexpectedly at a DRC. Though the DRCC was away, his assistant provided an excellent impromptu demonstration of the MP3 and its contents. Monitoring visits assist teachers to apply their training. For classroom level practice, the monitoring approach included a pre-observation conference and review of lesson plan, classroom observation based on a tool linked to expected outcomes of training, feedback to the teacher, and joint planning of actions to improve the teaching. DRCCs report that they enjoy working with CSTs, as they are eager to learn and receptive to advice and support. EQUIP2 recently completed the Strategic Framework for School Health and Nutrition, using FTI funds. The CHANGES2 program operates within this strategic framework, taking interventions to the school and community level, engaging district level health and community development workers in training and monitoring of schools. In addition to 31 joint monitoring visits for academic work and health promotion, officers for building, procurement and planning officers participated in the identification, selection, training, and provision of technical advice to CHANGES2 and QUESTT sub-grantees. Their involvement was well within the mandate of the established district sub-grant committees, with the added benefit of being able to participate in the training and support of community-based project teams and Parent Community School Committees (PCSCs). The small grants program and processes, described below in the section on “School and Community Level,” has built confidence between CSs and district staff. This confidence and collaboration, as well as the training and insights gained by the district officers, will facilitate the effective implementation of the government grant funds that EQUIP2 has helped secure as an earmark for CSs. Promising practices at the Provincial and District Levels Promising practices at the Provincial and District Levels include: • inclusion of CSs in routine monitoring and promoting community confidence in government structures to support their schools without taking over; • delivering education support services through the existing structures and thereby strengthening these structures. For example, when the DRCCs and ZICs work as in-service trainers they establish support and mentoring relationships with teachers, and, at the same time, the receptivity of the CSTs boosts the morale of the district staff. • actively engaging district officers in the small grants process which promotes the coordination and support role of district staff helps ensure quality of infrastructure and reasonable pricing, and builds community confidence in district structures. Working with the Teacher Education and Standards directorates, CHANGES2 and QUESTT have developed monitoring and support models designed to promote a shift from “inspection” to “support”, emphasizing professional development and improved quality rather than compliance. In the “inspection” model, as Figure 3 below shows, the focus is on compliance and documentation for use at district, provincial or national levels. The “inspection” may or may not include classroom observation, and the report is rarely discussed with school or community leadership. While this may be useful for planning purposes it does not necessarily help the school and community focus their attention on improvements that are well within their means to achieve. Figure 3: “Inspection” Process by Officer CHANGES2 and QUESTT both utilized standardized tools for documenting monitoring and support visits to teachers as a follow up to training activities. CHANGES2 trained district and zonal teams in a paradigm shifting approach to providing school level 32 support. The tools developed for monitoring and support were linked to the national level tools, though focused on specific aspects of the school that were most affected by the interventions, notably school management, classroom practices and environment and hygiene. According to project reports, officers valued the team approach as it enabled on the spot problem solving. District staff interviewed by the evaluation team further noted that the receptivity of CSs to monitoring and support visits was mutually motivating. The support model, as Figure 4 demonstrates, focuses on school level improvements based on how a school is doing against a set of established standards, including but not limited to management, teaching and learning, community relations, hygiene and environment or development of infrastructure. The observations are shared and discussed with management, teachers, community and joint problem solving and action planning result in concrete ways to improve the quality of the school. The focus is on school level problem solving as a first course of action, with larger issues referred appropriately to higher levels. Figure 4: Support for Professional Development Process by Officer The practice of sharing observations at the school level and joint problem solving has the potential for school led improvements in these key areas. Integrating the tools and approach developed by CHANGES2 into the MoE standard monitoring forms needs to occur at the national level through the Directorate of Standards. The placement of USAID’s support has the potential to leverage the development of quality standards, staff skills and attitudes as well as tools that will focus on actions that can be taken at the school and community level as well as higher levels to improve the quality of teaching and learning. Outstanding challenges Continuing challenges at the Provincial and District Levels include: • the time available to DRCC staff and ZICs for support to CSTs may be insufficient to meet increasing demand from teacher development projects providing short in-service courses as support to CSs grows. At the same time, although the USAID-supported projects involve Colleges of Education, there is sometimes a lack of coordination between pre- and in-service teacher training; • transportation, the condition of the roads, and funding for allowances constrain monitoring and support visits. The number of CSs, their remote locations, and the limited transportation available in the districts are serious constraints to reliable support. The MoE has recently assigned vehicles to certain districts, but this is not likely to meet the increased demand; 33 • the need for a performance or quality improvement approach to school monitoring that would benefit CSs and is a natural extension of outcomes￾based learning. Monitoring standards can be based on standards (outcomes),which can be reviewed on a regular basis at school, zonal or district levels; • the need for continuous assessment to provide measures of pupil performance over time. CA data must be analyzed at the classroom and school levels to identify the causes of performance achievements and gaps. CA tools should be widely shared and discussed with everyone involved in implementing the new curriculum in the schools, including programs of USAID and other cooperating partners, international and national NGOs, and the teacher and school development structures of government; • Although the Guidelines squarely place monitoring of the CSs in the hands of provinces and districts, not all officers feel that CSs are an explicit part of their described duties. This resistance is in part due to the legal status of CSs and to the traditional expectation that extra duties would bring additional allowances; • government-funded teachers who are seconded to CSs as “teachers in charge” take up the duties of a school manager/head teacher but the title of “teacher in charge” provides no direct benefit in terms of pay or promotion; • Constraints on government resources may affect equity, including the number of staff available to support the quality of teaching and learning in CSs. Proactive districts have established approaches to sharing resources with CSs but others, in the absence of funding formulas, are less bold in their allocation. Community and School Levels Community Level The CHANGES2 program operated in 32 districts, affecting 948 CSs and related communities, while the QUESTT program operated in all 72 districts, and targeted approximately 1,000 CSs and communities. EQUIP2, operating at the headquarters level had less direct involvement at these levels, although they exerted substantial influence through their policy role. Training of teachers and school managers had an impact at the community level in terms of perceptions of quality but the most direct impact was through the small grants program implemented by CHANGES2 and QUESTT. CHANGES2 awarded grants in its four provinces of operation (Copperbelt, Central, Lusaka and Southern), and QUESTT awarded infrastructure grants in the remaining five provinces (Western, Eastern, Luapula, Northern and Northwestern). The overall approach of both projects was the same, building on district processes and structures, as well as the previous CHANGES2 efforts. 34 The small grants programs have proved to be a very successful tool in empowering of communities to improve infrastructure. Permanent structures created a sense of stability about the school. The grants programs of both QUESTT and CHANGES2 supporting a total of 321 schools, built capacity through training and implementation support. School managers, pupils, teachers and community members report confidence in themselves and each other as a result of the implementation of the grant. They also report that they gained skills in budgeting, budget management, procurement, financial accountability and transparency, and inventory management. Some schools have utilized this experience to apply for grants from other sources, including the U.S. Embassy Self Help Grants, and local NGOs such as the Firelight Foundation. In all cases, the grants were used effectively with well developed budgets and procurement plans. District Building and Procurement Officers provided support in terms of quality and costs of construction, based on the MoE standard designs and pricing. For example in Northwestern Province, District MoE Procurement Officers taught the community members how to acquire and compare quotations from different suppliers. Before the purchasing was done, the Procurement Officers went physically to the shops shown on the quotations to verify the pricing and to ensure good accountability by community members. These interactions make MoE bureaucracy at the district level accessible to ordinary citizens, which in turn strengthen decentralization. It was also reported that ZICS used the bicycles provided by QUESTT to monitor schools and provide program support. Community members also praised the value of IRI as an interesting way to learn, promoting English, listening skills and literacy. Some respondents suggested using the program for adult literacy, and a prison in Zambezi is now using it for that purpose. A visit to Maranatha Community Radio Station gave a good indication that community radio stations are interested in broadcasting Learning at Toanga Market, as well as other education programming. The coverage area of Radio Maranatha is an 80 km radius from its Kabwe station. While worth exploring, especially in Central and Southern Provinces, where reception of ZNBC is poor, community radio stations may not be a universally viable option. The signal from community radio stations reaches no further than 100 km and sometimes less. Improved reception is provided within the locality of the station but the stations do not cover all the districts of Zambia. Additionally, even the most committed stations, such as Chikuni Radio Station, will not commit three and half hours per day every day for all the IRI programmes, and with the present power outages, some community radio stations find it difficult to keep to schedule. In terms of sustainability, QUESTT is working to ensure that the IRI programmes continue after USAID ceases its support and the project ends. Currently, the MoE pays ZNBC for airtime but can only just find sufficient funds to pay ZNBC and will not be able to find the funds to pay community radio stations to broadcast the programmes. At the moment 12 community radio stations are broadcasting lessons for two or three grades as part of their service to the communities. These challenges will need to be addressed to increase the viability of community radio stations as a learning modality in CSs. 35 School Level At the school level, teachers and school managers reported improved morale due to their participation in the CHANGES2 trainings. They felt recognized and confident, as well as more professional as they applied their new skills at school. School managers valued the CHANGES2 training in leadership, management of finances and records; monitoring the academic work of the school; and community relations, as well as specialized topics such as life skills and School, Health, and Nutrition (SHN). Through the training, school managers came to understand their leadership roles in the overall management of the school as well as in the academic work of the school (e.g., checking prep books). Therefore there were no conflicts of roles or overlapping of responsibilities. School managers related well with the communities and also executed their duties well in administration. One manager reported that he now knew “what must take place at school.” Improved management of the academic work of the school, when coupled with improved planning and lesson delivery, improved pupil and parent responses. Another finding was the contribution of the ZATEC program to teacher retention in CSs. The pre-condition attached to sponsorship of a CST for the ZATEC program was that the teacher should have served as a volunteer teacher for at least one year, and would continue working in the field during the two years of the ZATEC by distance program. The CSTs only went to the ZATEC residential school when the CSs were closed. CSTs also agreed to remain at the school for at least one year following completion of the program. These conditions mean that a teacher must remain at particular CS for at least four years in order to complete the ZATEC program. School managers and teachers also reported positive changes due to improved teaching and IRI programs. For example, teachers appreciated IRI because it was planned, organized, and interactive, and the children enjoyed it. Teachers report that IRI sharpens children’s listening skills and requires them to think faster to respond to the radio teacher. They also report improved writing and speaking skills in English. While some teachers felt that “IRI meant that you didn’t need to prepare a lesson”, most felt that IRI did in fact require and prompt good preparation. Perceptions of Quality Perceptions of quality at the school and community levels, as Figure 5 shows, have been enhanced by improvements in infrastructure, provision of furniture and teaching and learning materials, and training of teachers, managers and community members. Infrastructure improvements create a sense of achievement and stability among community members and committees. Training for teachers, school managers, PCSCs and other community representatives increase skills, motivation and confidence that radiates to the pupils. 36 Figure 5: Web of Community Perceptions of Quality Pupils report that among the things they like best at their schools are the classrooms, desks and good teaching. Good teaching encompasses practices such as IRI as well as appreciation for the volunteerism of the teachers. As members of one focus group noted, “Our teachers do not go on strike.” Teachers who had participated in in-service programs were proud of what they had to offer pupils. They noted, “We know how to plan a lesson and use time; before we were cheating the pupils by just giving them a bit of work.” All schools visited reported increased enrollment and attendance over the past year. They attribute this growth to perceptions of increased quality. Parents who feel that the CS is on a par with the nearest government school enroll their children in CSs even if they can afford the cost associated with Basic Schools. Promising Practices One of the promising practices of school quality improvement was in capacity building and production through small grants for infrastructure development and income generation. This evaluation found that the small grants programs of CHANGES2 and QUESTT provided a model that built community capacity as well as infrastructure development. Onsite support as well as training at each step of project design and implementation promoted confidence, discipline, and transparency in the use of funds and cost sharing (community contributions). The size of the grants, which at the time of the evaluation ranged from US$2,500 to US$5,000, seemed to provide enough support to complete an activity but not to fully fund it, which retained the community contribution and ownership. Schools receiving 37 both infrastructure and income generating grants could develop the means to increase their support to community volunteer teachers. For example, one CS used a maize field and a school garden for feeding and income generation programs. Community members and school managers report that training and timely support visits by project and district staff were critical to their success. Connecting CSs with one another was another promising practice of school quality improvement. Networking of CSs was of vital importance as it enabled schools to learn from one another. For example, QUESTT used a CS with a good classroom block design and grant as the training site for other schools. CHANGES2 enabled one school with a small surplus in their grant to “donate” the funds to another school that had completed infrastructure but still needed desks. These practices promote community pride and sense of possibility. Another promising practice is connecting communities to district resources. As mentioned earlier, CHANGES2 and QUESTT both engaged District Building and Procurement officers as community resources for design and budget planning. This practice retained community direction while positioning district staff as resource persons for the community. District officers were similarly involved in monitoring the application of training and the quality of teaching in the classroom. Basic skills training for teachers, training of School Managers and Community Members create a shared understanding of roles as well as specific skills for each position. This is a promising practice that should be continued. School Managers understand their jobs, including school and community partnerships; and project management/grants training promotes cooperation between the school manager and community, and results in completed infrastructure. Outstanding Challenges Working at community and school levels presents challenges. • In the Southern Province, the way in which IRI centers were defined made them ineligible for CHANGES2 grants, even though most of the original IRI centers have expanded to become CSs. • Parents and community members are involved in the development of the school and in personnel issues. They send their children to CSs, and keep them there, based on their concepts of quality education and they seem to respond well to new teaching practices. They are not yet systematically involved in assessing the quality of the teaching and learning interaction, however. • The increased government support for CSs could have negative side effects. On the one hand, communities may have unrealistic expectations about the amount and timing of government support. The support proposed and the conditions for providing that support may be misunderstood by communities, and by the advocacy groups for CSs that feel that government has the responsibility for 38 providing support. On the other hand, in providing this support, government personnel may inadvertently disempower communities. • There is a need to clarify the role and expectations of teachers who are government-paid and seconded to CSs. Where government teachers were appointed as “teachers in charge” of CSs or where a school had a large number of seconded government teachers, the role of community volunteers and the role of communities in support to the school could become eroded. In one case, the secondment of seven trained teachers displaced community volunteer teachers, who in fact had been receiving regular support from a local NGO. In another case, the PCSC sent away the “teacher in charge” for not consulting with the PCSC or reaching out to the community in accordance with its expectations. One community member said, “We’re tired [of doing everything.]. We want government to take care of our school now.” • At all levels of the system the reported challenges to full implementation of the guidelines come down to resources. At the community and school levels, there was a need for resources for infrastructure, teaching and learning materials, teachers, and in some cases, even land. Though progress has been made, CSs still lack teachers’ accommodations, good sanitation and clean water, as well as a means to consistently support and therefore retain volunteer CSTs. Classroom Level Classroom level insights were gathered through direct observation and interviews with teachers. Classroom observations provide information about specific teaching and learning activities, and an opportunity to see what teachers are doing as a result of their short courses in basic skills, IRI, or distance lessons of ZATEC. The evaluation team visited 17 schools in 11 districts in five provinces. These were schools where CHANGES2 or QUESTT had conducted teacher training and awarded small grants. The team observed teachers at 11 schools. Northwest schools were not observed and the remaining schools were conducting end of term exams. Among the teachers who were observed, eight had participated in CHANGES2 basic skills training and four were IRI-trained mentors. Teachers trained in basic skills portrayed knowledge of schemes for planning work, weekly focus and daily lessons in the classroom. Teachers developed their lessons using a question and answer method. They also invited pupils to write on the board and answer questions. In about half the classrooms, children were seated in groups and had activities to do as groups or in pairs, learning from each other as well as the teacher. Teachers correctly used the “tool kits” that were provided by CHANGES2. They used wall charts, the compass and the blackboard ruler as needed. The calculator and globe were often prominently displayed in the office. Teaching aids made during the basic skills training were also displayed, especially the cardboard shapes and alphabet charts. 39 The teachers who were interviewed revealed that after the CHANGES2 intervention they felt more confident in teaching their pupils. Some teachers noted that pupils were cheated in the past since teachers had no idea about how to plan lessons. Rather, they simply picked a topic and gave the pupils some work. The syllabus and planning skills gave them confidence. An indicator that quality education is available at CSs is that many reported sending pupils to Grade 8, which suggests that students are not dropping out, but continuing on with their education. Teachers also reported that the IRI methodology that uses English helps the children learn the language quickly. It also increases listening skills and comprehension of both pupils and teachers. Teachers feel that the instructions are easy to follow and that it is very easy to prepare the lessons. They appreciate the variety of activities throughout the lesson. Most teachers recognize that IRI requires preparation on their part, as shown in their prep books. The reception is good in the CSs that are near the radio stations. Reception in Southern Province is a major challenge, although QUESTT has notified ZNBC of the problem. In four schools, the IRI methodology could not be used because of poor reception. MP3 players were used well by classroom teachers when they were available. In addition to not having to rely on the radio signal, the players can be stopped and restarted if the lesson is moving too quickly for the classroom teacher or the pupils. All the teachers followed the mentor’s guides and wall charts, but in some places the pupils lacked textbooks. Promising practices The application of training in the classroom is the most promising and important classroom practice. Classroom observations and teacher interviews reveal that the basic skills training have been relevant and transferable to the classroom. The 20- day program provides an introduction and practice of key teaching skills, and supports the use of learner-centered methods and the national syllabus. IRI has several elements of good teaching practice embedded in each broadcast. One government teacher, appointed as a “teacher in charge” of a CS described these key pedagogical aspects of the IRI lessons, including learner-centered, activity-based, integrated curriculum, and gender awareness and language skills across the curriculum. He also noted that one could try the same approaches in other lessons. Another promising practice is the provision of teaching and learning materials, and training in their use. Three key resources provided are the teachers’ tool kit; the Grade 1-7 syllabus; and the MP 3 player. The tool kit is very basic, practical and well-used. The syllabus helps teachers know what to teach and how much to teach. All of these materials are introduced in the context of a training program, ensuring that teachers have the opportunity to master their appropriate use. The MP3 player gives a teacher the chance to pause, ask questions, and explain each lesson before proceeding to 40 another stage. This capability addresses the concern that the pacing of the lessons is too fast. Outstanding Challenges • Most schools lack enough text books and exercise books, • Some schools have well renovated classroom blocks but do not have enough desks. • The 9 a.m. Grade 1 Radio lesson has commercial adverts interrupting the broadcast. • Though training is a motivator, it also enables CSTs to move on to other CSs or other greener pastures. 41 VI. CONCLUSIONS USAID assistance to strengthen the MoE’s support to CSs has been comprehensive and timely. It has enabled the MoE to develop a comprehensive framework and operating guidelines for CSs. Its programs have provided models for delivering support to CSs in ways that build systemic capacity and enhance the quality of education. And it has recognized and strengthened existing models of support provided by district offices and NGOs. The evaluation question ‘how has the community schools landscape changed?’ can be answered from two perspectives. First, the MoE has made giant steps to embrace these informal structures as critical partners in the provision of basic education. Second, communities, particularly in proactive districts, feel that they are now recognized by MoE officials as key partners in the delivery of basic education. USAID program support, through the CHANGES2, EQUIP2 and QUESTT has contributed to these changing perceptions. The EQUIP2 headquarters presence has helped to influence policies and practices that move from the center to the school level, through provinces and districts. CHANGES2 is well connected to professional technical units at headquarters and has a strong presence from the district to the school levels. This presence has enabled CHANGES2 to strengthen the professional development system and improve the skills of teachers and school managers, and to promote partnerships for infrastructure development. QUESTT, providing a single, though complex methodology, and engaging new and viable technologies, has strong relationships in select units of headquarters and provinces. They also use the MoE’s extensive professional development system. Taken as a whole, the programs in the USAID Basic Education portfolio have been necessary, feasible and effective. The programs have contributed to strengthening Ministry structures at all levels, from the classroom to national headquarters. The Operating Guidelines for Community Schools provide a cohesive framework for continued support by government, cooperating partners, and international and national NGOs. The major themes of the findings, discussions and conclusions in the prior sections reflect accomplishments in the area of policy implementation; teacher development, placement and retention of teachers in CSs, and in the provision of infrastructure, and teaching and learning materials. The next section of the report provides conclusions in these thematic areas. Systemic Support to Policy Implementation Districts are apt and appropriate points of coordination and support to CSs. Proactive districts have developed responses to CSs; others will find their roles defined in the guidelines. District-led support and coordination provide the necessary flexibility in responding CS issues and opportunities, and further support decentralization. QUESTT and CHANGES2 have demonstrated that working with and through district structures is an effective way to deliver services to CSs. 42 There is no need at present to “replace” ZCSS. The roles and responsibilities for civil society and governments spelled out in the guidelines should be given a chance to mature. There may be a role for a national umbrella organization to help monitor the implementation of the guidelines and related advocacy work. The USAID program, operating at the national, district and school levels, should closely coordinate in key areas related to improving the quality of teaching and learning, including school management, materials provision and infrastructure. Government at all levels, as well as communities, recognize the need to maintain balance between community leadership and initiative and government support. Small grants for infrastructure development, whether through government or external sources, promote continued community participation. The small grants programs described here have generated community skills and cooperation, connected communities with relevant district offices and completed standard infrastructure. Perceptions of Quality Quality education is often measured by indicators that are defined by national bodies. Another lens is a community-level lens, which reflects the views of pupils, teachers and community members as they make decisions about joining a school, attending regularly, and generally participating in all aspects of school life. The evaluation revealed that quality is perceived to be higher when teachers, school managers and school committees are trained and feel more competent, when infrastructure is improved, and when creative and innovative practices are used in the classroom. The USAID-funded programs have influenced perceptions of quality in these areas. The holistic approach has altered both the reality and the perception of CSs. School managers speak of their schools as “no longer stigmatized” as being for poor, orphaned or otherwise disadvantaged children. The inclusion of school managers and community members in training, the provision of small grants for infrastructure, the availability of teaching materials, as well as the inclusive content of the basic skills training of CHANGES2 and the radio programs of QUESTT have enhanced the self-regard of volunteer CSTs, and the community’s ability to mobilize and utilize resources. The strategic framework and policy guidelines provide a vision for a CS as a full and active partner in the provision of basic education. Teacher Development and Support Training provided to teachers and school managers has improved the basic operation of the schools, teaching practice and community relations. Teachers value the basic skills and IRI training they have received, reporting increased skills, confidence and motivation, and positive pupil response. 43 Teaching practice is improved through training and follow up support and the availability of teaching and learning materials. Teachers have demonstrated their ability to use very basic teaching aids such as those provided in the teachers’ tool kit, to make and use teaching aids such as the geometric shapes, and alphabet charts, and to use modern technology such as an MP3 player. The provision of the syllabus during basic skills training has enabled teachers to know what they should be teaching. Together with lesson planning and learner-centered methods, CSTs are doing most of the things that “trained” teachers are doing. Short professional development courses are not linked to a qualification process though USAID and others remain committed to upgrading volunteer CSTs. The same is true of professional development of school managers. EQUIP2 is working with the MoE on a course for school administrators, which would be open to all head teachers, GRZ or community. CHANGES2 has substantial experience in terms of the needs and entry levels of community school managers that can be applied to this process. Models of monitoring and support have been developed by CHANGES2 and QUESTT, with a focus on classroom practice. These models create expectations for applying training and support to teachers. Teacher placement and retention Placement of trained teachers at CSs has been achieved by seconding teachers from a GRZ school establishment. Although the seconded teachers are often placed as “teacher in charge” there were no identified tensions in the schools visited. CSTs who were sponsored for ZATEC often were not returned to their communities. Teacher morale and retention are improved by consistent (even when small) allowances/support from the community. The ZATEC distance model has been a motivator for those enrolled and by design has kept the CSTs in a school for at least four years. Teaching and learning resources CSTs have demonstrated their ability to use teaching and learning materials that include the syllabus, tool kit, IRI-related materials and materials that they construct themselves. Textbooks remain in short supply. Infrastructure and equipment The small grants programs of CHANGES2 and QUESTT have provided a model for infrastructure development that maintains community leadership and ownership, links schools to district resources, and results in improved infrastructure. Radio infrastructure to support and sustain IRI remains a challenge. ZNBC’s signal does not consistently cover the country even in areas where coverage should be expected (e.g., the Central and Southern Provinces). Still, its coverage exceeds that of the 44 community stations, which generally broadcast only within an 80 km radius of the station. All of the infrastructure, teaching and learning materials need to be maintained. The community training for grants includes a component on maintenance, which is a positive beginning. There is an informal network of support for the MP3 players at the community level (i.e., posting near Peace Corps Volunteers who tend to be avid users of MP3s). Some schools need help troubleshooting their radios (extending antenna) and the system of sending all radios to Lusaka for repair is inefficient. Resource mobilization Successful use of grant funds attracts additional resources. This has been encouraged during training of communities in grants and project management. About five schools have sought and/or received support from other sources, based on the improvements made through the CHANGES2 and QUESTT grants. Support from local and regional private sector entities to CSs can provide additional resources to CSs. Examples include mines, tourism-related businesses; and local NGOs. Defining and monitoring quality The changing landscape of CSs includes improved quality, from the perspective of pupils, parents, and teachers. District and national offices are engaged in monitoring quality largely for planning and program purposes. Communities have not yet been directly involved in the definition and monitoring of quality education, which would add a new dimension to their ownership of the school. 45 VII. RECOMMENDATIONS The recommendations presented here are in broad thematic areas, based on lessons learned from the three projects, and cutting across institutional levels. Given the findings of this evaluation, there is a sense that a concerted effort to address these recommendations would not only clarify roles, responsibilities and expectations, but also create a robust support framework for CSs. Policy Implementation 1. Continue to deliver program and project activities through existing MoE structures, paying attention to the capacity building, staffing levels, resources and results at each level of the education delivery system. 2. Manage misinterpretation of guidelines to avoid unrealistic expectations among communities about the pace and extent of government action, or concerns that government may be “taking over” CSs. The influences of EQUIP2, QUESTT and CHANGES2 at various levels of the education delivery system should be maximized. Consistent messages should be developed collaboratively at the national level communicated to the school and community levels through the QUESTT and CHANGES2 (or similar programs). The home offices of EQUIP2 and CHANGES2 have experience in developing such “social marketing” behavior change programs, involving an integrated “school and community partnership” campaign on the role of communities, government intentions, and the roles and responsibilities of the partners vis-a-vis all schools, with an intensive focus on CSs. The good will, training and equipment investments of QUESTT in community radio stations represent another asset toward this goal. 3. Provide continued capacity building and resources to districts to retain flexible and comprehensive support to CSs. Capacity includes level and types of staffing and staff development as well as transport and funding. 4. Establish an approach to monitoring the implementation of the Guidelines. This might include a role for an umbrella organization or association of CSs. Teacher Development and Support 1. Develop a pathway to professional recognition for teachers and school managers, linking academic record, performance and short courses, through a competency framework that can be applied to government and CSs. Competencies for teachers should include attention to content mastery, as well as the twelve teaching skills and the life skills (SHN and HIV/AIDS) that are included in the MoE strategic framework as well as the implementation experiences of EQUIP2 and CHANGES2 program. This competency framework 46 will give meaning and specificity to the pathway for qualification described in Figure 3 on page 32. The competencies for school managers should include leadership, management and administration, including financial management and recordkeeping, monitoring and supporting academic work of the school, and building/maintaining community relationships. 2. Promote regular support to community volunteer teachers; encourage predictable levels of support. Teachers report appreciating the regularity of support even if the support is not substantial. 3. Ensure that district level staffing, and DRC and ZIC work loads are sufficient to serve the expanding requirements for district support to teacher professional development. Consider making ZICs full-time positions. 4. Review the ZATC modules and assignments in collaboration with the Colleges of Education to ensure that contact sessions complement and complete the work of the modules and distance assignments, rather than simply delivering all the content of the program. Teacher Placement 1. Continue to pursue the establishment of the legal status needed for CSs to have established teaching posts. In the meantime, encourage districts to continue seconding trained and salaried teachers to CSs. 2. Work to include CSTs who finish ZATEC in the recruitment and posting cycles as soon as possible, returning them to their communities of origin for the agreed upon period of service. 3. Promote balance between the assignment of trained government teachers to government schools and the potential displacement of community volunteers. 4. Provide public recognition of CSTs (volunteer and government), elevating their status and making CS placement more attractive to student teachers and graduates. This could be as simple as inclusion in speeches or World Teachers’ Day activities. Teaching and Learning Materials 1. Continue the provision of teaching and learning materials, including the IRI￾related materials and the core materials of the CHANGES2 Basic Skills programs, School Community Partnership, School Health and Nutrition, Project Management; and the SPRINT materials that support the integration of HIV/AIDS 47 and Gender in the curriculum. Continue to provide CSs with the Grade 1-7 Syllabuses. 2. Maintain the benefits of IRI by establishing a system for more locally-based troubleshooting and repair of radios. Continue connecting MP3 users with people who are familiar and comfortable with the technology (Peace Corps Volunteers and others who might be identified in colleges or towns.) 3. Expand the use of MP3 players in remote areas and in places where there is poor reception from ZNBC. Continue using community stations to rebroadcast Learning Toanga Market, where affordable. 4. Listen to the Grade 1 radio broadcast to ensure that the program is broadcast intact, i.e., to ensure that the adverts noted by the evaluation team are removed. 5. Review the pacing of the IRI programs. The timing of activities may not reflect the realities of class size and the time needed for pupils to reach the chalkboard or move into groups. 6. Maintain the teachers’ toolkit and seek funding for more durable globes. Infrastructure development 1. Maintain and expand the small grants program, using the current planning and implementation approach, which builds community capacity and engages relevant district staff. 2. Ensure that the size of individual grants is small enough to require a cost share as well as community prioritizing and careful planning yet large enough to complete something tangible. The current range of US$2,500 to US$5,000 has had good results. 3. Increase the number of small grants available to CSs, at a pace that ensures continued district involvement, support and capacity building. 4. Consider conditions under which a school or community might be eligible for a second grant, while maintaining the concept of leverage, e.g., investment in an income generating activity that could be used for regular allowances to teachers and run as a business. Encourage local partner NGOs to work with district officers in school construction and rehabilitation (rather than outside the system). 5. Radio signal strength is an infrastructure challenge, whether ZNBC or community stations. In areas where the ZNBC signal is weak and can be strengthened this should be done; in areas where the signal cannot be strengthened without significant investment, an analysis of the cost of using community stations and/or MP3 players should be considered. 48 49 Expanding Available Resources 1. The Government of Zambia and its cooperating partners are not likely to meet the demand for resources for CSs. An expanded participation of private sources is needed and should be pursued on two levels. At the district/provincial level, regional business interests and NGOs might be sources of funding. These could include major industries (tourism, mining, retail, telecommunications, agriculture/food processing, financial institutions) as well as smaller NGOs, church groups or organizations such as Rotary and Lions Club. The MoE, with appropriate technical support, can explore ways to impart skills, particularly among district staff, through face to face solicitation and workshops presenting the opportunities for private sector financing of education; and then the development and dissemination of guidelines and matching opportunities. 2. MP3 players are much more expensive than radios. At the national or international level explore a partnership that would provide MP3 players and accessories at a deeply discounted price. Manufactures, retailers and NGOs are options. World Bicycle Relief is a good example of an NGO with a very specific technology to promote. 3. Utilization of community radio stations to rebroadcast IRI over the course of a year may require resources beyond the current “mandate” of stations to provide public service programming. Securing regular rebroadcast times that would serve the needs of schools with multiple sessions will require funds beyond those currently allocated to ZNBC. Defining and Monitoring Quality Develop a set of standards and related tools for assessing school quality. Tools should include those that can be used for internal monitoring by school management and community members, as well as “external” monitoring by district or provincial teams. Community members can be involved in monitoring quality at the school. Practical, goal￾focused training has enabled them to recognize and practice concepts such as transparency, maintaining a cashbook, timely expenditure of funds, partnership and teamwork. In the same way, using highly practical and visual methods, community members can be involved in monitoring other aspects of quality, including teaching and learning. The Integrated Education Strategy of the USAID-funded Uganda Program for Human and Holistic Development (UPHOLD) developed a “ checklist” composed of pictures of what should be taking place at school or in a classroom, and parents were invited to come to school, observe classrooms and tick the practices that they found. The results were used to promote further valuable parent and community involvement in the school. This page left intentionally blank Annex A: CHANGES2 Program Findings National Level What is the impact of the teacher training on attrition and on the overall teacher training system? How was CHANGES2 involved in strengthening MoE system and structure? Were the CHANGES2 FTI activities effective in strengthening MoE system and structures? The team was unable to determine impact of teacher training on attrition. CHANGES2 strengthened the MoE system and structure for teacher development and grants management at the school level. The impact on the overall teacher development system included pre-service training, in-service training and monitoring and support of teachers. At the pre￾service level, CHANGES2 provided training and materials to Colleges of Education, providing them with state of the art information that was being delivered through the in-service system. In addition, CHANGES2 funded the placement of student teachers at community schools, providing potential mentors to community school teachers (CSTs) and providing the opportunity for college staff to support teachers in low resource settings. The placement of ZATEC students in community schools was valued by students and described by college staff as too difficult to monitor. On the in-service side, CHANGES2 used and strengthened existing staff and structures by engaging them as trainers, providing training in improved monitoring and support services, and facilitating support visits and teacher group meetings. DRCCs report having increased skills and confidence based on their work with the program. They also report that working with CSTs boosts their morale as the CSTs are motivated and interested in improving their teaching skills. CHANGES2 support to Teachers’ Group Meetings that were either focused on or inclusive of CSTs helped establish a norm of full participation of CSTs, who feel that they are becoming more accepted by trained teachers. CHANGES2 has shared its field experiences and approaches within the national teacher development system, engaging relevant players at critical stages of program development and review. CHANGES2 continues to work to institutionalize its approaches and materials through the in-service and pre-service teacher training systems of the MoE, engaging MoE and other partners in its planning and program development. Annex A: 1 Provincial/District/Zonal Level What are the qualitative changes in behavior/attitude of officers after training, monitoring, and supporting teachers and PCSCs in infrastructure grants process? What has been the impact/use of library books and other materials at DRC/TRCs? DRCCs, ADRCCs and ZICs who have been involved in training, monitoring and supporting teachers report mutual improvement in morale. They feel that CSTs are interested in improving their teaching practice and are receptive to the training and support visits. The relationship that is developed between CSTs and DRC staff enables CSTs to more readily access materials that are available at the DRC. Buildings and procurement officers who have been involved in training and monitoring for the infrastructure grants also value being appreciated and viewed as resources rather than “inspectors”. Several district officers report being more motivated to actively seek out resources for community schools, based on the experiences created in implementing the CHANGES2 activities. Library materials and other resources made available to the DRCs are largely used by teachers who are enrolled in distance learning programs and who can get to the DRC. Many CSTs are not able to access the centers due to distance. Community Level What changes have occurred in the behavior and attitude of PCSCs after teacher training, monitoring visits, grants? Where the CHANGES2 FTI activities effective in improving the physical infrastructure of community school through the provision of small grants? What benefits to communities as a result of sub-grant implementation? (Infrastructure, ability to generate, manage resources) How have perceptions of education quality changes based on teacher training and infrastructure? Are parents more wiling to send kids to school? The monitoring visits by district officers, whether following up teacher training or grants implementation confer a feeling of status on the school and community. Seeing teachers returning from training with materials (the toolkit, booklets) lets parents see that something serious has happened. The small grants program was successful in improving infrastructure, including rehabilitation of classroom blocks, construction of latrines or provision of furniture. The grants funds were used effectively, against well developed budgets and procurement plans. The involvement of district buildings and procurement officers provided good support in terms of standards and quality of Annex A: 2 construction and pricing for materials. Inclusion of preventive maintenance in the training for PCSCs increases the likelihood that infrastructure will last. Infrastructure improvements provide pupils, teachers and community members at large with a sense of stability, permanence and pride in the school. The small grants program had benefits of a more enduring nature than just infrastructure. These include the knowledge, skills and confidence developed through the process of learning together to plan, budget and execute the grant and the relationships established with district officers. School managers, pupils, teachers and community members report confidence in self and each other as well as skills in budgeting, budget management, procurement and financial accountability and transparency and inventory management. One school manager reported that parents can organize themselves to complete required tasks without being reminded. Several schools report applying for grants from other sources, including the U.S. Ambassador’s Self Help Fund, CARE International, and the Firelight Foundation. . The manner in which CHANGES2 staff worked with schools and committees was also very effective, setting benchmarks and following up; linking schools to other resources were key parts of the process. In one case, a school which remained with a surplus helped other schools to buy furniture; at least three schools had been encouraged and empowered to apply for U.S. Embassy Self Help grants after successfully completing their CHANGES2 grant. Other successfully received grant funding from CARE, Firelight and other NGOs. School managers and teachers report increased enrolment and improved attendance in community schools that received infrastructure improvement grants. Pupils perceive that their community school is just like any other school and in some cases, parents are taking children, especially young children and girls from distant GRZ schools and enrolling them in the community schools. These anecdotes suggest that the community has positive perceptions of community schools. It was often noted that the community takes better care of the school now. One school manager reported that the local chief sent his indunas to patrol the school grounds, and another that the chief was charging parents one chicken for failure to attend a school/community meeting. . Annex A: 3 School Level How has the morale of teachers and head teachers in community schools changed as a result of CHANGES2 intervention? What qualitative changes are there in behavior/attitudes of school managers after training, monitoring and support? What has been the participation and benefits of GRACE meetings and other continuous professional development events? What is the impact of the materials on school quality? What changes or benefits do pupils perceive as a result of CHANGES2 intervention? School managers are more knowledgeable about the kind of administrative and academic records to keep. They also understand the importance of monitoring the academic work of the school, including checking prep books and lessons. In several schools, enrollment figures for the current year were posted prominently as was a timetable. A school manager, though a trained teacher indicated that she now “speaks boldly”, knows her leadership style, works with teams, and works through delegation. Community school teachers participate in the term GRACE meetings, and are eager to learn. The response of the GRZ trained teachers to the community school teachers is mixed. Some are encouraging and engage them as colleagues; others are disdainful, cautioning their pupils “not to behave like a community school teacher.” The use of DRCs by teachers is mixed, largely due to distance. Since DRC staff delivered the CHANGES2 training, they feel comfortable using the DRC, but the distance often prohibits its use. Teachers who are writing assignments and those near centers try to use them regularly. Annex A: 4 Annex A: 5 Classroom Level What has been the impact of teacher training? What positive teaching practices are evident in the classroom? What has been the impact on teachers’ self esteem, confidence, sense of professionalism, retention? Was the teacher “tool kit” useful? If so, in which way (s)? How have the educational materials been used by community school teachers? What has been the impact on pupils and community perceptions of community schools of these improved teaching practices? Teachers report increased skills, confidence and morale due to their participation in the CHANGES2 training and subsequent monitoring and support visits. There is a sense that the MoE and partners know that they are out there, and when the PEO distributes end-of-training certificates, teachers feel recognized and honored. Basic teaching skills enable CSTs to participate confidently in GRACE meetings. One teacher said that while trained teachers once looked down on them (CSTs), she does not fear them any more. Beyond the morale and confidence, the following observations were made in classrooms, confirmed by discussions with head teachers, teachers, pupils and community members. Teachers proudly and confidently plan their lessons using the syllabus. Having the syllabus helps teachers know what to teach in which grade. Even if they are without textbooks, they can choose something appropriate for the grade. One teacher reported that prior to learning how to plan and scheme, she would just give children some work. Additionally, there is strong evidence of active learning, e.g., paired or small group work and active use of textbooks. There is evidence of the toolkit in schools that have participated in training. Only one school manager reported having the entire kit at home, due to lack of security at the school. Teachers use the globe, atlas, dictionary and mathematics instruments, though the ball, board ruler and calculator may be the most popular items. Several pupils feel that now their school is like other schools. New classrooms, desks and good teaching are at the top of the list of “what I like best” at my school. They also recognize that parents and community members have contributed to school improvements. . There is also some anecdotal reporting of improved pupil’s performance. According to one DRCC who participated in the exercise, several community schools performed very well in the national assessments of learning. Pupil attendance is reported to be more regular due to the presence of a permanent classroom, even without desks, better use of class time, and better organized lessons. Annex B: EQUIP2 Program Findings National Level Were the EQUIP2 activities effective in enhancing policy frameworks and improving MoE institutional capacity to support community schools? Who introduced the need for the policy? How was it formulated? Did the policy frameworks improve the delivery of services to community schools for equitable and quality education? The need for a strategic framework and implementation guidelines for community schools became apparent as the ZCSS met its demise. Policy dialogue between EQUIP2, the MoE and partners who had traditionally supported the ZCSS and community schools, resulted in a broad based consultative process to develop a the strategic framework and operating guidelines. The process included representatives of all levels of civil society and government, national and international NGOs active in the sector, and bi- and multilateral cooperating partners. This ensured that the frameworks were grounded in reality, as well as possibility. The process of developing and vetting the guidelines and the dissemination process that began early in 2008 have strengthened consensus and created a platform for MoE support to community schools as appropriate at all institutional levels. It is too early to tell if the policy frameworks have improved delivery of services to community schools but the Strategic Framework and Operational Guidelines spell out ideal conditions for the delivery of services. They clarify previous formulas, e.g., BESSIP and Sector Pool and codify some existing practices of districts and provinces, especially with respect to seconding teachers and administration of grant funds. They have also enabled the MoE to include community schools explicitly in the MoE’s Annual Work Plan and budget. Provincial/District/Zonal Level How are these levels involved in policy development and dissemination? How far have they gone? What are the gaps in dissemination? How are CSs managed and financed in terms of teacher employment and infrastructure? Who monitors/ supervises the quality of education services delivery? How are MoE at this level coping with current and projected responsibilities for community schools? All PEOs and DEBS reported being involved or represented during the formulation of the Operating Guidelines and in reviewing the guidelines once drafted. To varying degrees, provincial and district officers were engaged as task team members and/or reviewers during national conferences. This ensured that the current practices and experiences of provinces and districts as well as local partners were reflected in the guidelines. At the time of the evaluation, national and provincial level dissemination meetings had taken Annex B: 1 place and districts were planning their outreach to schools and communities to share the guidelines. Districts were expected to do this using their own funds and most had planned to use GRACE meetings during the April 2008 term break for dissemination. A missed opportunity may have been the teacher training events conducted by CHANGES2 and QUESTT during the same holiday. The guidelines may create unrealistic fears and/or expectations of government involvement and support to community schools. All 5 PEOs and 11 DEBS in visited districts mentioned that volunteer community school teachers are employed and paid by their communities. GRZ only pays teachers against established posts, and then only if they are qualified and trained. Pro-active districts have seconded teachers from GRZ schools to nearby community schools as a way to support the community schools. The teachers are often designated “teachers in charge” and the presence of a government employee enables schools to access direct funding from the district. Districts share available grant funds with community schools, largely at the discretion of the district. Previous formulas for sharing the grant funds were not equitable and as clarified in the guidelines are not longer in effect. The earmarking of funds in the MoE budget for community schools, and the new funds available for classrooms provide opportunities for much needed support to community schools. At provincial and district levels, standards officers report that they monitor community schools in the same fashion that they monitor GRZ schools. At the provincial level, this means that community schools will be among the types of schools that are sampled for monitoring on a term basis. Six of 11 district level standards officers confirmed visiting community schools, though added that it was difficult to visit them because of lack of time, funds and transport. Community Level Who in the community monitors the quality of educational services? At the community level, only Zonal In-Service Coordinators formally monitor quality of schools. Communities do however have perceptions of quality and their perceptions are reflected in support for the school, attendance of enrolled pupils and enrollment overall. Annex B: 2 Annex B: 3 School Level How are teaching and learning materials allocated to community schools? Who monitors and supervises delivery of quality services? Are community school head teachers better equipped to respond to MoE reporting requirements? The teaching and learning materials are allocated by the DEBS officers and most report that they are allocated based on the number of pupils in the schools. In the past, community schools were not always included in the allocation of teaching and learning materials but that has changed as the GRZ realized the importance of community schools in the provision of basic education. As noted in the section on Provincial and District Levels, Standards Officers strive to monitor quality of services provided in community schools. Community schools Head Teachers are beginning to realize their role in keeping track of enrollment, attendance and academic work of the school. Very few head teachers could distinguish between the EQUIP2-led recordkeeping training and the CHANGES2 school management training. EQUIP2 training focuses on complete, accurate and timely completion of all required forms, while the CHANGES2 training focuses more broadly on the management and administrative tasks of the school as a whole. Classroom Level Are community schools striving for quality? What is quality from community school perspective? Tools used? Who monitors? Community perceptions of quality include infrastructure, trained teachers, teaching and learning materials and pupil interest/performance, and support from government and other partners. Infrastructure generally includes classroom blocks, toilet and teachers’ accommodation; trained teachers include community volunteer teachers who have participated in short courses as well as the formally trained teachers. Support from partners comes in the form of funding, materials, and in-kind contributions such as housing. Community schools are striving for quality, as indicated by their hopes to reach Grade 7; the shift from the SPARK curriculum to the Basic Education Curriculum, and by the investments that community members make in the school (time, materials, produce, labor, cash). There are no standard practices for community level monitoring of quality. Generally, one might say that communities monitor and then act; enrollment and attendance increase; support for the school, including volunteer teachers increases. Annex C: QUESTT Program Findings National Level Is ZATEC by distance an appropriate way to train community school teachers? What is the quality of formal training of community schools teachers? Are graduate teachers confident in IRI? These findings are based on interviews with ZATEC students who were doing practicum at a community school in Northwestern Province; student teachers enrolled in ZATEC distance courses, their school managers and others in the teacher education field at district, provincial and national levels. Three organizations provide ZATEC distance education: ZOCS, Save the Children/Norway, and QUESTT. Each program varies in the structure of the contact time. The ZATEC distance program has several advantages: 1. The conditions of sponsorship keep CSTs in place for four years. 2. A formal qualification is a tremendous motivator and reward for the individual teacher. 3. Standardized approaches to teaching and learning among GRZ and community schools. ZATEC raises systemic challenges. The first is the posting of newly qualified teachers back to their communities. This has two dimensions. The first is the situation in which a teacher prefers to return to the community but is unfamiliar with the procedures that will enable that to happen. The second is the general pooling of candidates (national, provincial or district levels) that might inadvertently result in a posting even out of the province. The third issue is related to the overall conditions of service for teachers. Once a CST has ZATEC, they are eligible for the same conditions of service as other teachers, and some feel that they would like to receive those benefits, which may not be available at their community schools. These issues of posting are common to all sponsors of ZATEC by distance. Recognizing that QUESTT did substantial training of college lecturers to develop the modules and that they were reviewed for instructional design, lecturers may still need additional support in distance implementation. Discussions with colleges regarding the confidence of ZATEC graduates in implementing IRI indicated that it is too soon to tell. The ZATEC students met by the team during their teaching practice had not studied IRI at college. Some ZATEC students are unable to implement the IRI methodology fully because not all places use radio. Annex C: 1 Provincial/District/Zonal Level What is efficiency of distribution of radios and mentor guides? How has support to zone/district improved quality? Are ZICs using bicycles to monitor IRI sites? Has the training model contributed to sustainability? Distribution of radios and mentor guides are not the reported constraint. The challenge to schools comes when materials are not produced in sufficient quantity or timely and/or when radios are in need of repair. In two provinces, the team encountered situations where radios were not working and schools did not know what to do with them. In one case, the ZIC had collected a radio but had not returned it in several months. The condition of the radios was a surprise to project and district staff who would have taken immediate action. Reliance on a troubleshooting and repair service in Lusaka for the radios does not support locally sustainable practices. ESODOLs report satisfaction with the training and support approach followed by QUESTT. Cascading through district and zonal staff creates a broad base of support for the program. In one DRC where an impromptu stop was made, the Assistant DRCC was able to demonstrate the use of the IRIpod for radio instruction and to describe and scroll through the resources available on the device. A second DRC reported a “malfunctioning” IRI pod, and promised to seek guidance from another district. The practice of locating school based IPODs near Peace Corps Volunteers may provide another line of support as this new technology is introduced and installed. At another DRC, there was a feeling that the 3 day training is insufficient, especially for CSTs who have only completed Grade 9. The concern was that the English level of a Grade 9 is not sufficient to master the content of the program and IRI methodology in a three day period. Provision of bicycles improves the ability to monitor schools, and ZICs use the bicycles for general monitoring and in some cases to take things to the district. ZICs are full time teachers which reduces the time they have available for school based monitoring and support. Some schools report reducing the class load of the ZIC but this is not a standard practice. The shift this year to ZICs as the face to face trainer of teachers should push the training closer to the school level. If this model grows, the workload and expectations for ZICs, and DRCs should be considered. While they can play a critical and sustainable role in school based continuous professional development, providing training and support to teachers, they are still designated as classroom teachers. Morale issues, conditions of service and other expectations are worth examining if they are to continue to play a pivotal role in an expanding number of schools. Annex C: 2 Community Level Have communities supported IRI? Were Community Radio stations effective ways to mobilize and inform? What changes arose in CBOs/NGOs through the grants? Were grants used effectively? What were the impact of small grants on schools and communities? School managers and teachers report that community members value IRI as an interesting way to learn that promotes English, listening skills and literacy. Some community members are interested in using the program for adult literacy. Community radio stations, though present in several districts were largely viewed as methods to disseminate Learning at Taonga Market, HIV/AIDS and SHN messages. Schools also used them to announce enrollment periods. Listener feedback reports are favorable however, especially for the Radio Maranatha programs on Re-Entry to School and Pupils with Learning Disabilities. The CBO/NGO grant program of QUESTT provided grants to pre-identified high performing organizations. A total of 18 grants were awarded for income generating, training for school managers, based on the specific needs and interests of the organization. Infrastructure grants were awarded in 5 provinces only to avoid duplication of CHANGES 2 grants. In all cases, QUESTT reports that the grants were used as intended, Even in instances of “misapplication” the argument has been made that the purpose for which the grant was used supported the overall purpose of the grant program even if it was not used as planned. In all cases, the grants funds, in part because they were small, were used effectively, against well developed budgets and procurement plans. The involvement of district buildings and procurement officers provided good support in terms of quality of construction and pricing for materials. Transport costs however were often quite high for materials that needed to be brought from towns. The infrastructure improvements provide pupils, teachers and community members at large with a sense of pride in the school and themselves. Having permanent structures creates a sense of seriousness about the school… it is a “real” school. School managers, pupils, teachers and community members report confidence in self and each other as well as skills in budgeting, budget management, procurement and financial accountability and transparency and inventory management. In the Northwestern Province, one community school structure became the model for a neighboring school (about 5 km away). QUESTT training was conducted at the “model” school, creating an additional point of pride for that community, and building a supportive relationship between the two schools. Community members and school managers report that training and timely support visits by project and district staff were critical to their success. Annex C: 3 School Level What are some positive changes for teachers and pupils due to IRI and related training? (learning, motivation, other) Teachers report appreciating the methodology because it is planned and organized; it is interactive; the children enjoy it; and it makes teachers and pupils listen and think faster. Writing and speaking skills in English are often reported as benefits to the pupils. The most articulate user of IRI encountered during the field work also noted that the programs used learner centered methods, promoted use of teaching and learning aids; integrated themes across the curriculum, including life skills, and with the pre- and post-broadcast activities provided good ways to assess the learning objectives were a good example of teaching practice. While some teachers felt that IRI meant that you didn’t need to prepare a lesson, many felt that IRI did in fact require and prompt good preparation. IRI TEACHER OBSERVATION The evaluation team visited 5 provinces and observed 11 teachers. Out of 11 teachers, 9 community teachers were observed using IRI methodology. The lessons observed were: lessons 46 to lesson 49. Five were trained teachers and four were untrained. The teachers observed had knowledge about how to prepare a schemes work, weekly focused and the lesson plans. The introduction of the lessons observed were quite good which showed that, the teacher training in IRI had an impact on the teachers and the pupils. The trained teachers portrayed knowledge and skills that they had learnt at the college. Teachers developed their lessons using question and answer method. they also used participatory methods, where pupils participated in writing on the board or answering questions. Most teachers gave children some activities to do in pairs or groups which encouraged unity in class. All the pupils were aware when to stand up and start writing because of songs that are included in the IRI lessons. All the lessons observed had good activities for the children to do, except that the 10 minutes from the radio teacher was not enough to finish all the activities set in the lesson. But for those who used the IRI pod had a chance to go back until it was done. The reception was good to those community schools that were nearer the radio stations. In almost 4 schools the IRI methodology could not because of poor reception. In Some schools, teachers observed used ordinary methodology because the radios were broken and it was not clear on who should do the maintenance. Benefits include: • IRI methodology uses English which make the children to learn the language quickly • It also increases children listening skills and understanding • The instructions are easy to follow hence teachers find it very easy to prepare the lessons. • There are a lot of activities for the children throughout the lesson. Once oriented to the flow of the programs, pupils quite enjoy themselves. They sing and move to the introductory note, respond loudly to radio and classroom teacher prompts, and follow the Annex C: 4 Annex C: 5 flow of the lesson. In classrooms where the teachers were relatively new to the methodology, or where they were conducting demonstrations for the evaluation team outside of their usual grade level, it was a bit harder to generate the intended interaction. Classroom Level How have MP 3 players been used? Have they made teachers more effective? Do teachers feel better able to teach? MP3 use observed during the field work was exclusively for Learning at Taonga Market. The MP3 has largely been used to provide LTM where reception is poor or for afternoon sessions. The clarity of the programs does make it easier for the teacher to be able to teach. Annex D: Sites Visited Evaluation: USAID/Zambia Assistance to Strengthen Ministry of Education Support to Community Schools Sites Visited NORTHWESTERN PROVINCE District: Solwezi (QUESTT grant sites) Schools: Katoka Community School Kamijiji Community School Kyalalankuba Community School COPPERBELT PROVINCE District: Mufulira Visited Mufulira DRCC Schools: Kawama West Community School Mufulira College of Education District: Ndola Visited Ndola DRCC Schools Talent Community School Province: Central District: Kabwe Schools: Chililalila Community School Salamano Community School Radio Maranatha Natuseko Community School Central Provincial Training: Basic Skills Part 2 (CHANGES 2, DRCC trainers) SOUTHERN PROVINCE District: Livingstone Schools: Mwandi Community School Nalituwe Basic School District: Kazungula Schools Mubiana CS District: Kalomo: Visited DRC School: Matondo CS (IRI pod) District: Choma Schools Munzuma CS District: Gwembe DEBS Staff Only LUSAKA PROVINCE District: Lusaka Schools Fight Poverty CS Garden IRI Center District: Chongwe Schools Lumano CS Nangombe CS Annex D: 1 Annex E: Individuals Contacted Interviews and Discussions Name Designation Ministry of Education Headquarters Mrs. Lillian Kapulu Permanent Secretary, Ministry of Education Mr. Charles Ndakala Systems Development Manager, MOE Mr. Alex Kaba Planning Officer, Policy and Research, MOE Mrs. Ruth Mubanga Director Teacher Education and Specialized Services, MOE Mrs. Stella Kasase. Chief Education Officer, Open and Distance Education,, MOE Mr. Jeff Mulenga Senior Education Standards Officer/ Exams, MOE HIV/AIDS Advisor, HRA National and International NGOs Ms. Miriam Chinyoma Executive Director, Zambia National Education Coalition (ZANEC) Mrs. Harriet Sianjibu-Miyato, Programme Coordinator, Zambia Open Community Schools (ZOCS) Mr. Gibson Nchimunya, Gibson Program Officer, Save the Children Norway Mr. Steve Power Assistant Country Director, CARE Mr. Rhodewell Chitanda Program Officer, CARE Cooperating Partners Ms. Melissa Williams Mission Director, USAID Ms. Sheila Lukens Deputy Mission Director, USAID Mr. Rick Henning SO 6 Team Leader, USAID Mr. Cornelius Chipoma Deputy Team Leader SO6, EQUIP CTO, USAID Ms. Beatrice Mweene CTO QUESTT, USAID Ms. Katie Donohoe CTO CHANGE 2, USAID Mr. Vincent Snijders First Secretary, Education Royal Netherlands Embassy Ms. Given Daka Education Program Officer, Royal Netherlands Embassy USAID PROGRAM STAFF Mr. Arnold Chengo Chief of Party, EQUIP 2 Mr. Sri Perrera Information Systems Advisor and Deputy Chief of Party, EQUIP 2 Mr. Joe Kanyuki Policy and Research Advisor EQUIP 2 Mr. Pule Mundende HIV AIDS Workplace ECZ Advisor SHN Advisor Mr. Edward Graybill Chief of Party, CHANGES 2 Ms.. Joan Woods Deputy Chief of Party and HIV/AIDS Advisor,/CHANGES2 Ms. Joy DuPlessis Teacher Education Advisor, CHANGES 2 Mrs. Katherine Phiri School Health and Nutrition Advisor, CHANGES 2 Mr. Sitwala Mubunda M&E Officer, CHANGES 2 Mr. Daniel Mapoma M&E, CHANGES 2 Annex E: 1 Name Designation Mr. Josiah Zulu Grants Manager, CHANGES 2 Mr. Ian Milimo Copperbelt Provincial Manager, CHANGES 2 Mr. Cosmos Mukobe Central Provincial Manager, CHANGES 2 Mr. Benedicto Phiri Central Province Grants Manager, CHANGES 2 Mr. Peter Sampa Lusaka Provincial Manager, CHANGES 2 Ms. Juliet Mbesha Provincial Grants Manager, CHANGES Mr. Evans Mumba Southern Provincial Manager, CHANGES 2 Ms. Saboi Sichili Southern Province Grants Manager, CHANGES Mr. Richard Trewby Chief of Party QUESTT Mr. James O’Rourke Deputy Chief of Party and Teacher Ed Advisor, QUESTT Mr. Francis Sampa Teacher Education Coordinator, QUESTT Mr. Hitler Muleya Grants Coordinator, QUESTT Ms. Francesca Mubiana Outreach Coordinator, QUESTT Ms. Mary Kantema CBO Grants Manager, QUESTT Ms. Miriam Shakafuswa Community Radio Mr. Bernard Solochi QUESTT Provincial Outreach Coordinator, Northwestern Ms. Irene Changwe QUESTT Provincial Outreach Coordinator, Copperbelt Mr. Cosmos Musumpuka QUESTT Provincial Outreach Coordinator, Central Mr. Peter Mufwindi QUESTT Provincial Outreach Coordinator, Southern Mr. Emmanuel Malubila QUESTT Provincial Outreach Coordinator Acting, Lusaka Northwestern Province Mrs. Malama Provincial Education Officer, Northwestern Province Mr. Frederick Mukinyi District Education Board Secretary, Solwezi Distrit Mr. Collins Monde DESO, Solwezi District Mr. Movens Nswana Planning Officer, Solwezi District Mr. Ben Kankomba Procurement Officer, Solwezi District Mr. Morris Shumba Chair PCSC Katoka CS, Solwezi District Mr. Laidone Musheka School Manager, Kamijiji Community School, Solwezi District Mr. Lazarus Phiri, ZATEC Student Practicing at Kamijijij CS, Solwezi District Mr. Oliver Mbiliti ZATEC Student Practicing at Kamijijij CS, Solwezi District Mr. John Mubambe School Manager, Kyalalankuba CS, Solwezi District Ms. Rosita Kalilemba IRI Mentor, Kyalalankuba CS, Solwezi District Mr. Richard Sakara Chair, PCSC Kyalankuba CS, Solwezi District Copperbelt Province Mr Kamutuwa, Muyangwa Provincial Education Officer, Copperbelt District Education Board Secretary, Mufulira Mr. Jeremiah Mbulya Chair, District CS and AL Committee, Mufulira Ms. Grace Chilekwa Vice Principal Mufulira College of Education Mr. Roid Hamasuku School Manager Kawama West CS, ZATEC Distance Student Mr. Gordon Mwewa Information Systems and Global Learning Portal, Mufulira Annex E: 2 Name Designation College of Education Mr. Baron Mwangwe Lukwesa District Resource Center Coordinator, Mufulira Ms. Charity Ngabwe Asst. District Resource Center Coordinator, Mufulira College of Education Mr. Enoch Kabaso School Coordinator, Kawama West CS, Mufulira Distirct Ms. Ngosa Shakabozha DRCC, Ndola Mr. Silwamba District Planning Officer, Ndola Distirct Mr. Makewell Kaongoa Makewell School Manager, Talent CS, Ndola District Central Province Mr. David Bwale Chisenga, Provincial Education Officer, Central Mrs. Betty Mzumara SEO, School Guidance, PEO Besie Temba EO/Teacher Education, PEO Mr. V. Sinda FPP Community Schools, PEO Evelyn Musanje, SESE/BS Jennifer Banda DEBS, Kabwe Paxina Chinanda ADRCC,Kabwe Enoch Phiri Zonal InService Coodinator, Chindwin Zone, Kabwe Mr. Chama Gregory, School Manager, Salamano CS, Kabwe District Mr. Chama Director of Project, Salamano CS, Kabwe District Mr. Mbaliti School Manager, Natuseko CS, Kabwe District Mr. Matthews Mulumbi School Manager, Chililalila CS, Kabwe District Laurens Haangala Station Manager, Radio Maranatha Ms. Miriam Mazuba, Producer and Presenter Mr. B. Singoyi PRCC, Central DRCC DRCC DRCC Southern Province Mr. Johnwell Siwingwa Provincial Education Officer Chula, RMZ, Mrs. School Health and Nutrition/Southern Musa, VL Mr. , SESO/ODL, Southern Province (FPP QUESTT) Mr. Litiya Nyaywa SESO/SE, Southern Province Siamusiye,Regina, Teacher Education, , Southern Province Malambo Haunguwa Livingstone DEBS Vivien Mwayopwa ESO/ODL, Livingstone George Lubinda DRCC/TE Livingstone Mrs. Joyce Kanimba School Managers, Mwandi CS, Livingstone District Zach Alston Simpila Mwandi CS, STC Norway sponsored ZATEC distance Annex E: 3 Name Designation Beenzu Mwanashimbala ESO General Inspection, Kazungula District Mr. Lingamba School Manager, Mubiana CS, Kazungula District? Mr. D. Kaluba DEBS Kalomo Mr. D. Siamulaba School Manager, Matondo CS, Kalomo Mrs. Mungala Senior Education Officer, Kalomo and Choma Districts Mr. EB (Big Brains) Himwiita School Manager, Munzuma CS, Choma District Venta Ng’andu DRCC, Gwembe Chacks Nzala Planning Officer, Gwembe District Victor Simulya Human Resources Officer, Gwembe Alice M. Nzala Provincial Education Officer. Lusaka Province Mr. Kalwani PESO, Lusaka Province Mr. Zimba SESO,Lusaka Province Mrs. Banda EO/.TED, Luska Province Mrs. Sitali SEO/ODL, Lusaka Province Ms. Prisca Simukonda DEBS Lusaka Mr. O. Kangulu Planning Officer, Lusaka District Mr. Simulu Buildings Officer, Lusaka District Mr. Henry Munje School Manager,Fight Poverty Community School Mr. Mvula, IRI Mentor Garden IRI Center at Ngwelele Basic School, Lusaka Mrs. Ngonya H.NMwanga DEBS Chongwe Mr. Musoni DESO, Chongwe Annex E: 4 Name Designation Ms. Rhodia Mudenda DRCC, Chongwe Mr. Hambozi Guidance Coordinator, Chongwe Mrs. Kembo ESO/Gender, Chongwe Ms. Rita Mwemba School Manager Lumano CS, Chongwe District School Manager, Nangombe CS, Chongwe District Note: At each school focus group discussions were held with pupils and community members. The pupils names are recorded on the next page. Community members ranged from 1 indiividual to 28 and not all names recorded. Annex E: 5 PUPILS INTERVIEWED IN COMMUNITY SCHOOLS 1. KAWAMA COMMUNITY SCHOOL- MUFULIRA-COPPERBELT 1. Hmasiku Royd 2. Bufumi Evans. 3. Katongo Mutale. 4. Mwansa pauline 5. Mweene Steria 6. Kapiya Carol 2. MAATONDO COMMUNITY SCHOOL 1. Charles Masiye 2. Winny Maambo 3. Emmy Mbambiko 4. Hassan Mudenda 5. Mauritions Simabeche 6. Jackline Munsaka. 7. Godwin Mbozi 8. Lactant Siamfumba 9. Wisper Mwaanga 10. Ashelly Siamasiku 11. Mcfallen Siamufalali 12. Royce Mukonka 3. Munzuma- Choma 1. Carlo Chimfwembe. 2. Junior Efford. 3. Busiku Muleya 4. Cholwe Chooga Annex E: 6 Annex E: 7 5. Lumamba Choonga 6. Yvonne Muntanga. 7. Oberty Chongo. 8. Jackueen Munsaka 9. Simon Kangwa. 10. Junior Sianpongo 11. Mamvwu Munkombwe. 12. Peggy Mandumandu 4. Fight Poverty- Lusaka Province 1. Gabriel Soko 2. Mbango Mushe 3. Memory Nobutu. 4. Micheal Sakala 5. Isaac Tembo. 6. Richard Malunza 7. Joyce Bwelenga. 8. Joyce Hambofu 9. Julia sabwacha. 10. Innonge Mwauluka 11. Eunice Tembo 12. Rachael Banda Annex F: Interview Schedule and Questionnaire INTERVIEW SCHEDULE FOR PROGRAM STAFF 1. Name of Organisation/Employer of Respondent. 2. Position of Respondent. 3. What are the impacts or teacher training? 4. How has the morale of teachers and headteachers in community schools changed as a result of CHANGES 2 interventions? 5. What direct and indirect benefits have accrued to schools and communities as a result of implementing the sub-grants? 6. What quantitative changes are manifest in the attitudes and behaviours of provincial and district-level officers as a result of the training and monitoring and support activities? Probe: What plans do they have to continue this work ‘on their own’ in the future? 7. What impacts, or potential impacts, have been achieved through the provision of books to teacher resource centres? 8. How are community schools striving to achieve quality education? Probe: What tools are there for assessing this? 9. Has IRI been a positive change for children or their teachers in GRZ and community schools? 10. Have communities receiving radio programs become mobilized to support their volunteer teachers or provide more for their children? 11. Have the small grants been used effectively by communities to support their IRI centres? Probe: Have they been used as they are intended to be used? 12. Is ZATEC an effective way to train volunteers while also keeping them in their centres as teachers? 13. Has the use of MP3 players in the pilot activity made teachers more effective? 14. Are teachers graduating from the College of Education program confident that they can use IRI in their classes? Probe: Are they confident that they can use IRI in their classes? 15. To what extent has the use of made an impact on GRZ and community schools? 16. Are the ZICs using the bicycles to monitor IRI schools? 17. How effective are the community radio stations in informing and mobilizing the communities? 18. What programs have been implemented by the CBOs as a result of the training they received? Annex F: 1 INTERVIEW SCHEDULE FOR SCHOOL MANAGERS 1. Name of Respondent 2. Name of School. 3. What are the impacts of teacher training? 4. How have the educational materials been used by your teachers? Probe: What are the impacts of the materials on school quality? 5. What skills and competencies have members of the PCSCs developed as a result of their training? 6. Has IRI been a positive change for children or their teachers in GRZ and community schools? 7. Have communities receiving radio programs become mobilized to support their volunteer teachers or provide more for their children? 8. Has the use of MP3 players in the pilot activity made teachers more effective? 9. How effective is he training of teachers in the use of IRI? 10. Has there been efficiency in the distribution of radios and mentors’ guides for the IRI class? 11. Are teachers graduating from the College of Education program confident that they can use IRI in their classes? 12. Probe: Are they confident that they can use IRI in their classes? 13. What is the impact of grants on the communities and learners? Annex F: 2 INTERVIEW SCHEDULE FOR PUPILS 1. Name of school. 2. What is your name? 3. What grade are you doing? 4. For how long have you been at this school? 5. What do you like more at your school? 6. What changes or benefits have you seen at your school from last year? Annex F: 3 FOCUSED GROUP DISCUSSION SCHEDULE FOR PCSCs 1. Name of school. 2. What skills and competencies have members of the PCSCs developed as a result of their training? 3. How effective are the community radio stations in informing and mobilizing the communities? 4. Changes implemented by the CBOs as a result of he training they received? 5. What is the impact of grants on the communities and learners? Annex F: 4 QUESTIONNAIRE FOR TEACHERS NAME OF SCHOOL: -------------------------------------------------------------------------- Dear Respondent, The information sought through this questionnaire will be used to Evaluate the Impact of FTI Funding to the Zambian Education. Your truthful responses to the items will be greatly appreciated and treated with the greatest confidentiality. Instructions 1. For questions where optional answers (a, b, c…) are given, please Tick in the brackets provided. 2. Where blank spaces are provided, write your own answers. Section A 1. Name of Respondent: ---------------------------------------------------------------------------- 2. Name of school: ---------------------------------------------------------------------------- 3. Sex of Respondent: a. Male [ ] b. Female [ ] Section B 4. What are the impacts of teacher training? ---------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------- Annex F: 5 5. a) Was the teacher ‘tool kit’ useful? Yes [ ] No [ ] b) If your answer to a) is YES, in what ways? ------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------- 6. How do you perceive the effectiveness of the training teachers in the use of IRI? ------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ 7. What would you comment about the efficiency in the distribution of radios and mentors’ guides for the IRI classes? ---------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------- Annex F: 6 Annex F: 7 SCHOOL OBSERVATION LIST 1. Yes and No will be used to indicate the availability and the scale of 1-3 for the status where applicable. 1 stands for good, 2 for average and 3 for poor. Yes No 1 2 3 Comments 1. Administration a) b) c) 2. Physical Facilities a)Classrooms b)Water supply c)Toilets d) e) f) An assessment will be done in this section based on the sliding scale of 1- 3. 1 stands for adequate, 2 for inadequate and 3 for not available. Items that affect teaching and learning 1 2 3 Comments Instructional Materials a) Text books b) Teachers’ Guides c) Visual aids d) Laboratory apparatus & chemicals e) Chalk f) Chalkboards g) Schemes of work & lesson plans h) Syllabus Annex G: Integrated Information Gathering Tools INFORMATION GATHERING TOOLS: Evaluation of USAID’s Fast Track Initiative Support to Community Schools in Zambia Overview of the Tools The tools provided here include interview questions, focus group discussion guides and classroom observation tools. They are organized per data source, as follows: Data Source Tool Provincial Education Officers, District Officers, including DEBS, Statistics, Planning Officer, Building or Procurement Officer, G&C,. Interview Questions, Section 1, pp 2-10 School Manager Interview Questions, Section 2,pp 11-15 Teachers10, DRCCs, ZICs, Colleges Interview Questions, Section 3, pp 16-22 Observation Tool (IRI/CH2) Program Staff Interview Questions, Section 4, pp 23-28 Community Members Focus Group Discussion Guide Section 5 pp 29-31 Pupils Focus Group Discussion Guide Section 6, pp 32-33 The questions were derived from the table identifying key evaluation questions by project and by institutional level. The table was presented in the original DevTech proposal and reviewed and enriched by input from project staff in Lusaka. Annex G: 1 Section 1: Interview Guide: PEOs and DEBS Introductions: Contact person will introduce the team members and provide an overview of the evaluation. State the purpose of the evaluation: to determine how USAID supported activities have contributed to the improvement in access to basic education and quality of community schools. Key activities that will be examined are those of the three major USAID Basic Ed projects: CHANGE 2, EQUIP 2, and QUESTT. (State which are active in the current province) Record Name of respondent Position: Province/District Contact Info Section 1.1 Background : 1.1.1 In general, how do you view the role and contribution of community schools in your province/district? 1.1.2 As you look out over the next 5-10 years, what is your vision for community schools? Annex G: 2 1.1.3 Several training activities have taken place in the past year for district and zonal staff. Which have you participated in? Training that has taken place in the province: Dates IRI CHANGES2 TT Part 1 CHANGES 2 TT Part 2 HT Training, CH 2 Grants Management QUESTT Grants Management CH MOE Record Keeping MOE Continuous Assessment for Community Schools Roll out of CS Implementation Guidelines Section 1.2 Policy Formulation Process in Community Schools 1.2.1 How were you involved in developing the policy and the guidelines? Be specific. 1.2.2 Would you say that the guidelines promote quality education and development in community schools? Give reasons for your answers. Annex G: 3 1.2.3 What in your view led the Ministry to release the guidelines at this time? Section 1.3 Impact Of Policy On Management, Coordination And Finance In Community Schools 1.3.1 How are the following resources and opportunities allocated to community schools: - a. Teaching and learning materials b. MoE Grant funds c. In-service professional development opportunities Annex G: 4 d. Support visits by district and other staff. 1.3.2 Are you satisfied with the quality and quantity of management, financing and coordination you provide to the community schools? Section 1.4 Quality Education Assessment in Community Schools 1.4.1 What measures do you use to assess the quality of education in community schools? (Give examples of specific criteria) Annex G: 5 1.4.2 How has IRI affected quality of teaching and learning in GRZ and community schools? 1.4.3 How has CHANGES 2 Teacher Training affected the quality of teaching and learning in the community schools? 1.4.4 How has CHANGES 2 Head Teacher Training affected the quality of teaching and learning in the community schools? 1.4.5 What kind of support is provided to community schools by district staff (Standards, Teacher Education, Building, Guidance and Counseling, DRCs, Infrastructure development) Annex G: 6 1.4.6 How have district officers and ZICs improved their ability to support community schools through the training and follow up activities of CH2 and QUESTT. 1.4.7 How have the QUESTT/CHANGES 2 grants improved quality in the community schools? (probe for skills, management, harmony, confidence , transparency as well as the physical works completed; include training in Grants/Project Management) 1.4.8 How often do CSTs use the District Resource Centers? Annex G: 7 Section 1.5 MoE Provision Of Teachers And Salaries In Community Schools. 1.5.1 Do you have a specific criteria you use to deploy teachers to community schools? 1.5.2 What incentives are given to CS teachers? Probe: by Ministry, school managers, NGOs, community. Give reasons for answer. Section 1. 6: MoE Change in Status of Community Schools 1.6.1 What influenced the Ministry of Education to change the current status in community schools? Annex G: 8 1.6.2 Who else was influential in changing the status? NGOs, other government departments? Were there any external organization working with MoE? Give reasons for your answers. Section 1.7: Management of Community Schools at Zonal and District levels. 1.7.1 Are the community school projects effectively managed at both district and zonal levels? Give reasons for your answer. Section 1. 8 : Conclusions/Summary 1.8.1 What changes have you seen in the overall perceptions of community schools in the past year? ( perceptions of parents, pupils, teachers, MoE staff, NGOs) Annex G: 9 Annex G: 10 1.8.2 What are the three major challenges you face in supporting community schools and implementing the new policy guidelines? Section 2: Interview Guide School Managers Introductions: Contact person will introduce the team members and provide an overview of the evaluation. State the purpose of the evaluation: to determine how USAID supported activities have contributed to the improvement in access to basic education and quality of community schools. Key activities that will be examined are those of the three major USAID Basic Ed projects: CHANGE 2, EQUIP 2, and QUESTT. (State which are active in the current province) Record School Profile School: District/Province School Manager: History: No. of teachers: M= F= No. of learners M= F= Teachers present on day of visit: PCSC Members FTI/USAID TRAINING COMPLETED Name Designation Training Course Title of Training* Dates Enrolled in ZATEC Distance MoE Recordkeeping MoE Continous Assessment IRI School Management for Annex G: 11 CS Head Teachers Training of CS Teachers Part 1 Training of CS Teachers Part 2 School Health and Nutrition Project Management (CHANGES 2) Grants Management (QUESTT) Annex G: 12 Section 2.1 Impact of training. 2.1.1 What difference has this training made for your school? 2.1.2 What skills and competencies have been gained by School Manager: Teachers PCSC Annex G: 13 2.1.3 Does the PCSC now have the confidence to apply for other grants? Have they done so? Section 2.2 Educational materials 2.2.3 What education materials have been received at your school? 2.2.4 How have the educational materials been used by your teachers? (Mentors guides, tool kit, DRC resources?) Probe: What are the impacts of the materials on school quality? Annex G: 14 Section 2.3 Quality of Teaching and Learning 2.3.1 Has IRI been a positive change for children or their teachers in GRZ and community schools? In what ways? 2.3.2 Have the basic skills trainings and teaching learning methods promoted by CH2 made a positive change for teachers and pupils? In what ways? 2.3.3 How have teachers, parents and pupils responded to these interventions? Annex G: 15 Section 2.4 Grants. 2.4.1 What differences do you see in the community and the learners as a result of the grants? 2.4.2 What changes have you seen in past year in the attitudes and support that people have for community schools. (community members, local leaders, parents, pupils, MoE staff). Section 2.5 Awareness and Mobilization 2.5.1 Is there a community radio station serving this area? Annex G: 16 Annex G: 17 2.5.2 Have communities receiving radio programs become mobilized to support their volunteer teachers or provide more for their children? 2.5.3 Are you aware of the new implementation guidelines recently released by the MoE for Community School? How did you learn about it? SECTION 3: Interview Guide for Teacher Education of Community School Teachers (use for teachers, College of Education, DRCC, ZICs) Introductions: Contact person will introduce the team members and provide an overview of the evaluation. State the purpose of the evaluation: to determine how USAID supported activities have contributed to the improvement in access to basic education and quality of community schools. Key activities that will be examined are those of the three major USAID Basic Ed projects: CHANGE 2, EQUIP 2, and QUESTT. (State which are active in the current province) Note: The CH2 and QUESTT classroom observation guides will also be used with selected teachers in each school. Name Position Organization District Province Section 3.1 Impact of Training 3.1.1 Which of the training programs have been the most helpful to you and why? Teacher Annex G: 18 : School Manager Teacher Educator: Zonal Coordinator: Annex G: 19 3.1.2 Please Respond to the following: What changes have you seen as a result of the training? (tick all that apply; give specific examples if possible) Pupils +Motivation +Attendance +Learning +Enrollment Other: describe Parents +Encourage attendance +Participation in Meetings +Move children to CS +Support School and/or teachers Other: describe Teachers’ Teaching Skills/Use of Materials Morale Confidence Retention Other: describe School Managers Management Skills Confidence Financial Mgmt School/Comm unity Relations Other: describe Zonal/Distri ct Staff Support to CSs Confidence Conducting Continuous Professional Development Monitoring Other: describe Section 3.2 Quality of community schools. 3.2.1Over the past year, what improvements have you seen in: Training and support for CSTs? Teaching and learning in the classroom? Provision of teaching/learning materials Use of teaching learning materials Continuous assessment of students School management Annex G: 20 3.2.2 What changes have you seen in perceptions of community schools over the past year? Section 3.3. Challenges in implementing new guidelines: 3.3.1 What are the biggest challenges in implementing the new policy guidelines? Section 3.4. For Zonal and District Staff: 3.4.1 What are the benefits and challenges of training zonal and district staff as in￾service trainers? Annex G: 21 Annex G: 22 3.4.2 What are the benefits and challenges of zonal and district staff making follow up monitoring and support visits after training? Section 3.5. For College Staff: 3.5.1 What have been the benefits and challenges of training college staff in IRI? 3.5.2 Are graduating teachers confident in their ability to use IRI in the classroom? 3.5.3 Is distance education an appropriate way to train CSTs? Concerns? Section 4: Interview Questions for Program Staff Introduction: Introduce team members and program staff. State the purpose of the evaluation: to determine how USAID supported activities have contributed to the improvement in access to basic education and quality of community schools. Key activities that will be examined are those of the three major USAID Basic Ed projects: CHANGE 2, EQUIP 2, and QUESTT. (State which are active in the current province) Record Name Title Organisation Province/District Section 4.1 Overall success of program 4.1.1What achievement of your program (community schools) are you most proud of? Why? Section 4.2 Training 4.2.1 What are the impacts of teacher training ( IRI, CST Training Part 1 and 2; Head Teacher Training; Grants/Project Management) Annex G: 23 4.2.2 What programs have been implemented by the CBOs as a result of the training they received? Section 4.3 Grants 4.3.1 How are funds/grants distributed to community schools? 4.3.2 Have they been used the grants as they are intended to be used? What big changes have been as a result of funding to communities/community schools over the past year? Annex G: 24 Section 4.4 Community mobilization 4.4.1 How effective are the community radio stations in informing and mobilizing the communities? 4.4.2 Have communities receiving radio programs (Community radio stations) become mobilized to support their volunteer teachers or provide more for their children? 4.4.3 How have the SHN activities, including gardening, mobilized communities? Annex G: 25 Section 4.5 Resource Materials 4.5.1 What impacts, or potential impacts, have been achieved through the provision of books to teacher resource centres? 4.5.2What are the maintenance issues related to the IRI radios? Are schools, ZICs and district staff aware of procedures for repairing/replacing them? What do schools do when their radio is out of service? 4.5.3 Are the ZICs using the bicycles to monitor IRI schools? Do they also use them for other support related activities? Annex G: 26 Section 4.6 Quality 4.6.1 How are community schools striving to achieve quality education? Probe: What tools are there for assessing this? 4.6.2 How often are community schools monitored? (by program staff? By district officers?) Section 4.7 MoE Support to Community Schools 4.7.1 How has support to community schools changed over the past Annex G: 27 Annex G: 28 4.7.2 What are the main challenges facing the MoE in implementing the community schools guidelines? Section 5: Focus Group Discussion Community Members Introductions: Contact person will introduce the team members and provide an overview of the evaluation. State the purpose of the evaluation: to determine how USAID supported activities have contributed to the improvement in access to basic education and quality of community schools. Key activities that will be examined are those of the three major USAID Basic Ed projects: CHANGE 2, EQUIP 2, and QUESTT. (State which are active in the current province) (Note: an assigned interpreter from evaluation team, program staff, ministry or community should be familiar with the purpose of the evaluation and sensitive to the participation of all, including women) Name of School: Name of Respondent Position 1. 2. 3. 4. 5. 6. 7. 5.1 What changes have you seen in your school in the last one year? Annex G: 29 5.2 In the past one year, who present participated in training for: Training Program Names of participant Grants Management (QUESTT) Project Management (CHANGES 2) Gardening/other IGA (CHANGES2) 5.3 What skills and competencies have members of the PCSCs developed as a result of their training? 5.4 What is the impact of grants on the communities and learners?(Probe using: How did the grants training help you manage your school?; What have you learnt by implementing this grant?) 5.4 Have you applied for any other grants since completing the training? Y/N 5.5 Is there a community radio station here? How effective are the community radio stations in informing and mobilizing the communities? Annex G: 30 Annex G: 31 5.6 What kind of help do you get from: Source Type of Assistance Ministry of Education 5.7 What additional help do you want to receive from government? 5.8 Are you aware of the government’s new guidelines for community schools? Section 6: Focus Group Discussion for Pupils Introductions: Contact person will introduce the team members and provide an overview of the evaluation. State the purpose of the evaluation: to determine how USAID supported activities have contributed to the improvement in access to basic education and quality of community schools. Key activities that will be examined are those of the three major USAID Basic Ed projects: CHANGE 2, EQUIP 2, and QUESTT. (State which are active in the current province) (Note: an assigned interpreter from evaluation team, program staff, ministry or community should be familiar with the purpose of the evaluation and sensitive to the participation of all, including women) Record: Name of School: Names of Respondents Grades Length of Stay at School 1. 2. 3. 4. 5. 6. 7. 8. 6.1 What do you like best at your school? Annex G: 32 Annex G: 33 6.2 What changes or benefits have you seen at your school from last year? (Probe for infrastructure, teaching quality, learning materials, parent support) 6.3 How did these changes happen? Who helped make the changes? 6.4 What other improvements do you want for your school? Who can help you? Annex H: Questions for MoE and Cooperating Partners USAID Evaluation: Strengthening Support to Community Schools INTERVIEW QUESTIONS: Ministry of Education Date:___________________________ Respondents: Name Position Contact Info 1. Introductions: MoE and Team 2. Purpose of the Evaluation: assess the extent to which USAID support has strengthened MoE support to CSs. 3. General Questions: i. What in your view has been the role and contribution of CSs to education in Zambia/ ii. What is your vision for CSs in the next 5-10 years? 4. What in your view led the MoE to develop and release these guidelines at this time? Annex H: 1 Annex H: 2 5. What was the role of your department in developing the policy and guidelines? 6. How, in your view, will the guidelines contribute to reaching the goals of FNDP and MDGs? 7. Which elements of the guidelines are the major responsibility of your directorate? 8. Which elements of the framework do you feel reflect existing practices within CSs and which are new? Elements Existing Practices New Establishment, Registration, Upgrading and Change of Status Institutional Framework Stakeholders and Responsibilities Teacher Training Teacher Deployment Infrastructure Development Records Management Resource Allocation Access and Utilization of Resources Monitoring, Standards and Assessment Annex H: 3 What have been the contributions of the USAID supported parties in building capacity and contributing Elements Responsible Units Contributions EQUIP Contributions QUESTT Contributions CHANGES 2 Establishment, Registration, Upgrading and Change of Status Institutional Framework Stakeholders and Responsibilities Teacher Training Teacher Deployment Annex H: 4 Annex H: 5 Elements Responsible Units Contributions EQUIP Contributions QUESTT Contributions CHANGES 2 Infrastructure Development Records Management Resource Allocation Access and Utilization of Resources Monitoring, Standards and Assessment What steps has the Ministry taken to ensure that the benefits of these projects are sustained? (Note; sustaining benefits and continuing activities are not always the same thing) What in your view are the biggest challenges to implementing the new guidelines at the following levels? What is the role of GRZ, Civil Society and Cooperating Partners? Challenges GRZ Civil Society Cooperating Partners National Provincial District DRCs Colleges of Education Zonal Structures School Level Community Classroom Annex H: 6 This page left intentionally blank Annex I: References REFERENCES CHANGES2 – FTI Community Schools Monitoring Instrument CHANGES2 – Student Questionnaire CHANGES2 – Teacher Questionnaire CHANGES2 (2007) Ongoing Professional Development. CHANGES2: Lusaka. CHANGES2 (2007) USAID/Zambia CHANGES2 Program Annual Work Plan 2007. CHANGES2: Lusaka. CHANGES2 –Books for Resource Centres-Sorting Guide. CHANGES2: Lusaka. CHANGES2 –Classroom Observation Lists (2 Types) CHANGES2 School Check List Education Development Centre (2006) Supplemental Program Description, Attachment 1. EDC: Newton. EQUIP2 (2007) EQUIP2 Zambia 2007 Work Plan. EQUIP2: Lusaka EQUIP2 (2007) EQUIP2 Zambia Performance Report on Community Schools Initiatives Under FTI Funding: January – December 2007. EQUIP2: Lusaka. EQUIP2 (2008) Quarterly Report October – December 2007. EQUIP2: Lusaka. International Book Bank (2007) Packing List. International Book Bank, Inc: Baltimore. Ministry of Education (2003) Zambia Basic Education Course Syllabi-Grades 1-7. Curriculum Development Centre: Lusaka. Ministry of Education (2006) Dropouts and Grade Repeaters: What are Policy Implications for Systems Efficiency? Issue Brief No. 06-03. MoE: Lusaka. Ministry of Education (2006) Education Statistical Bulletin. MoE: Lusaka. Ministry of Education (2006) Impact of Community Schools on the Provision of Basic Education: Issues and Strategic Options, Issue Brief No. 06-02. MoE: Lusaka. Ministry of Education (2006) Learning at Taonga Market: Grade 2 Evaluation of Interactive Radio Instruction in Community and GRZ Schools. QUESTT: Lusaka. Ministry of Education (2006) Moving Towards Educating Our Future: Challenges and Oportunities Facts form the Briefs. Issue Brief No. 05-02. MoE: Lusaka. Annex I: 1 Ministry of Education (2006) School Health and Nutrition Teachers’ Guide. CHANGES2: Lusaka. Ministry of Education (2007) Education Statistical Bulletin. MoE: Lusaka. Ministry of Education (2007) Hand Book for Training of Community School Teachers Part2-Draft. CHANGES2: Lusaka. Ministry of Education (2007) Hand Book for Training of Community School Teachers. CHANGES2: Lusaka. Ministry of Education (2007) Infrastructure Improvement Grants for Community Schools-Operational Guidelines: Improving the Learning Environment in Zambia Community Schools. CHANGES2: Lusaka. Ministry of Education (2007) National Implementation Framework 2008-2010. Implementing the Fifth National Development Plan. MoE: Lusaka. Ministry of Education (2007) Operational Guidelines for Community Schools. MoE: Lusaka. Ministry of Education (2007) Project Management: A Hand Book for Parents Community School Committees. MoE: Lusaka. Ministry of Education (2007) Project Management: A Handbook for Parents Community School Committees. CHANGES2: Lusaka. Ministry of Education (2007) Report on Use of Cellphones Training for Colleges of Education, Teachers’ Resource Centres and ZATEC Students. QUESTT: Lusaka. Ministry of Education (2007) Review of the Ministry of Education Sector Plan Zambia. MoE: Lusaka. Ministry of Education (2007) School Program of In-service for a Term (SPRINT): a Teachers’ Guide for School-Based Conitnuing Professional Development. CHANGES2: Lusaka. Ministry of Education (2007) School-Community Partnerships: Training Manual for Teachers and Communities. CHANGES2: Lusaka. Ministry of Education (2007) Training Manual for Community School Head Teachers- Draft. CHANGES2: Lusaka. Ministry of Education (2007) Training Manual for Community School Head Teachers. CHANGES2: Lusaka. Annex I: 2 Annex I: 3 Ministry of Education (2007) Training Manual for Community School Teachers Part 2. CHANGES2: Lusaka. QUESTT (2006) Work Plan 2006-2007. QUESTT: Lusaka QUESTT (2006) Work Plan 2006-2007. QUESTT: Lusaka. QUESTT (2007) Quarterly Report-April 1st to June 30th. QUESTT: Lusaka. QUESTT (2007) Quarterly Report-January 1st to March 31st. QUESTT: Lusaka. QUESTT (2007) Quarterly Report-July 1st to September 30th. QUESTT: Lusaka. QUESTT (2007) Quarterly Report-October 1st to December 31st. QUESTT: Lusaka. Annex J: Scope of Work SCOPE OF WORK The scope of work provides the context and technical requirements of the work to be accomplished. 1. METHODOLOGY AND TECHNICAL APPROACH The approach shall take into account the unique objectives of each program’s FTI activities, the overall FTI funding goal of improving community schools, and the complexity of educational delivery among institutional levels. Furthermore, the framework addresses the practical matter of completing the evaluation in a short timeframe as it focuses the questions by type of person(s) to be surveyed. This approach shall be used to provide USAID two sets of conclusions and recommendations that respond to the key objectives identified in the SOW. The first set is program-specific and assesses the coverage, impact and degree of local participation in each program’s FTI intervention. Recommendations at this level describe how program support to community schools can be improved. The second set is institution-specific and assesses the relative impact of FTI funding at different institutional levels. Identifying strengths and weaknesses in community school development along the chain of education delivery shall be used to illuminate at which institutional levels FTI funding has had the greatest impact and where additional funding could be focused for the greatest potential impact on community schools and teacher attrition in future USAID/Zambia programming. A. Program Assessments: In terms of assessing the effectiveness and impact of the each program’s FTI funding, the Team shall design their instruments to answer the questions highlighted in Figure 1. The following narrative summarizes these questions by program. A.1 CHANGES2 implemented by American Institute of Research (AIR) • Were the CHANGES2 FTI activities effective in improving the teaching-learning process, strengthening MOE systems and structures, and improving the physical infrastructure of community schools through the provision of small grants? • Did these activities maximize the impact of USAID support to MOE through policy dialogue that improved community school quality and increased their service to out of school youth? A.2 EQUIP2 implemented by Academy for Educational Development (AED) • Were the EQUIP2 FTI activities effective in enhancing policy frameworks and improving MOE institutional capacity to support community schools? Additionally, did they improve the delivery of services to community schools for equitable and quality education? Annex J: 1 Annex J: 2 A.3 QUESTT implemented by Educational Development Center (EDC) • Were the QUESTT FTI activities effective in: scaling up the use of IRI programs in GRZ and community schools; increasing the capacity of communities and NGOs to support education and retain volunteer teachers; and, developing a pilot activity to provide internet based resources to rural communities to improve education delivery? • Did the activities contribute to the attainment of Zambia’s Fifth National Development Plan (FNDP) education objectives and the Millennium Development Goal of universal primary education? The Team shall also ask the following questions of each program: • What were the programmatic factors that limited impact or success? • In general, how has the community landscape changed as a result of CHANGES2, EQUIP2 and QUESTT interventions? Did the FTI activities have an impact on the teaching-learning and school management processes in the areas in which they operated? If so, how are those contributions likely to have an impact even after the activities end? B. Institutional Assessment: Recognizing that program design depends on identifying the type and institutional level at which to intervene in addition to understanding the effectiveness of specific programs, the evaluation shall provide USAID/Zambia recommendations for future programming and ways to halt community school teacher attrition. To do this, the Team shall answer the following key questions: • What is the impact of FTI funding at each level of community education delivery? - Briefly describe the community development activities at each level and the linkages among each level. - Determine the degree of local participation and support of the funded activities. - Identify factors at each institutional level that limited impact or success. - Assess the coverage of the FTI interventions at each level. • What is the relative effectiveness? What would increase the effectiveness at various levels? - Compare the different institutional levels of the programs and explain their relationship, and strengths and weaknesses. - Examine which activities were most successful within each level, using clear determinants of success, and explanations of the process. • What are the predictors of community school teacher attrition? - At which levels are the factors for teacher attrition the highest? C. Evaluation of Teacher Training Interventions: DevTech shall conduct classroom observations in order to evaluate the Teacher Training Interventions under FTI. Direct classroom observations are imperative for gauging the impact of a teacher training intervention and gaining a clear picture of the state of the classroom, including its teachers and learners. The information collected during the observation shall be evaluated in light of the FTI Teacher Annex J: 3 Training Intervention’s curricular, pedagogical, discipline and professional policies. We believe that the key to a truly beneficial and effective evaluation is the interpretation of the observed data with defined criteria and standards. DevTech shall also conduct interviews with community members, parents and MOE officials to gain a broader understanding of the intervention beyond the classroom level. The Evaluation Team shall: • Develop a data collection instrument that includes a checklist of expected classroom practices based on the objectives and intended outcomes of the Teacher Training Interventions under FTI (i.e. elements of the “toolkit” that are expected to be in use at the time of observation). This instrument shall build on and incorporate existing data collection instruments used by CHANGES2, QUESTT and/or the Ministry of Education (MOE) for their classroom visits. • Conduct a short pre-observation meeting with the teacher to inform him/her about the purpose of the classroom observation; familiarize him/her about the data collection instrument and inquire about what shall make him/her most comfortable. • Observe teachers in the classroom. • Discuss the observations with the teachers, and explore factors that have helped and hindered application of the training (this might include the training approach, individual motivation, supportive environment among other factors); and changes in their view of themselves as teachers. • Discuss teaching practices and attitudes with head teachers, parents and students as part of a larger discussion on the school, its achievements and challenges. • Provide initial feedback to the head-teacher, teachers and parents in order to ensure that information is shared. This feedback shall focus on the positive and developmental feedback. Specifically, the following is a scenario of how the teacher training aspect of the evaluation shall be conducted: 1. Team members meet with head teacher to discuss the purpose and manner of information gathering, and to select/identify teachers for observation. If lesson plans are available in the head teacher’s office, then the Evaluation Team shall review the lesson plans prior to observation. 2. One team member shall continue interviewing the head teacher, while other team members conduct observations in two other classrooms. 3. A third classroom observation is conducted by the team member who interviewed the head teacher, while the other team members interview/observe pupils. 4. Team members interview community members and parents. Annex J: 4 5. Team members summarize their findings and share them with the head teacher and staff in order to share observations gathered regarding teaching practices being used; teacher/pupil ideas about the changes in the school related to the FTI activities; and community perceptions of quality in general. 6. Team members conduct brief feedback meeting with head teacher, available teachers and community members. If time allows, and parties are interested, district officer and team member conduct a one-on-one feedback session with the teacher they observed. 7. One copy of the completed data collection instrument is left with the school. In addition to the questions posed in Figure 1 above, additional questions shall be posed to evaluated teacher training interventions. For example, for a site with a CHANGES2 intervention, the Evaluation Team shall answer: • What are the impacts of the teacher training? • How has the morale of teachers and head teachers in community schools changed as a result of CHANGES2 interventions? • What changes or benefits do pupils perceive as a result of CHANGES2 interventions this year? • Was the teacher “tool kit” useful? If so, in what way(s)? • How have the educational materials been used by community school teachers? What are the impacts of the materials on school quality? • In general, how has the community school “landscape” (perceptions, commitments, realities on the ground) changed as a result of CHANGES2 interventions? For a site with a QUESTT intervention, the Evaluation Team shall answer in addition to other questions: • Has IRI been a positive change for children or their teachers in GRZ and community schools? • Have communities receiving radio programs become mobilized to support their volunteer teachers or provide more for their children? • Is ZATEC an effective way to train volunteers while also keeping them in their centers as teachers? • Has the use of MP3 players in the pilot activity made teachers more effective? • Are teachers graduating from the College of Education program confident that they can use IRI in their classes? • What is the effectiveness of the training of teachers in the use of IRI? The Teacher Training Intervention evaluation shall yield recommendations for programs providing teacher training interventions as well as for future USAID Programming, including identifying aspects of teacher training that mitigate teacher attrition. In this process, DevTech shall identify and report on the following: • Training practices observed in the classroom • Quality of the execution of training practices • Practices that were expected, but not observed and why they might have been missing Annex J: 5 • Factors that help and hinder the application of training. • Promising practices that teachers or head teachers attribute to training, but were not planned objectives/outcomes. Conclusions. As a result of the evaluation process, the Evaluation Team shall provide conclusions in their report that speak to programmatic inputs, outputs and outcomes as well as comparisons of efficiency and effectiveness in support to community school development among the CHANGES2, EQUIP2 and QUESTT programs. The Team shall also provide recommendations to USAID about future programming for community schools and the problem of teacher attrition. Finally, the Team shall document lessons learned and present findings. 2. Evaluation Approach The following approaches shall be utilized: In order to evaluate these activities, the DevTech Evaluation Team shall answer several sets of questions related to the effectiveness and impact of FTI program activities under CHANGES2, EQUIP2 and QUESTT. As graphically outlined in Figure 1, discrete data collection instruments shall be used for questions at the program and institutional levels along the education delivery chain (national, provincial/district/zonal, community, school/IRI center, and teacher/classroom). The evaluation model involves a mostly qualitative approach and method of data collection and analysis. Wherever solid quantitative indicators are available, they shall be identified and analyzed, but most of the primary data collection shall be based on interviews and observations, drawing on protocols developed under previous education evaluations conducted by DevTech. During initial meetings in Lusaka with USAID/Zambia, the Evaluation Team shall work with the program personnel to review the scope of the evaluation, the schedule, overall assignments and roles and responsibilities. Concurrently, the Evaluation Team shall also refine and finalize interview and focus group protocols for data collection. The Team Leader shall provide intermittent reports back to USAID as the evaluation unfolds. The preliminary draft shall be presented to USAID and revised based on feedback from USAID. Following the review of that draft by USAID, a final report shall be completed and submitted to USAID/Zambia. The Evaluation Team shall conduct individual interviews and focus groups, particularly with teachers, parents, students and community members who have participated and benefited from small grants, to establish the largest possible data source for their findings and conclusions. Interview and focus group protocols and quality standards shall be revised and adapted by the evaluation team to respond to the Zambian context. The Evaluation Team also shall observe classrooms to evaluate teacher and training activities at the school level. We anticipate using the following approaches: ♦ Document Review. Team members shall review the background documents provided by USAID/Zambia including FTI-related planning documents and reports. These documents usually include a wealth of information and data and shall be crucial for informing the findings and conclusions resulting from this evaluation. The Evaluation Team shall also Annex J: 6 review other documents such as initial project designs, and annual work plans and reports to USAID. The team may also review donor documents pertinent to the evaluation including regional and country educational strategies, poverty reduction strategies, loan papers, and others referring to the education system and the community school sub-sector. The Team’s local Zambian specialists shall be invaluable during this stage of the evaluation. ♦ Data Collection Instruments: The Team shall develop a summary instrument and method of analysis based on the illustrative questions recommended for each program in the SOW. This shall include the adaptation of previously-developed instruments on classroom observation that assess the quality of teaching and training. Whenever possible, the Team shall draw upon data collection instruments used in the monitoring activities of the three programs in order to make subjects more comfortable during the evaluation process. ♦ Site Visits: The Team shall visit representative program sites to verify results. Site visits provide an “on the ground” perspective and allow the team to interact with project beneficiaries to assess how the program was implemented and the quality of that implementation. The Evaluation Team shall use district-level staff as data collectors to facilitate data collection and institutional buy-in required for on-going impact of FTI activities. ♦ Interviews: While using a range of interview protocols, it is of critical importance to be flexible in the interviewing process to follow new directions and leads. The team anticipates interviewing program staff, teachers, school directors, MOE officials at the national, provincial, district, zonal and local levels, and parents, Parents Community School Committees (PCSCs) and children/youth. The interviews provide a chance for the team to evaluate the validity of reports and ascertain the knowledge, abilities, commitment, behaviors and attitudes of individuals to each program’s FTI activities. Contextual conditions, project successes and obstacles, views and experiences of stakeholders, and concrete examples, which may not be found in project or other documents, can often be elicited in interviews. Local research and evaluation studies that may not be known at the national level can also be identified through the interview process. It is also possible to identify patterns throughout the five provinces that may not be found in official documents and that shed light on some of the inherent structural issues in CSs in Zambia. ♦ Focus Groups: The evaluation team shall hold focus groups with community members who have participated and benefited from small grants as well as other program stakeholders. Focus groups and group discussions with program stakeholders shall supplement the information gathered from interviews and documents. The groups often lead into areas not previously considered by either project staff or anticipated by the Evaluation Team. It also permits the team to meet with many more people and obtain a wider set of perspectives than would be true if only individual interviews were held. ♦ Classroom Observations: Classroom observations provide a description of the specifics of the teaching and learning activities that cannot be culled by other methodologies. The Evaluation Team shall observe teaching and training activities to assess qualitative changes at the classroom level on teaching (i.e. are teachers using the “tool-kit”?) and learning (i.e. is Annex J: 7 This page left intentionally blank