Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 i ACKNOWLEDGEMENTS The Evaluation Team from Khulisa Management Services (Pty) Ltd would like to thank the many people that provided support during this program performance evaluation. These include all the key stakeholders involved in the Education Program for Support to Orphans and Vulnerable Children in South Africa. These stakeholders include: - USAID: COTR and Evaluations Manager - KZNDOE: Province and District - Management Agent: Research Triangle Institute (RTI International): - Implementing Partners: MiET Africa and JET Education Services - Eight selected schools visited by the fieldworkers The team appreciates all these stakeholders’ time and information provided during interviews and providing critical feedback on the draft report. The engagement throughout the process was in the form of interviews, briefings; assistance during the fieldwork (logistics and accompaniment by district officials); feedback or inputs or comments on draft and final reports, all of which was valuable. The time set aside by principals; ILST teachers; learners, and parents to engage with the evaluation team is also much appreciated. Khulisa is a South African Monitoring & Evaluation firm founded in 1993. Based in Johannesburg, Khulisa conducts evaluations and assists with developing monitoring systems. Implemented by: Khulisa Management Services (Pty) Ltd 26 7th Avenue, Parktown North PO Box 923 Parklands 2121 Johannesburg, South Africa Email: info@khulisa.com Phone: +27 11 447 6464 Fax: +27 11 447 6468 www.khulisa.com Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 ii EVALUATION OF THE EDUCATION PROGRAM PERFORMANCE EVALUATION OF EDUCATION SUPPORT TO ORPHANS AND VULNERABLE CHILDREN IN KWAZULU-NATAL FINAL REPORT Program Title: Education Support to Orphans and Vulnerable Children in KwaZulu-Natal; Purchase Order Number: AID-674-O-11-00123 Contractor: Khulisa Management Services (Pty) Ltd Date of Publication: October 2011 Author: Khulisa Management Services (Pty) Ltd DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 iii ACRONYMS ABET Adult Basic Education and Training AIDS Acquired Immunodeficiency Syndrome APS Annual Program Statement BEOP Basic Education Operational Plan CBST Circuit Based Support Teams COP10 Country Operational Plan 2010 COTR Contracting Officer’s Technical Representative DBE Department of Basic Education DBST District Based Support Team DCES Deputy Chief Education Specialist DOA Department of Agriculture DOE Department of Education DTT District Training Team EHRD Educational Human Resource Directorate EMIS Education Management Information System EOVC Education Support to Orphans and Vulnerable Children FET Further Education and Training FSS Full Service School GET General Education and Training HIV Human Immunodeficiency Virus HSRC Human Science Research Council ID Identity Document IE Inclusive Education IEPTT Inclusive Education Provincial Task Team ILST Institutional Level Support Team IR Intermediate Result JET JET Education Services KZNDOE KwaZulu-Natal Department of Education LCD Link Community Development LO Life Orientation MEC Member of the Executive Council NCESS National Committee for Education Support Services NCSNET National Commission on Special Needs in Education and Training NGO Non-Governmental Organization NQF National Qualifications Framework OTT Operational Task Team OVC Orphans and Vulnerable Children PEPFAR President’s Emergency Plan for AIDS Relief PMC Project Management Committee PMP Performance Monitoring Plan RTI Research Triangle Institute (trading as RTI International) SIAS Screening, Identification Assessment and Support SICLCS Schools as Inclusive Centres of Learning, Care and Support SMT School Management Team SNES Special Needs Education Service Directorate SOW Scope of Work SPSS Statistical Package for Social Sciences SSRC Special School as a Resource Centre STI Sexually Transmitted Infection TB Tuberculosis TOR Terms of Reference TLS Teaching and Learning Support USAID United States Agency for International Development USG United States Government Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 iv DEFINITION OF TERMS  Inclusive Education At the beginning of 1997, the National Commission on Special Needs in Education and Training (NCSNET) and National Committee for Education Support Services (NCESS) were appointed to investigate and make recommendations on all aspects of special needs and support services in South Africa. The Education White Paper 6 on Special Needs Education: Building an Inclusive Education and Training System outlines how the system should transform itself to accommodate the full range of learning needs and establish a caring and humane society. The following two South African definitions of Inclusive Education are the perspectives of the reference committees and consultative bodies who were commissioned to investigate the future of Special Education. Inclusive Education is defined as a learning environment that promotes the full personal, academic and professional development of all learners irrespective of race, class, gender, disability, religion, culture, sexual preference, learning styles and language. In the Education White Paper 6 on Special Needs Education: Building an Inclusive Education and Training System (2001), Inclusive Education is about:  Acknowledging that all children and youth can learn and that all children and youth need support.  Accepting and respecting that all learners are different in some way and have different learning needs which are equally valued and an ordinary part of our human experience.  Enabling education structures, systems and learning methodologies to meet the needs of all learners.  Acknowledging and respecting differences in learners whether due to age, gender, ethnicity, language, class, and disability or HIV status.  Changing attitudes, behavior, teaching methodologies, curricula and the environment to meet the needs of all learners.  Maximizing the participation of all learners in the culture and the curricula of educational institutions and uncovering and minimizing barriers to learning.  Empowering learners by developing their individual strengths and enabling them to participate critically in the process of learning.  Acknowledging that learning also occurs in the home and community, and within formal and informal modes and structures. The inclusion of learners with 'special education needs' or affected by 'learning barriers' into mainstream classes is part of a universal human rights movement. Inclusive Education addresses the educational needs of all learners in a non-threatening, supportive learning environment, this includes learners who were formally disadvantaged and excluded from education because of 'barriers to learning'. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 v  Structure of education system in South Africa1 South Africa's National Qualifications Framework (NQF) recognizes three broad bands of education: General Education and Training (GET), Further Education and Training (FET), and Higher Education and Training. School life spans 13 years or grades, from Grade 0, otherwise known as Grade R or "reception year", through to Grade 12 or "matric" – the year of matriculation. GET runs from Grade 0 to Grade 9. FET takes place from Grades 10 to 12, and also includes career-oriented education and training offered in other FET institutions – technical colleges, community colleges and private colleges. Diplomas and certificates are qualifications recognized at this level. Table 1: Basic Progression / Structure Age in years National Qualifications Framework Description 0-5 Early Childhood Development Pre-primary school. 6-7 Level 1: Foundation (Grade 0 / Grade Reception) at primary school or nursery. This was made compulsory in 2000 due to numerous early entries into grade 1 and thus high repetition rates. 7-9 Level 1: Foundation Junior Primary School (Grades 1-3). Students learn to read and write in their own language and start to learn another. They also do mathematics. 9-12 Level 1: Intermediate Senior Primary School (Grades 4-6). Studies in the above are deepened and pupils also begin to learn geography, history, general science and practical skills. 12-15 Level 1: Senior Secondary School (Grades 7-9) The focus moves to languages, mathematics, technology, art, natural and social sciences. At the end of Grade 9, students obtain the GET Certificate. This marks the end of compulsory education. 15-18 Levels 2-4 FET provided by Senior Secondary school (Grades 10-12). Pupils must take a public exam in at least six subjects to obtain a Senior Certificate. A number of other bodies also provide FET including 1 http://www.schome.ac.uk/wiki/Education_system_in_South_Africa Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 vi Age in years National Qualifications Framework Description technical colleges which award N3 Certificates. 18-21+ Levels 5-8 Universities offer 3-year long bachelor’s degrees as well as longer courses for vocational training, master’s and doctorate programs. Courses at Technikons vary from 1-4 years and are more vocational than those at universities. Both require Senior Certificates but higher academic achievement is necessary for a place at university. Adults (any age) Equivalent to level 1 ABET (Adult Basic Education and Training) Programmes created to tackle the inadequate schooling of many adults during apartheid. These are largely provided by a wide range of non-governmental organizations, serve a variety of purposes based on individuals’ needs including literacy, numeracy and basic education.  School types in South Africa Education System Ordinary – Public Schools: State supported schools and include a range of school types e.g. primary schools; secondary schools; and combined schools. Special Schools: School for learners with special needs. Farm Schools: Ordinary school located on private farm land. Full Service School: Mainstream education institutions that provide quality education to all learners by supplying the full range of learning needs in an equitable manner.2 2 Department of Education South Africa. Conceptual and Operational Guidelines for the Implementation of Inclusive Education: Full-Service Schools. 2005. p8 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 vii CONTENTS ACKNOWLEDGEMENTS ....................................................................................................................................... I ACRONYMS....................................................................................................................................................... III DEFINITION OF TERMS ...................................................................................................................................... IV CONTENTS ....................................................................................................................................................... VII EXECUTIVE SUMMARY ....................................................................................................................................... 1 INTRODUCTION..............................................................................................................................................................1 PERFORMANCE EVALUATION ............................................................................................................................................1 EVALUATION QUESTIONS, FINDINGS & RECOMMENDATIONS..................................................................................................1 1. INTRODUCTION ......................................................................................................................................... 4 1.1. BACKGROUND ..................................................................................................................................................4 1.2. PROJECT DESIGN AND THEORY OF CHANGE ............................................................................................................5 1.3. PROJECT COMPONENTS......................................................................................................................................6 1.4. PROJECT IMPLEMENTING PARTNERS .....................................................................................................................6 1.5. PROJECT IMPLEMENTATION.................................................................................................................................6 2. EVALUATION APPROACH AND METHODS................................................................................................... 8 2.1 EVALUATION TEAM ...........................................................................................................................................9 2.2 PRIORITY EVALUATION QUESTIONS AND THEMES ....................................................................................................9 2.3 REFERENCE PERIOD OF PERFORMANCE EVALUATION................................................................................................9 2.4 EVALUATION DESIGN AND APPROACH .................................................................................................................10 2.4.1 DATA COLLECTION STRATEGY ............................................................................................................................10 2.4.2 EVALUATION LIMITATIONS ................................................................................................................................11 3. EVALUATION ANALYSIS AND FINDINGS .....................................................................................................12 3.1 PROJECT IMPLEMENTATION........................................................................................................................12 3.1.1 PROJECT CONCEPTUALIZATION...........................................................................................................................12 3.1.2 PROJECT STRATEGY..........................................................................................................................................13 3.1.3 PROJECT IMPLEMENTATION PLAN.......................................................................................................................14 3.1.4 MANAGEMENT AND OPERATIONAL STRUCTURES...................................................................................................15 3.1.5 COLLABORATION AND PLANNING .......................................................................................................................19 3.1.6 PROJECT RESULTS FRAMEWORK AND REPORTING ..................................................................................................20 3.2 PROJECT INTERVENTIONS...........................................................................................................................21 3.2.1 SOCIAL FACTORS INFLUENCING EOVC IMPLEMENTATION AT SCHOOL LEVEL ...............................................................21 3.2.2 UTILIZATION AND BENEFITS OF PROJECT INTERVENTIONS AT SCHOOL LEVEL................................................................24 3.2.3 IMPACT ON LEARNER ACCESS, RETENTION AND PERFORMANCE................................................................................28 3.2.4 THE IMPACT OF THE EOVC PROJECT ON DEPARTMENTAL PROCESSES........................................................................31 3.2.5 SCHOOL‐COMMUNITY RELATIONS & PERCEIVED VALUE OF EOVC PROJECT ...............................................................32 4. ACHIEVEMENTS, CHALLENGES AND RECOMMENDATIONS.........................................................................35 4.1 THEME 1: PROJECT IMPLEMENTATION........................................................................................................35 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 viii 4.1.1 KEY ACHIEVEMENTS.........................................................................................................................................35 4.1.2 CHALLENGES ..................................................................................................................................................35 4.1.3 RECOMMENDATIONS .......................................................................................................................................36 4.2 THEME 2: UTILIZATION, BENEFITS AND EOVC PROJECT IMPACT ..............................................................36 4.2.1 KEY ACHIEVEMENTS.........................................................................................................................................36 4.2.2 CHALLENGES ..................................................................................................................................................37 4.2.3 RECOMMENDATIONS .......................................................................................................................................37 4.3 THEME 3: SCHOOL-COMMUNITY RELATION AND PERCEIVED VALUE OF EOVC PROJECT..........................37 4.3.1 KEY ACHIEVEMENTS.........................................................................................................................................37 4.3.2 CHALLENGES ..................................................................................................................................................37 4.3.3 RECOMMENDATION.........................................................................................................................................38 5. OVERALL CONCLUSIONS............................................................................................................................39 LIST OF TABLES TABLE 1: BASIC PROGRESSION / STRUCTURE V TABLE 2: EOVC PROJECT SCHOOLS 7 TABLE 3: DATA SOURCE PER CATEGORY 10 TABLE 4: LEARNER ENROLMENT TRENDS, DROPOUTS FIGURES AND ATTENDANCE RATES 28 TABLE 5: GENDERED PERFORMANCE RATINGS IN NUMERACY 30 TABLE 6: GENDERED PERFORMANCE RATINGS ON READINGS IN MOTHER TONGUE LANGUAGE 30 TABLE 7: GENDERED PERFORMANCE RATING ON READING IN LANGUAGE OF INSTRUCTION 30 TABLE 8: GENDERED PERFORMANCE RATINGS ON WRITING 31 LIST OF FIGURES FIGURE 1: THEORY OF CHANGE.....................................................................................................................................5 FIGURE 2: OPERATIONAL STRUCTURES .......................................................................................................................16 FIGURE 3: SUMMARY ON LEARNER PERFORMANCE...................................................................................................29 LIST OF APPENDICES APPENDIX A: STATEMENT OF WORK FOR THE EVALUATION OF THE EDUCATION SUPPORT TO ORPHANS AND VULNERABLE CHILDREN IN SOUTH AFRICA 2011 APPENDIX B: WORK PLAN FOR SEPTEMBER - OCTOBER 2011 APPENDIX C: EVIDENCE-BASED AND CONSULTATIVE PROJECT INTERVENTIONS DESIGNED APPENDIX D: PROJECT PARTNER ROLE AND RELATIONSHIP APPENDIX E: DEBRIEFING PRESENTATION TO USAID APPENDIX F: MATRIX: COMMENTS ON DRAFT REPORT ADDRESSED APPENDIX G: DRAFT INFORMATION DISSEMINATION SEMINAR REPORT Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 1 EXECUTIVE SUMMARY INTRODUCTION The United States Agency for International Development (USAID)/Southern Africa commissioned Khulisa Management Services (Khulisa) to conduct an Education Program Performance Evaluation of the Education Support to Orphans and Vulnerable Children (OVC) in South Africa Program (EOVC Project). The EOVC Project was implemented by Research Triangle Institute (RTI Institute) in partnership with the KwaZulu-Natal Department of Education (KZNDOE), MiET Africa and JET Education Services (JET) from 30 April 2009 to 31 August 2011 and had an assigned budget of US$2,399,016, including cost share of US$631,007. Based on the USAID Education Strategy and the KZNDOE’s Inclusive Education policy priorities, the EOVC Project was implemented in Sisonke district of KZN, South Africa to support the improvement of OVC’s access, performance and retention within the system of education. RTI International’s proposal was to address the gap between actually assessed OVC needs and the currently available supply of OVC services to improve learner attainment and retention. The project began with a very comprehensive baseline study which included a household survey and learner testing. From this documentation, a design workshop was held to design interventions. Schools were given a “menu” of these interventions and conditional grants to address their specific needs. When the USAID evaluation policy was launched, the USAID/Southern Africa eliminated the endline comparative survey and commissioned this external evaluation. PERFORMANCE EVALUATION The evaluation assessed the performance of the EOVC Project. The Khulisa evaluation team comprised Mr. Sandile Mbokazi, Ms. Elna Hirschfeld, and was supported by Ms. Jennifer Bisgard. The Evaluation began in September 2011 and USAID required that the team complete fieldwork by the end of the third term (by September 30) and specified a limited sample of schools. Thus, four Umzimkhulu and four East Griqualand schools were purposefully selected from 42 beneficiary schools. The methodology included document review, key informant interviews, and school observations. At schools, discussions were held with principals, teachers involved in the Institutional Level Support Team (ILST), school governing body (SGB) parents including caregivers, and learners. EVALUATION QUESTIONS, FINDINGS & RECOMMENDATIONS The following were the USAID specified evaluation questions and responses: Theme 1: What processes were followed in the implementation of the EOVC Project? How was the EOVC Project implemented at province, district, circuit/ward, and school level? What worked and what did not work during the implementation of the EOVC Project? What are some of the socio-cultural factors influencing – enhancing or inhibiting – the implementation of the EOVC Project? The EOVC Project 24 month timeframe required a clear implementation strategy to ensure optimal use of existing resources. Due to personnel challenges, the need to ensure KZNDOE involvement, technical difficulties with the baseline, mobilization took eight months. During this time, project management and implementation structures were strengthened and/or established at provincial, district, circuit/ ward. This was to ensure that activities are aligned with district priorities and avoid duplication. Further delays were experienced due to the World Cup hosted by South Africa (where the winter break was extended to Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 2 five weeks) and the three week teacher strike, which reduced implementation time even further. The four month no-cost extension was granted only on the final contract day, so while it enabled further activities, this late extension had a negative effect on the project. Institutional capacity building activities took the form of training and ongoing support of the DBSTs as well as the District Training Team (DTT), ILSTs, teachers, and community care givers by MiET Africa’s Project Coordinators in each circuit. Training and learning support were provided primarily on five focus areas. These were: the advocacy and training of district officials, school managers, and members of the ILSTs; caregiver training on how to care for and support children’s education; a multi-sectoral collaboration with relevant government departments and integrated service delivery events and systems; supporting the establishment of learner support mechanisms in each project school focusing on literacy/language and numeracy/mathematics; and working with circuit and ward managers to increase caregiver involvement in the education of vulnerable children. The baseline study documented the numerous socio-economic and cultural issues encountered in education. It highlighted the diversity in the district. From this process, a limited number of socio￾economic interventions were designed and schools were allowed to choose from a 10 intervention menu. However, the project did not address child labor or other cultural inhibitions preventing learner retention. EOVC Project interventions at school level included training that focused on supporting the establishment of learner support mechanisms; providing conditional grants to address OVC needs (e.g. uniforms, educational materials), establishing food gardens; and providing support services to OVC through multi-sectoral collaboration with relevant government departments. The stakeholders were very complementary about the project management structures and there was evidence that some of these structures would continue to function. All schools in the sample showed at least minimal evidence of implementation, especially of food gardens, participating in jamborees, and using the conditional grant to support OVC. Primary schools participated mainly in reading competitions which are being sustained. Implementation was noticeably strongest at provincial/district level where there was evidence of “owning” the project. However, ownership was much weaker at school level. Teachers and caregivers expressed appreciation for the training but the training lacked follow-up, attendance was inconsistent, and therefore it was too early to see evidence of other learning outcomes although there was limited evidence that foundation phase children were reading better. Generally, stakeholders strongly felt that the limited timeline and implementation delays prevented full project maturation. Theme 2: Did schools use the pilot interventions implemented under the EOVC Project in a way that helps reduce learning barriers amongst learners especially OVC learners? If so, in what way? If not explore the challenges faced during the continuation of the EOVC Project. Did the targeted learners benefit? If they did, how – outline the key areas in which the EOVC Project benefited target learners. There is general consensus among informants that the project interventions contributed towards meeting immediate material needs of learners, such as food and uniforms. However, there is still much room for improvement in terms of mitigating the social and cultural factors charactering the school-community contexts. All learners were considered vulnerable, and thus the Project did not address only OVC. Retention and access was reportedly improved by school uniforms; books, stationary; sanitary pads; food; and knowledge that learners and/or families could ask the schools for help. However, stigma was still an issue. During Jamborees, learners and their families received direct assistance from other governmental departments (e.g. birth certificates, identity documents (IDs), social development grants, seeds and tools, etc.). Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 3 The main benefit was learner exposure to array of interventions were primarily material (food, clothing, etc.), but only three of the eight schools visited could show documentation on who received what services. There is not enough evidence to state that educational outcomes were improved. Theme 3: How is the EOVC Project perceived and valued? Was there a more involved relationship amongst parents, community and schools to support learning especially for OVC? The Project was highly valued by the KZN Province, but less so by the District, although still valued. Schools valued the additional resources (training, gardens, and conditional grants). Caregivers and/or parents and learners appreciated the uniforms and food provided, but accountability chain was weak, only real accountability was for conditional grants. The schools differ in terms of the progress they have made to build and strengthen the relationship with parents and community in supporting learning for OVC. The community mapping exercise has worked for a few schools and they have managed to forge relationships with existing community structures that have resources to support OVC. Such structures include local churches, community centers and other households in the form of food, clothes, and blankets. In one context, churches conducted house visits to provide such support twice weekly to OVC. So, though such a relationship has increased, it was not sustained, and there was still limited involvement. A more involved relationship has been achieved at most selected project schools, but their experience of this relationship appears to be in a continuum that ranges from poor involvement (which was also linked to poor support from SGBs) to a much stronger involvement. Jamborees, especially the benefit it made to community members, appear to have played an important role in encouraging this strong relationship. Theme 4: What can the KZNDOE take forward to scale-up the EOVC Project? First of all, it is important to note that KZNDOE is committed to implementing the IE policy and has participated in sustainability planning to carry on with project activities. However, it was clear that without an implementation agency (performing the joint role, played by MiET Africa and RTI International) it is unlikely to succeed. If replicated, it will be crucial to provide with more intensive training and support of ILST and link accountability for outcomes (particularly training results attainment). When implementing, it is crucial to differentiate between schools and their implementation ability and only provide resources and training to schools that are ready and accountable. KZNDOE needs to:  Clarify roles and responsibilities especially of the Full Service and Special Schools.  Strengthen sustainability plan (time, resources, M&E, reporting). The current Sustainability Plan of the district needs to show clear mechanism for resilience, evaluation, monitoring, and accountability, as well as reporting procedures.  Put in place at structures at provincial, district, and school levels to build on the multi-sectoral efforts to support OVC. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 4 1. INTRODUCTION Khulisa Management Services (Khulisa) was commissioned by the United States Agency for International Development (USAID)/Southern Africa to conduct a Education Program Performance Evaluation of the Education Support to Orphans and Vulnerable Children (OVC) in South Africa Program (EOVC Project) APS No. 674-08-011. The EOVC Project was implemented by Research Triangle Institute (RTI International) in partnership with the KZN Department of Education (KZNDOE), MiET Africa and JET Education Services (JET). The EOVC Project was implemented from 30 April 2009 to 31 August 2011 and had an assigned budget of US$2,399,0163, including cost share of US$631,0074. 1.1.BACKGROUND In 2008/09 at the time of the drafting of the USAID Annual Program Statement and the conceptualization of the EOVC Project, HIV prevalence declined by more than 25% amongst the 15-24 age groups in thirteen African countries including South Africa5. However, in South Africa, the number of orphaned children as a percentage of all children has risen steadily since the onset of HIV/AIDS. The number of orphaned children as a percentage of all children was estimated to be between 5% - 7% before the HIV epidemic; it had risen to an estimated 12% in 2003, and is predicted to increase to 18% by the year 20126. Many studies have evaluated the consequences of being orphaned or living in vulnerability and have investigated the barriers to OVC school attendance. These include the need to stay home to care for sick and dying parents, care for younger siblings, teenage pregnancy resulting from abuse, etc. However, even if the barriers to school attendance were lessened, if the barriers to successful school performance are not removed, OVC will continue to leave school at an early age without the basic skills needed to succeed in adult life. Education White Paper 6 on Special Needs Education: Building an Inclusive Education and Training System (2001) – (White Paper 6)7 of the national Department of Basic Education (DBE) established the framework for inclusive education and training in South Africa. The purpose is to “…provide educational opportunities to those learners who experience or have experienced barriers to learning and development or who have dropped out of learning because of the inability of the education and training system to accommodate their learning needs”8. The EOVC Project design was guided by the operating premise that: i) although there is information on barriers and catalysts to education access among OVC, factors related to school performance among OVC are not well documented; ii) data related to barriers and/or catalysts to learning among OVC will inform the development of targeted interventions to support learning among OVC; iii) learning support interventions will add value to KZNDOE Inclusive Education (IE) Strategy and the Department’s 3 Budget total excludes the amount of US$100,00.00 used by USAID to conduct the external Performance Evaluation of the EOVC Project. 4 Draft EOVC Project Accruals Report - 31 August 2011. RTI International (One page statement provided to us via email from RTI International) 5 UNAIDS 2010: Outlook Breaking News. Young People are Leading the HIV Prevention Revolution.p3. UNAIDS Joint United Nations Programme on HIV/AIDS. 20 Avenue Appia, CH-1211 Geneva 27, Switzerland 6 Our Children our Future. From Vision to Innovative Impact. Community responses to Orphans and Vulnerable Children. A Census Report of Orphaned and Vulnerable Children in Two South African Communities. Sean Jooste, Azwifaneli Managa, Leickness C. Simbayi. 2006 p 7 Human Research Council. Cape Town // Sean Jooste, Azwifaneli Managa, and Leickness Simbayi, A Census Report of Orphaned and Vulnerable Children in Two South African Communities (Cape Town, South Africa: Human Science Research Council [HSRC], 2006), p. 7. 7 Education WHITE PAPER 6 Special Needs Education Building an Inclusive Education and Training System July 2001 8 Education WHITE PAPER 6 Special Needs Education Building an Inclusive Education and Training System July 2001 Executive Summary, p6 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 5 Schools as Inclusive Centres of Learning, Care and Support (SICLCS); iv) embedding activities into the KZNDOE IE Strategy will ensure success and sustainability, and will avoid redundancy9. The EOVC Project, implemented in Sisonke district of KZN, South Africa aimed to improve learner performance, access and retention among OVC in selected primary and secondary schools. This report documents the findings of the evaluation of the EOVC Project. 1.2.PROJECT DESIGN AND THEORY OF CHANGE As referred to in par 1.1 the EOVC Project design was guided by four main operating premises. The intended project theory of change as outlined in RTI international’s proposal was to diagnose and then address the gap between actually assessed needs of OVC and the currently available supply of OVC services. The proposed impact was to improve all learner attainment and retention through the implementation of the IE Strategy thus benefiting all learners including OVC.10. The overarching project goal was to improve performance, access and retention among OVC in both primary and secondary schools in the Sisonke District of KZN. The EOVC Project’s wider aims included: i) increasing the number of OVC participating in learning support programs in the target area; ii) enabling improvement in performance of OVC in the target schools and community; iii) contributing to the current knowledge about real learning challenges facing OVC in a learning environment and supporting government strategies to address these in practical and cost￾effective ways; and iv) strengthening educational policy related to IE, care and support for OVC and address identified barriers to performance and retention11. Figure 1 reflects a simple project model or Theory of Change, with project interventions at the centre of the picture. It shows that improvement in learner performance will occur as a result of home-grown project interventions, which in turn are the product of all the important project inputs namely, advocacy, research, training, multi-stakeholder collaboration, and conditional grant funds. Project premises were declared in the project proposal and the following assumptions relating to the implementation of the EOVC Project were identified: i) factors relating to school performance of OVC not well-documented; ii) data on barriers/catalysts to learning among OVC would inform the targeted interventions; iii) learning 9 Education Support to OVC in South Africa (2009-2011). Reducing barriers to learning for orphans and vulnerable children (OVC) in KwaZulu-Natal. . Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. JET Education. MiET Africa.. p. 1. 10 Education Support to OVC in South Africa. Tier II Application–Revised. Attachment 2 674-A-00-09-00054-00. February 19, 2009. RTI International. P.O. Box 12194 Research Triangle Park, NC 27709-2194. p 2 11 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. p. 9 Figure 1: Theory of Change Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 6 support interventions will add value to KZNDOE’s IE strategy and SICLCS; and iv) embedding activities in IE strategy will ensure success & sustainability, avoiding redundancy12. 1.3.PROJECT COMPONENTS A three-pronged project approach was designed for implementation of the EOVC Project over the project period i) conducting of a baseline study to assess learner achievement in literacy, numeracy, language and mathematics in grades 3 and 10 as well as gathering real-life data on the child’s learning experience through learner, school, and household interviews to inform design if project interventions; ii) designing of interventions specifically to improve the OVC performance; and iii) implementation of interventions and learning support activities in the various schools as an integral part of the KZNDOE’s IE Strategy. Annex D contains a table entitled “Evidence-based and Consultative Project Interventions Designed” with the short-list of proposed. The implementation of the interventions is discussed in more detail in par 3.2 of the report. 1.4.PROJECT IMPLEMENTING PARTNERS USAID/Southern Africa awarded a Cooperative Agreement to RTI International in April 2009 to be implemented in close partnership with the KZNDOE. RTI International, Implementing Partner, led and coordinated the implementation of the EOVC Project through its offices in Pretoria. RTI International liaised with KZNDOE and USAID/Southern Africa, provided project direction, managed the project budget, prepared project reports and monitored district and school-level activities through regular field visits. Implementation on the ground was the task of local partner, - MiET-Africa. JET, with support from RTI International and MiET-Africa, implemented the initial assessment and household survey – referred to as the baseline study. The University KZN was also engaged briefly to document some of the lessons learned emerging from the EOVC Project. Refer to Annex C for a diagram outlining the various roles and relations of the EOVC Project partners. 1.5.PROJECT IMPLEMENTATION The EOVC Project was implemented over a period of 28-months - 30 April 2009 to 31 August 2011. The initial area targeted for implementation was Umzimkhulu Circuit within the newly incorporated Sisonke District situated in the south-western corner of KZN province. On request of the KZNDOE, the East Griqualand Circuit was included in the EOVC Project in June 2009. It was motivated that the IE Strategy has already started within one of the wards in this circuit and by including the Greater Kokstad Ward, the EOVC Project would be able to demonstrate firstly how the EOVC Project can add￾value and enhance the role of SICLCS; and secondly show the extent to which supportive interventions are able to open gate-ways for improved learning amongst vulnerable children working hand in hand within government strategy. The schools identified for inclusion in the Umzimkhulu and East Griqualand Circuit were selected by the KZNDOE. The EOVC Project interventions were implemented in 42 project schools primarily from the Mkhatshana Ward in Umzimkhulu Circuit and from the Greater Kokstad Ward in the East Griqualand Circuit. 12 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. Annex C. Information Dissemination Seminar Report. p5. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 7 Table 2 provides a summary of the schools’ profiles included in the EOVC Project. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 8 Table 2: EOVC Project Schools School Profiles Umzimkhulu East Griqualand Total Primary Schools (Grades R – 7): 3 21 24 Junior Secondary (Grades 8 – 9) incl FSS 9 0 9 Senior Primary (Grades 5-7) incl FSS 0 1 1 Senior Secondary Schools (Grade 10-12): 2 0 2 Secondary Schools (Grades 6-12) 0 3 3 Combined (Grades R-12) 1 1 2 Special Needs School 1 0 1 Total 16 26 42 In addition, broad-level policy training on the IE Strategy was provided to an additional 168 schools in Umzimkhulu largely through the DBST structure supported by the implementing partner, MiET Africa. Map 1 below outlines the geographical location of all the schools selected for participation in the EOVC Project. In addition, broad-level policy training on the IE Strategy was provided to an additional 168 schools in Umzimkhulu largely through the DBST structure supported by MiET Africa. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 9 2. EVALUATION APPROACH AND METHODS The purpose of the evaluation is to assess the performance of the EOVC Project with specific reference to project achievements; how it was implemented against the project design, management and operational decisions taken; how the project is perceived and valued; and whether the expected project results were achieved. The evaluation fieldwork assessed which interventions were applied and how these were applied; and if they had an impact on learners and, in particular, on the more vulnerable learners; what can be learned from the project for replication in other schools in the area by the KZNDOE. Map 1: Map of Project Schools and Schools included in Performance Evaluation Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 10 2.1 EVALUATION TEAM The evaluation team comprised of Sandile Mbokazi, Team Leader and Elna Hirschfeld supported by Jennifer Bisgard. Mr. Mbokazi is from KZN. Each member of the evaluation team has more than 10 years of experience in conducting research and evaluations in various settings – urban, rural, including farm communities and using an array of techniques including qualitative methodologies, participatory and real world approaches to evaluation as applied within in the education sector. 2.2 PRIORITY EVALUATION QUESTIONS AND THEMES There are ten questions that guided this evaluation. While the USAID/Southern Africa specified nine critical questions in the evaluation terms of reference (TOR)13, the evaluation included the tenth one. The answers to all these questions have been organized into three main themes, in addition to the recommendations, challenges and lessons learnt: 1. Project Implementation: What processes were followed in the implementation of the EOVC Project? How was the EOVC Project implemented at province, district, circuit/ward, and school level? What worked and what did not work during the implementation of the EOVC Project? A further question was added with concurrence from USAID/Southern Africa, and this formed part of the first theme on the implementation of the Project. This is: What are some of the socio-cultural factors influencing – enhancing or inhibiting – the implementation of the EOVC Project? 2. Utilization, Benefits and Impact of the EOVC Project: Did schools use the pilot interventions implemented under the EOVC Project in a way that helps reduce learning barriers amongst learners especially OVC learners? If so in what ways? If not explore the challenges faced during the continuation of the EOVC Project. Did the targeted learners benefit? If they did, how – outline the key areas in which the EOVC Project benefited target learners. Whether the interventions have had any positive influence on learner access and retention? 3. School-Community Relations and Perceived Value of the EOVC Project: How is the EOVC Project perceived and valued? Was there a more involved relationship amongst parents, community and schools to support learning especially for OVC? 4. Recommendations: What can the KZNDOE take forward to scale-up the EOVC Project? These evaluation questions were further developed in the Data Collection Framework and Analysis Plan contained in the Workplan (Annex B) approved by USAID/Southern Africa on 13 September 2011. 2.3 REFERENCE PERIOD OF PERFORMANCE EVALUATION The Performance Evaluation was conducted during September and October 2011, following EOVC Project closure on 31 August 2011. 13 Terms of Reference of Performance Evaluation is attached as Annex A to the report Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 11 2.4 EVALUATION DESIGN AND APPROACH 2.4.1 DATA COLLECTION STRATEGY There are a number of key stakeholders involved in the implementation of the EOVC Project that were interviewed as a critical component of the evaluation process. Eight schools were purposefully selected from the 42 schools involved in the EOVC Project based on the following stratification criteria:  Four schools were selected from the Umzimkhulu and East Griqualand circuits;  Schools were chosen to represent categories of schools e.g. primary; secondary; combined schools; special school; and full service schools; and geographical location, such as farm, rural; and urban. In addition to reviewing multiple documents, at each of the schools, in-depth interviews and observations were conducted to understand the ways in which the EOVC Project has impacted on school￾communities. These were conducted with principals, teachers involved in the Institutional Level Support Team (ILST), school governing body (SGB) parents including caregivers, and learners. Table 3: Data Source per Category Stakeholder USAID/Southern Africa KZNDOE RTI JET MiET Africa OTT PRINCIPALS TEACHERS PARENTS (SGB) LEARNERS N#.  134 1 1 1 2 1 4 8 23 24 69 In addition to the interviews with a number of informants, about 45 documents were reviewed. The aim of the document review and interviews was to strengthen answers to critical questions and to develop useful recommendations for the KZNDOE to better respond to the educational needs of OVC. The conversations with these informants was voice recorded and some photographs taken linked to observations made for better capturing of information in each school. Learners were identified as an important source. The evaluators obtained formal consent from principals, and facilitated discussion through age appropriate activities. The evaluation team minimized disruption of learning and teaching. The qualitative data was analyzed and reported in conjunction with the some statistical data gathered during the evaluation, in order to produce the evaluation report. The analysis drew largely from the various data sources as referred to in Table 3 above. The methodology was designed to enable triangulation of findings from a range of sources within the scope of this Performance Evaluation. Findings from the evaluation process were compiled into a comprehensive report. Analysis of all findings was a joint team effort, as was the writing of the evaluation report. To this end daily summaries and team meetings were conducted to enable initial analysis. The following methodologies were used:  Document review of 45 documents from USAID/Southern Africa, RTI International, MiET Africa, KZNDOE and other relevant literature which are listed in Endnote and References section.  Key informant interviews (KII) conducted with representatives from USAID/Southern Africa, RTI International, KZNDOE provincial and district and the implementation partners. Who was interviewed is attached as part of the Workplan in Annex B.  Focus group interviews were conducted with district, circuit and school level informants on their experience of the EOCV Project.  School profile data to collect basic statistical information at the selected schools was administered. The data received from the eight schools were verified against district level Education Management Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 12 Information System (EMIS) data. The information gathered through this technique was captured through Statistical Package for Social Sciences (SPSS), a statistical package, for better analysis.  Performance ratings: The 69 learners that participated in focus group interviews were rated by their class teachers on how they performed in numeracy, reading in mother tongue language, reading in the language of instruction, and in writing.  Appreciative Inquiry technique was used when engaging parents from the SGBs, caregivers, community members and learners.  Participatory process: The Performance Evaluation was designed to allow for some participation of key stakeholders in the EOVC Project. Stakeholders were invited to make inputs on the evaluation tools and to shape the workplan; district officials accompanied the evaluation team during school visits; initial findings and challenges identified were presented to the Sisonke District team; and provided with the opportunity to make recommendations for inclusion in the report. 2.4.2 EVALUATION LIMITATIONS The Performance Evaluation had some limitations, and these related to the following:  USAID/Southern Africa specified the small sample size due to financial and time constraints (in particular the need to complete fieldwork by the end of the third term on September 30, 2011) with limited ability to generalize the data. This was mitigated to some extent by ensuring that the sample was purposefully selected as per 2.4.1 above.  The absence of relevant baseline information to inform the performance evaluation. Though project baseline data was made available to the research team the TOR excluded learner assessments as was done in the baseline study resulting in limited usability of this data in this evaluation. The school specific baseline data was not formatted in a way that could be utilized.  The limited time allocated for the Evaluation allowed very little participation of stakeholders and restricted the number of engagements with government and other stakeholders. It also prevented the pretesting of the research tool. While 10 learners were interviewed in each of the 7 schools visited, the technique did not work for the special school, because the evaluation tool did not speak to their specific needs. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 13 3. EVALUATION ANALYSIS AND FINDINGS The Performance Evaluation was guided by ten critical questions as outlined in 2.2 above. These questions were grouped in broad themes for purposes of analysis and presentation of findings and recommendations. The themes were project implementation; utilization, benefits and impact of the EOVC Project; as well as the perceived value of the project and school-community relations: 3.1 PROJECT IMPLEMENTATION Four of the evaluation questions relates to project implementation as a broad theme and includes reference to the conceptualization context in which the project emanated from; the processes followed in the implementation of the project with specific reference to the project strategy and implementation plan as well as the timeline. Lastly it focused on the various implementation structures assigned with the implementation of the project. The related question on what worked and what did not work during the implementation of the project is outlined in Section 3 of the report. 3.1.1 PROJECT CONCEPTUALIZATION In 2008/09 USAID/Southern Africa and DOE aimed to scale up activities to expand the U.S. Government (USG) support in education on HIV and AIDS programs. Envisaged integrated health and education interventions were to align with the goals of the Annual Program Statement (APS) of USAID/Southern Africa and contribute towards the achievement of the HIV/AIDS and sexually transmitted infection (STI), Strategic Plan for South Africa, 2007-201114. The scale-up focused on five focus areas including support to OVC within the DOE strategy to use schools as full service centers for learning, teaching, prevention, care and support. The EOVC Project proposal was based on the SA Education White Paper 6 on Inclusive Education15 framework by reducing barriers to learning due to HIV/AIDS, and work with targeted structures and mechanisms to improve the retention and participation rates of learners. The EOVC Cooperative Agreement was awarded to RTI International on 30 April 2009 by USAID/Southern Africa and implemented in partnership with KZNDOE. When the proposal was conceptualized, the funding was projected to be channeled through President’s Emergency Plan for AIDS Relief (PEPFAR). However, when funding was received, the EOVC Project only received 20% PEPFAR with the remaining through rescheduled funds16. The project award also reduced the timeframe from three to two years. The shift in focus from a PEPFAR OVC focus to an education project with an OVC focus created implementation difficulties. The tension between the time limitation and funding available on the one hand and the need for a longer term approach to assess the impact of the interventions was evident throughout the implementation of the project. This became one of the biggest risk factors threatening the long term sustainability of the EOVC Project. 14 US Government Mission to South Africa Annual Program Statement in Support of Education in South Africa. APS 674-08-001. p. 1-2 15 Department of Education. Education White Paper 6 Special Needs Education Building an Inclusive Education and Training Workshop July 2001 16 Project received approximately 17% of its obligated funds from PEPFAR. Education Support to OVC in South Africa. Annual Work Plan: May 2009 – April 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. p. 16. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 14 3.1.2 PROJECT STRATEGY The project goal was to inform the KZNDOE IE Strategy and implementation framework through the piloting of evidence-based interventions focusing on the impact that vulnerability has on learner performance in schools in Sisonke District and how best to counter this impact17. In response to the project strategy the EOVC Project was designed as outlined in 1.3 above along the implementation of three primary activities: i) conducting a baseline study; ii) piloting interventions; and iii) engaging in ongoing dialogue with the KZNDOE and other stakeholders on policy recommendations emerging. The baseline study focused on obtaining evidence to inform project interventions. EOVC Project activities were aimed to create a more enabling environment for learning through training interventions directed at stakeholders at various levels including teachers, caregivers and parents, and district officials. The baseline report analysis led to piloting interventions and facilitating multi-sectoral collaboration with relevant government departments to begin setting up integrated service delivery events and systems. Though the finalization of the baseline study was delayed, the preliminary findings were used in the March 2010 Intervention Design Workshop to identify and formulate pilot interventions. The final results and recommendations of the baseline study were published and distributed at the project Information Dissemination Seminar in August 201118. The study – as became evident through a range of KIIs conducted --was regarded as an essential building block in guiding project interventions to ensure they are well grounded within the KZNDOE’s IE Strategy. The KZNDOE, especially, is highly appreciative of the collaborative design of the project interventions on the basis of evidence, experience and real beneficiary needs19. . The pilot interventions identified through a multi-stakeholder collaboration process and directly linked to the findings of the baseline study were grouped into four main categories of i) training and support; ii) school food gardens; iii) conditional grants; and iv) inter-departmental cooperation and service delivery events (referred to as Jamborees). Following from the Intervention Design Workshop a Task Team was assigned to translate the interventions into practical activities for schools and the district. The beneficiary schools were given the option to select and implement those interventions most suitable for their context and need. The various socio-cultural and contextual factors that influenced the implementation of the various interventions are discussed in the relevant section below. Though the pilot interventions were designed to benefit OVC, the improvement in teaching or school management within the framework of the IE Strategy benefitted all learners in the school regardless of their vulnerability status. EOVC Project activities were mainly focused on primary and mainstream schools as only five of the schools included in the project were secondary schools (excluding the combined schools) and only one was a special school as outlined in 17 Education Support to OVC in South Africa (2009-2011). Initial Assessment and Household Survey. August 2010 Revised. Funded by USAID/Southern Africa. RTI International. JET Education. MiET Africa. Department Education KwaZulu-Natal. Department of Basic Education. p. xiii 18 Education Support to OVC in South Africa (2009-2011). Initial Assessment and Household Survey. August 2010 Revised. Funded by USAID/Southern Africa. RTI International. JET Education. MiET Africa. Department Education KwaZulu-Natal. Department of Basic Education. 19 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. p.55 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 15 Table 2. The training strategy followed - due to funding and time limitations, was to use a cascade model (that is, to task the trained participant to train the other teachers in their school). The planned post-evaluation impact study was to comprise qualitative case studies and an endline survey/learner testing was to provide feedback and recommendations20. It was replaced by this independent project performance evaluation as required by the new USAID Evaluation Policy21. However, a range of ongoing project activities provided constant policy feedback to the Inclusive Education Provincial Task Team (IEPLT)22 aimed at strengthening the KZNDOE IE Strategy. The research conducted by a team of School of Education Honors students from the University of KZN, (Pietermaritzburg Campus) also documented initial lessons learned and best practices emerging from the EOVC Project23. In addition, a 360 degree review of the project was conducted to shed light on project lessons and provide recommendations for the KZNDOE to consider in the strengthening and roll-out their IE Strategy. During the Information Dissemination Seminar, RTI International presented EOVC Project highlights, challenges as well as emerging project lessons including policy implementation lessons, identified through the 360 degree review to the various stakeholders engaged in the project24. A significant sustainability component of the project lies in the fact that project activities were embedded in KZNDOE’s IE strategy from the start of the project. Project activities implemented in partnership with departmental counterparts directly impacted positively on the implementation framework of the strategy. 3.1.3 PROJECT IMPLEMENTATION PLAN The intention – as outlined in the APS, was to award projects for a period of eighteen months to two years with a possibility of an optional one year extension depending on the recipient’s performance and the availability of funds25 - thus an anticipated project period of three years. The final agreement entered into based on a revised scope of work linked to the funding available provided only for the implementation of the EOVC Project over a two year period. The EOVC Project implementation timeframe of 28 months (including the no-cost extension of an additional four months) required a clear implementation strategy to ensure optimal use of existing resources for the implementation of activities and the assurance that the EOVC Project is well embedded in the IE Strategy of the KZNDOE for longer term sustainability. To achieve this, the first six months of the EOVC Project (April – Sept/Oct 2009) focused on planning, advocacy, orientation, and readiness training as well as the planning and preparation required for the baseline study scheduled for September-October 200926. The Intervention Design Workshop was scheduled to take place in December 2009 but due to the delay in the finalization of the schools identified for inclusion in the baseline study, the sampling framework; school examinations and school holidays, the finalization of the baseline study was delayed. In consultation with KZNDOE, the Intervention Design Workshop was rescheduled and held in March 2010. The agreed identified learning support interventions were introduced in the 42 pilot schools in May 2010. At this point there was only 11 months left of the 20 Education Support to OVC in South Africa. Annual Work Plan: May 2009 – April 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. p.17 21 Evaluation. Learning from Experience. USAID Evaluation Policy. January 2011. Washington, DC. 22 Minutes of the IEPTT meeting. 23 Draft Report Case Studies on Four Areas of Education Support to OVC Project. KZN University. RTI International. 24 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. p. 53 25 US Government Mission to South Africa Annual Program Statement in Support of Education in South Africa. APS 674-08-001. p. 2 26 Education Support to OVC in South Africa. Annual Work Plan: May 2009 – April 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. p.8 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 16 project (including school holidays and examinations, etc.) in which to also conduct the envisaged project impact evaluation and commence with project closure activities. The EOVC Project implementation plan was tight with little contingency time available as became evident in 2010 with several delays experienced with the implementation of project activities. Delays in finalizing the baseline results (approximately 12 weeks), the 5-week-long winter break during the FIFA World Cup™, as well as the protracted public service strike and the post-strike recovery period – a loss of nine weeks, all contributed to a loss of about 26 weeks (6.5 months) in project implementation time. Though big strides were made through the commitment of project staff to maximize the time remaining it was indicated from a project management perspective that it negatively impacted on the uptake of some learning support interventions; to bring about sustainable change in classroom practices; and especially impacted negatively on the activities planned with the secondary schools27. From July 2010, secondary schools focused on catch-up programs and the Grade 12 senior certificate examinations. A motivation for project extension was submitted to USAID/Southern Africa] by the KZNDOE as part of the RTI International’s quarterly report submitted to USAID/Southern Africa in July 201028. During this period, the project team also considered requesting a no-cost extension to recoup time lost, complete planned tasks and give EOVC Project schools the chance to implement the learning support activities and to show the impact these can have in rolling out best practices emerging from this project. According to the USAID Activity Manager the “nature of the Cooperative Agreement” prohibited an extension of the existing contract, but the evaluators could not confirm such a clause was included. RTI International, KZNDOE and the other partners were surprised when this statement was repeated, as they felt that they had actually been encouraged to submit such a request. No formal response on the EOVC Project funding extension motivation was provided to KZNDOE, although the Activity Manager stated that she had communicated regrets informally. During interviews with various key informants the lack of a formal response rankled. In January 2011, a request for a no-cost extension for an additional four months (through to August 2011) was submitted to USAID/Southern Africa. In the absence of a clear indication from USAID on whether a no-cost extension might be granted, project implementation was marked by a flurry of training activities. This uncertainty persisted, with the formal no-cost extension granted on the final day of the Cooperative Agreement in April 2011 following a request from USAID/Southern Africa to revise the workplan and budget to exclude the envisaged impact study. Time is a critical component of ensuring project sustainability as was highlighted in the presentation of the Sustainability Plan of the District Office during the Information Dissemination Seminar in August 2011: “However, with only just under six months of actual activity time available, it is felt that there was insufficient consolidation of the interventions at school level for them to produce the desired impact. The Sustainability Plan therefore outlines the actions and activities that will be undertaken.” (District Office Presentation – August 2011). This was echoed by MiET Africa that indicated the project had an insufficient lifespan to accomplish what it set out to do, particularly in the area of improving learner performance. During KIIs conducted with principals and teachers of the ILSTs it also became evident that it would have been ideal to continue with the EOVC Project for some time e.g. 5 years as it provided an ongoing learning opportunity for teachers to become better classroom practitioners29. 3.1.4 MANAGEMENT AND OPERATIONAL STRUCTURES 27 Education Support to OVC in South Africa. Annual Report: 1 October 2009 – 30 September 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. p.16. 28 Education Support to OVC in South Africa. Quarter Report: July - September 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. 29 ILST Teachers of Nombewu FSS Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 17 One of the project design operating guiding premises was that embedding activities into the KZNDOE IE Strategy and operational structures will ensure success and sustainability and avoid redundancy as outlined in Figure 2. For project activities to be sustainable - as outlined in the Annual Workplan of year two, government counterparts must see that project benefits exceed costs, and local institutional resource capacities exist to continue to support and manage activities30. Project management and implementation structures were strengthened and or established as a priority activity at provincial, district/circuit, ward and school levels for the purposes of project management and governance ensuring activities are aligned with district priorities and avoiding duplication. Institutional capacity building activities at project implementation level took the form of training and ongoing support of the DBSTs including the District Training Team (DTT), ILSTs, teachers, and community care givers by MiET Africa’s Project Coordinators in each circuit. Training and learning support were provided primarily on five focus areas31. Firstly, the advocacy and training of district officials, school managers, and members of the ILSTs was provided. Secondly, caregiver training on how to care for and support children’s education was conducted. Thirdly, a multi-sectoral collaboration was facilitated with relevant government departments and integrated service delivery events and systems were setting up. Fourthly, supporting the establishment of learner support mechanisms was done in each project school including academic support focusing on literacy/language and numeracy/mathematics. Finally working with circuit and ward managers to increase caregiver involvement in the education of vulnerable children. The Project Management Team32 (PMT) included in addition to the project team the Umzimkhulu District team from Umzimkhulu and Greater Kokstad. The PMT met bi-monthly during the first 6 months of the project to monitor project progress and make strategic decisions but was during the end part of the project absorbed into the IEPTT structure – chaired by the Special Needs Education Service Directorate (SNES). The IEPTT discusses all care and support projects in KZN, including inclusive education and only meets quarterly. This led to the establishment in February 2010 to the district/circuit level Operational Task Team (OTT) chaired by Deputy Chief Education Specialist (DCES) of the District Office. The OTT met at least once a month to make day-to-day operational decisions about the implementation of the EOVC Project and resolve any problems arising. It was anticipated that the establishment of the OTT will help in the decision-making processes and expedite implementation of the EOVC Project by integrating the project as part of the district’s planning and implementing. The KZNDOE played a decisive role in the design of the EOVC Project and was central in linking project activities to the implementation of education policies, in this case – IE Strategy of KZNDOE. By doing so ensuring support from the district and to collaborate the support of other units relevant to project activities e.g. Curriculum sub-directorate that is responsible for literacy and numeracy. The project’s Chief of Party (COP) highlighted the establishment of a project management structure – the PMT and the later IEPTT, was led by KZNDOE’s SNES Directorate and placed the EOVC Project 30 Education Support to OVC in South Africa. Annual Work Plan: May 2009 – April 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education.p.6. 31 Education Support to OVC in South Africa (2009-2011). Reducing barriers to learning for orphans and vulnerable children (OVC) in KwaZulu-Natal. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. JET Education. MiET Africa. p. 3 32 Also referred to as the Project Management Committee - PMC Figure 2: Operational Structures Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 18 squarely within the government structure ensuring it is implemented as a government initiative. Members of the EOVC project team confirmed that the IEPTT did devote quality time to the issues of EOVC Project and the establishment of the OTT assisted greatly in making ongoing operational decisions. Members of the OTT interviewed indicated that the OTT was able to facilitate the processes – “to iron out relationships” with project implementers – including MiET Africa and Link Community Development (LCD) at operational level and ensured that school-level challenges were addressed at this level without it ever reaching the provincial level. A senior manager from the District Office emphasized the OTT structure created a successful link between the project implementers and the KZNDOE as it created a platform for joint planning; the formulation of reports etc. with the main ownership residing with the district. “The OTT was established to facilitate the EOVC Program. It met and discussed reports before they are presented to the province.” (KII: OTT member). In addition to the establishment of circuit based support teams (CBST), the preparation of schools for the pilot interventions commenced by establishing and training the district based support team (DBST) chaired by the Sisonke District Manager. The main focus of the DBST team’s DTT training was to capacitate the team to assist and run provided training and support to ILST members and school principals and to lead the roll-out of the IE Strategy in the entire Sisonke district including the 168 schools in the Umzimkhulu circuit33. Through the engagement with and capacity building of the DBST – DTT ownership was vested in the roll-out of the IE Strategy in the schools in Sisonke. Before implementing the interventions, there was capacity building of key institutions in the EOVC Project, such as the DBST. Once the DBST had been trained, school preparation got underway with this cadre making up the training team. The sequence of training of schools (usually the principal and two ILST members) followed the sequence of modules delivered to the DBST during their training – a strategy demonstrating the important underlying principle of working with and within departmental structures; enhancing these structures and reinforcing sustainability. According to OTT members interviewed the training of district officials was successful and gave them confidence to extend the training to other schools in the district as was done with the Inclusive Education training conducted by the DBST for the 168 schools. As part of the training DBST officials were provided with an IE Essential Pack of the KZNDOE. A key part of ensuring that the IE Strategy was implemented effectively was to establish functional ILSTs in all schools. At the beginning of the EOVC Project almost half of the Umzimkhulu schools had not yet established ILSTs34. The ILST is made up of a group of teachers and school managers tasks with i) Whole School Management; ii) Educator Support; and iii) Learner Support under the chairmanship of the school principal. The Learner Support Committee was tasked with the implementation of the EOVC Project interventions at school level. Training workshops with the ILSTs aimed to increase their capacity to mentor and support OVC learning and retention as well as to strengthen the institutional capacity and education system to better respond to the learning support needs of OVC In addition to 33 Education Support to OVC in South Africa. Final Report (Draft). 31 States Agency for International Development. It was prepared Department. Education KwaZulu-Natal. p.38. 34 Education Support to OVC in South Africa. Quarter Report: O International. Department Education KwaZulu-Natal. Department of Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 19 the training on the IE Strategy including Screening, Identification, Assessment and Support (SIAS) model for identifying orphans and vulnerable children and their needs (SIAS) and key policies and practices with a focus on laws, policies and programs relevant to OVC, the training of ILSTs of EOVC Project schools focused on the project interventions such as the growing and maintenance of a school food gardens; the management of the project conditional grant; and facilitating inter-sectoral cooperation initiatives as discussed in more detail in later on in the report. The introduction of the SIAS model is one method that all schools indicated has greatly improved and enhanced their ability to support OVC. “We know now how to deal with learners with learning barriers and are having more fun in class. There are no miserable learners any more in the class as we are able to help such learners.” (Xoloxolo FSS ILST Teachers). Due to funding and time limitations, the task of taking the training to all teachers in the schools was left to the trained participants. Resource materials made available included the development of a blank ILST organogram for completion by each school as well as a poster highlighting the main care and support policies was also developed and distributed to the schools to make teachers aware of these guiding policies35. Members of the various ILSTs interviewed were comfortable that the training covered the basic concepts of inclusive education and its implementation strategy in schools. Almost all the schools visited is using the Screening, Identification Assessment and Support (SIAS) tools and reporting templates provided during the training workshops. “The ILST become more effective as a result of the project.” (KII: ILST Tafelkop). “The ILST committee is in place since 2010. A direct consequence of this project.” (FGI: ILST Nombewu). In all the schools visited ILSTs were in place and the poster designed on key policies relevant to OVC with the completed ILST school organogram was observed in most of the schools. Some ILSTs presented more evidence of their capacity - both in terms of human capacity and skills to lead and implement OVC support interventions. School management teams (SMTs) and ILSTs that attended some training sessions were very hopeful and positive about their roles in the EOVC Project, and the potential for a meaningful improvement in the lives of OVC. ILST members indicated that they are now able to identify children who need support and the type of support needed; maintain a register to record needs and the support provided; and how to access and involve other government departments to provide assistance to and referrals for learners. The introduction of the SIAS model for identifying orphans and vulnerable children and their needs, is one method that all ILSTs indicated has greatly improved and enhanced their ability to support OVC. “We are more empowered now to assist identified learners though the screening process.” (Teacher of the ILST Nombewu FSS). “Team work is encouraged amongst the teachers through the ILST and to be involved in the EOVC Project under the umbrella of the OVC program.” (KII: OTT member). Evidence from discussion with school principals, ILSTs and the district structures indicates that one of the major successes of the EOVC Project was the revitalization of in-school institutional structures. “We had our own committees but when they came with the concept of the ISLT we’ve realized that we can cluster our committees together into the three groups and that made our work easier for us. Although we did not understand it at first.” (ILST Vulekani). The ILSTs members emphasized that the MiET Africa Project Coordinators provided additional support in accordance with the specific needs of the various schools linked to the implementation of the various pilot interventions – this was also echoed by the school Principals. An example of such support rendered was to FSSs with specific reference to Xoloxolo and Nombewu that were included in this evaluation. A stakeholders’ meeting was instantiated and facilitated with the support of a MiET Africa Project Coordinator that was held Xoloxolo Senior Primary School. The focus of discussion was the school’s FSS status and to develop a strategic plan on how the various government departments can collaborate to 35 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. p.43 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 20 support the school. Present at the meeting was the Mayor of Kokstad, the departments of social development, health and other relevant community members. As highlighted above the EOVC Project’s management and implementation framework were embedded in KZNDOE’s IE Strategy and were implemented in partnership with the various levels of the KZNDOE department. The EOVC Project management team was absorbed by the SNES unit placing the EOVC Project within the department and as an integral part of its IE Strategy. The OTT based at the district/circuit level met regularly to manage every day operations of the project and is ensuring project activities are integrated into district plans. The DBST was reported to be now capable of rolling￾out the IE policy in all Sisonke schools based on intensive training on the IE Strategy36. ILSTs at school level were able to implement a range of interventions and capacitated to establish learning support mechanisms in place that are aligned with government’s Foundations for Learning campaign37. At the Information Dissemination Seminar it was reported by the District Office that: “The sustainability of the EOVC Project is in effect highly achievable because the project activities build on and strengthen the key component of the KZNDOE Strategy which has as its key focus the diminishing of barriers to learning.” (District Office Presentation). In the words of the MiET Africa’s OVC Program Manager: “We created an enabling environment for the improvement in learner performance to be achieved – insufficient time to make a bigger impact.” 3.1.5 COLLABORATION AND PLANNING The EOVC Project had an interwoven implementation and management structure as outlined in Annex D of the report. The priority set on joint planning and collaboration was experienced as a cornerstone for the implementation of the EOVC Project. Joint planning manifested at various levels of project implementation. At the program level USAID/Southern Africa and RTI International partnered with the KZNDOE from the conceptualization stage of the EOVC Project. Secondly through the close alignment between the EOVC Project and the USAID/Southern Africa funded Peer Education Project (LCD) that is also involved in schools in Sisonke to ensure the two projects complement one another. Thirdly by complementing the two OVC projects implemented in the education sector - one funded by the USAID/Southern Africa (the EOVC Project) and the other funded by the KZNDOE by ensuring schools are not recipients of both sources of funding. Examples of collaboration at project management level are the establishment of a joint management structure - the PMT/ IEPTT and the formation of the OTT at operational level. Strategic decisions taken such as to extend the project’s geographical area of implementation to include the East Griqualand Circuit as well as the joint submission of a further project proposal and the later on no-cost extension application are practical examples of a shared interest and understanding of the project’s primary intention. The Intervention Design Workshop was followed by the Intervention Design Reflection Meeting. At these sessions, the proposed school and district level interventions represented serious effort to ensure stakeholder participation. In this instance, the identification and fine-tuning of the pilot interventions identified. There was clear ongoing capacitating of structures such as the DTT as a partner in the training activities. The Sisonke District Office was heavily involved in decision making and monitoring and provided valuable input in the annual workplans. RTI International and MiET Africa engaged in regular feedback as well as the ongoing visible support. Altogether there was a collegial relationship that seemed to ease project implementation sometimes under very tight and difficult circumstances. This culminated 36 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. p. 45 37 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. p.44 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 21 with a successful project Information Dissemination Seminar to share project lessons and recommendations with project stakeholders. During KIIs, the cordial relationship between implementing partners was highlighted. The time taken by RTI International during the initial stage of the project to strengthen relationships and obtaining clarity on expected roles and responsibilities made it possible for the team to collectively achieve so much within the very short implementation time. The project Chief of Party (COP) was complimented for her facilitation skills; MiET Africa for its experience working in the province and with the KZNDOE that helped strengthening the relationship between the department and other members of the project team. The project partners complimented the KZNDOE staff involved for their cooperation, support and ability to remove hurdles during the implementation though they had many other responsibilities. 3.1.6 PROJECT RESULTS FRAMEWORK AND REPORTING At the outset, the EOVC Project articulated a broad set of goals and geared up to report against them as was outlined in the first Annual Workplan submitted38. The overall goal was improved school performance, access and retention. The two intermediate results (IRs) support the achievement of this goal as i) increased institutional capacity to support OVC; and ii) increased participation of caregivers and community volunteers in support of OVC. The monitoring and evaluation (M&E) approach intended to i) measure project outcomes and impacts, as well as processes and outputs, with sensitivity to gender; ii) derive information primarily from and useful to government and community sources; iii) use evaluations to directly inform best practice, bring to scale, and sustain capacity after expiration of the project; and iv) respond to USAID/Southern Africa’s operational plan and PEPFAR reporting requirements39. RTI International was to work with the implementing partners, national DOE, KZNDOE and USAID/Southern Africa to design and implement a performance M&E system that builds on the existing M&E tools and systems used to monitor the Department’s response to OVC, including M&E for education centers of care and support for vulnerable children in the province. Several sources of indicators for monitoring the EOVC Project were identified: i) project’s internal monitoring process conducted by the implementing partners in respect of their activity indicators at input level; ii) KZNDOE indicators for M&E, as part of its three year plan (2008 – 2011) to implement Education White Paper 6; iii) relevant PEPFAR indicators; and iv) USAID’s Basic Education Operational Plan (BEOP) indicators40. According to the EOVC Project management structure client-level reporting and communication was managed by RTI International, while internal government communications pertaining to the project remained with the different departmental structures41. All project activities were to be monitored regularly by MiET-Africa Project Coordinators on site according to a monitoring plan for each quarter and shared with the DBST to coincide monitoring visits as well as with the schools. In addition regular site visits were conducted and meetings were held with the implementing team by RTI International. Project indicators agreed at the July 2009 PMT were reviewed in December 2009. It was decided that a formal indicator review was needed to improve indicator quality and ensure clarity. In early 2010, at the request of the KZNDOE and with input from senior officials from the SNES Unit, the Sisonke District 38 Education Support to OVC in South Africa. Annual Work Plan: May 2009 – April 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education.17 39 Education Support to OVC in South Africa. Tier II Application–Revised. Attachment 2 674-A-00-09-00054-00. February 19, 2009. RTI International. P.O. Box 12194 Research Triangle Park, NC 27709-2194. p.12. 40 Education Support to OVC in South Africa. Annual Work Plan: May 2009 – April 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. p.19-20 41 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. p.11 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 22 Office, MiET-Africa, JET and RTI International, the indicators were simplified for better alignment to the project goal42. During the latter part of 2009, revised PEPFAR OVC indicators were established and the EOVC Project was required to report against these throughout the project period. Through the project’s work to improve school OVC record keeping system and use and maintenance of OVC registers, the COP indicated and as was reported in the first Annual Report submitted that it was possible to report on some of the new generation PEPFAR indicators related to Care Sub Area 5: Support Care and Health Systems Strengthening Sub Area 7: Health Information Systems43. Reporting on the set indicators was done on a quarterly basis as well as on an annual basis. Each of the quarter and annual reports contained a separate section on Indicators & Targets as is evident from the quarterly and annual reports submitted to USAID. In these sections RTI International reported on all quantitative IRI Outcomes based on information derived from mainly training attendance registers and school OVC. Reports submitted to USAID were a combined report based on reports received from implementing partners as well as the KZNDOE and would include minutes of meetings with the Department / district. According to one of the implementing partners reporting to RTI International was done through quarterly reports, which were derived from the monthly reports obtained from the various MiET Africa training coordinators. The various quarterly reports were consolidated by RTI International into an annual report. This form of reporting procedure was indicated to be very useful by MiET Africa44. During KIIs conducted it was confirmed by several partners – including the District Office, that communication was open and transparent. Reports, including financial reports were shared and the various partners – KZNDOE at various levels and the implementing partners participated in annual project planning, providing input and feedback on quarterly reports before it was submitted to USAID/Southern Africa as was also recorded in the minutes of the PMC of September 200945. At school level Educator Support Committees of the ILSTs interviewed indicated that no reporting template in addition to those related to the SIAS training was provided to report on interventions implemented at the schools except for reporting on the EOVC Project’s conditional grant spending. MiET Africa Project Coordinators frequently visited the schools and collated the information required from existing records. The Contracting Officer’s Technical Representative (COTR) of USAID/Southern Africa confirmed the reports received were of very good quality and based on the format provided by the agency. RTI International provided additional information e.g. school profile information and the numbers of learners benefiting from the project that made the content richer and made it more real on what USAID/Southern Africa is doing for each dollar spent. 3.2 PROJECT INTERVENTIONS Five of the evaluation questions relate to the project interventions per se. This section of the report focus on the various project interventions implemented at school level taking into consideration the specific school context; the perceived value of the interventions and project; and the impact and benefits of the various interventions and project. 42 Education Support to OVC in South Africa. Project Management Team Meeting: Draft Notes and Decisions – 24 February 2010. Ingeli Forest Lodge, Kokstad. 43 Education Support to OVC in South Africa. Annual Report: 1 October 2009 – 30 September 2010. Funded by USAID/Southern Africa. RTI International. Department Education KwaZulu-Natal. Department of Basic Education. p.16 44 Key Informant Interview conducted with MiET Africa on … 45 Education Support to OVC in South Africa. Planning Meeting Agenda, 29 September 2009. Ascot Conference Centre, Pietermaritzburg. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 23 3.2.1 SOCIAL FACTORS INFLUENCING EOVC IMPLEMENTATION AT SCHOOL LEVEL Social factors discussed here include the manner in which the project relate to other government interventions in the district, and socioeconomic and cultural factors that characterize the school￾community contexts. The project sits squarely within the DOE’s Inclusive Education framework, KZN Premier’s provincial Sukuma Sakhe, Department of Arts and Culture’s Library and Reading Clubs, and the Greater Kokstad Municipality Library Week Reading Competitions. While the DOAC and the GKM promotes reading in the district, the Sukuma Sakhe Program aims to implement a sustainable food security intervention that will systematically contribute to community development, job creation, strengthening of institutions and poverty alleviation. Through its development programs for OVC and the institutions that can support them such as caregivers, schools and district, the project aims to bring the needs of the most deprived first. In par 0 above it was indicated that each of the EOVC Project schools was given the option to select and implement those interventions most suitable for their context and need. This allowed for some variation among the schools. For instance, in Ladam Secondary High School it was decided Grade 12 be included, although the focus was on Grade 10 and 11. The SGB decision at Thibane Combined was that vulnerable learners be identified by educators using the SIAS grid. Tafelkop’ small enrolment size resulted in all learners receiving uniforms and tracksuits. The implementation of the EOVC Program progressed at school level according to the social dynamics and cultural factors that existed there in. As indicated in par 0 above, the EOVC Project schools were given the option to select and implement those interventions most suitable for their context and need. Social factors relating to cultural orientations have shaped the implementation of the EOVC Project in the various school-community contexts, especially in terms of how learners took advantage of the interventions. While most of these contextual issues were identified during the baseline study, some were identified through this evaluation. Context differed vastly between project schools based on farm, rural, and township and/or urban locations. Social factors relating to cultural orientations have shaped the implementation of the EOVC Project in the various school-community contexts, especially in terms of how learners took advantage of the interventions. While most of these contextual issues were identified during the baseline study, some were identified through this evaluation. Context differed vastly between project schools based on farm, rural, and township/urban locations. Farm Schools are characterized by high learner dropout, poor attendance and late coming due to distance and poor transport support. Learners fall victim to transactional sex46 and are preyed upon especially by travelers e.g. truck drivers. Additional risks include seasonal child labor47 and household tenure uncertainty (since housing is provided by the farm, conflict with farm management leads to homelessness or rapid mobility). Learner retention becomes more urgent in these contexts, and the EOVC Project has made some contribution towards achieving learner access to education and their retention. According to the district official interviewed, access to farm schools is sometimes problematic, because they are public schools built on private property. Informants spoke about the clear division between farm and township school project participation. Some farmers require advance requests for permission to enter the farm, even by departmental officials and parents. As a result, parental participation in schools was reportedly compromised. While respondents were adamant about these issues, the evaluators found the farm school in the sample, Tafelkop Primary school, does not experience access problems. 46 Transactional sex is a form of commercial sexual exploitation of children and is defined as a Worst Forms of Child Labour in terms of the International Labour Organisation’s Forms of Child Labour Convention (1999). Child Labour Programme of Action for South Africa development, Phase 2: 2008 to 2012 p.6. 47 Child labour as defined by the Child Labour Programme of Action for South Africa is work by children under 18 which is exploitative, hazardous or otherwise inappropriate for their age, detrimental to their schooling, or their social, physical, mental, spiritual or moral development. Child Labour Programme of Action for South Africa development, Phase 2: 2008 to 2012 p.6. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 24 Rural Schools are characterized by higher poverty rates due to high unemployment rates in rural areas. Families struggle to access government department services. Caretaker and parental illiteracy and/or lack of reading material hamper learners. Essential documents that are required for learners to secure social grants e.g. birth certificates, are often unavailable, and learners often rely on pensions received by their grandparents. Like Farm Schools, learner transport is difficult and risky. There were reports that the Department of Transport provided learners with bicycles to discourage dropout, but the evaluators observed only a few bicycles used by learners. This is an example of some achievement of multi-sectoral collaboration created by one of the EOVC interventions called jamborees. Urban Schools, also referred by most informants as township schools, are characterized by overcrowding linked to high population density, rapid spread of HIV and Tuberculosis (TB). In the townships there are informal housing (shacks), burst water pipes and discernible air and other pollution. Townships are also characterized by the presence of numerous taverns (that are reported to serve alcohol to children), gangs, high level of substance and alcohol abuse and this, in turn, leads to violence, high drug addiction, high teenage pregnancy rates, and sexual abuse. According to learners interviewed from Xoloxolo Full Service School, some learners are dealing drugs in one of the public primary school within Kokstad. This, they report, is also an issue for most township high schools, including Shayamoya Primary, Mt Currie High, and Carl Malcomes High – schools all included in the EOVC Project. According to learners interviewed, the same three high schools were well known for incidences of learners carrying weapons to schools. Learners said that Carl Malcomes High School, in particular, has been identified by the police as hot spot. Moreover, respondents noted that at the nearby truck stops learners engaged in transactional sex in an attempt to survive and sexual abuse is rife in households. Some learners reported that teachers at the three high schools have sexual relationships with learners, and that corporal punishment and bullying was still practiced at the school. However, they said that teenage pregnancy rates were low at the school. A District Official was concerned that some of these problems are underreported, and thus not addressed. It is inevitable, therefore that all these issues are in conversation with the EOVC interventions. Cultural factors were mostly reported in Umzimkhulu and, to a lesser extent in, Kokstad. These include forced marriages, cattle herding and initiation practices. The community, in which Thibane Combined School is located, was considered one of the hotspots for forced marriages, according to the district official interviewed. He said that forced marriages are considered a cultural matter and are both permissible and justifiable by parents. An example of this was a story he told of a learner in one of the schools, who was forced into marriage and the teacher who attempted to intervene was reportedly victimized by parents. Other cultural practices include boys herding family cattle, goats and sheep from approximately age 5 to 8, delaying or interfering with school attendance, and resulting in over-age children in the classroom. Male isiXhosa initiation into manhood ceremonies take 17 to 19 year-olds out of school for several weeks as they undergo circumcision and training/mentoring by tribal elders. This is a particular problem as secondary learners have often either started late or repeat grades, thus many of this age are in Grades 9 to 11. Many never return to complete schools. There is also an example of intolerance to teenage pregnancy as a customary issue in some households. According to the ILST in Carl Malcomes, one child in Grade 12 identified as vulnerable and living with her aunt got pregnant, was chased out of her home, stayed with friends, started drinking alcohol, became extremely violent against other learners, and her performance dropped considerably. A lot of poverty related issues were identified during the baseline, and the Project continues to grapple with them. Among these poverty related factors, unemployment is the main severe factor, which explains why most households are unable to make available materials for children to read. Learners are unable to do Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 25 homework, because their households lack basic services. Both learners and teachers travel long distances to and from schools with limited to none access to public transport especially in the rural areas. Vulnerabilities relating to socio-cultural factors include children born to teen mothers, single parenthood, grandparents-headed, child-headed, and other fostering arrangements. Various informants indicated that there are incidences of sexual abuse often perpetrated by step-parents, particularly fathers. There is high dependence on social grants: child support grants, disability grants and pensions. Learners discussed stigma attached to OVC status with the evaluators. They reported that boys believe that as Xhosa men, they must be the ones providing for families and not accepting handouts. In several schools, some learners were reported to be shy and afraid to come forward and ask for support through the EOVC Project. The publicity, at a community ceremony with photography, associated with the process of distributing uniforms discouraged learners from requesting assistance. Likewise, the practice of handing out food parcels in the school hall, publically, was seen as embarrassing. 3.2.2 UTILIZATION AND BENEFITS OF PROJECT INTERVENTIONS AT SCHOOL LEVEL EOVC Project school level interventions included training that supported the establishment of learner support mechanisms; providing conditional grants, establishment of food gardens; and providing of support services to OVC through multi-sectoral collaboration with relevant government departments. i. JAMBOREES Inter-sectoral cooperation (also known as multi-sectoral collaboration) is an essential component of the KZNDOE IE Strategy in facilitating access to services for OVC from various government departments and other service providers. The focus of the EOVC Project interventions as emphasized by the COP were not to address the various vulnerabilities identified in the baseline study as that is the role of other departments but to facilitate such support.48 To this end a coordinated strategy were to be developed in collaboration with the DTT, circuit, ward and schools to hold service delivery events – referred to as Jamborees to provide services including assistance to OVC and families obtaining required documentation to access government support grants; providing of primary healthcare check-ups, including TB, HIV and AIDS etc. Collaboration with local municipalities was essential as the Premier’s flagship programs are housed at this level to coordinate service delivery. Three jamborees were held during September/ October 2010 namely at Nombewu FSS, Ladam and at Xoloxolo FSS – all three schools visited by the evaluation team. There is consensus among different informants that jamborees played an important role to encourage a strong relationship between schools and the communities they serve, and with different stakeholders within the public sector. “They (parents) found it helpful to them. It is their wish that we should continue with these Jamborees.” (Thibane Combined School). “Community members benefited from the Jamboree that was held at the school. Various departments participated including the police that assisted with the certification of documents … Community members got wheel chairs and walking sticks…” (School Principal: Ladam Senior Secondary School). The collaboration of the Department of Agriculture (DOA) with the EOVC Project schools on the establishment of food gardens as discussed below is a further example of how multi-sectoral collaboration with relevant government departments was facilitated to render integrated services to schools. ii. SCHOOL FOOD GARDENS In partnership with the DOA food gardens were established or improved at various EOVC Project schools. The establishment of these was also linked to the Premier’s flagship project of “one- 48 Key Informant Interview with COP, 9 and 12 September 2011 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 26 school/home one-garden”. School managers and teachers from participating schools were trained by the DOA in establishing school food gardens. Training included topics such as the most suited crops for the soil type; sowing and harvest seasons; compost-making etc. as well as suggestions on how to integrate the school garden activity to learning in literacy, numeracy and life orientation. The information gathered from various schools visited indicates that food gardens were used both in similar and in different ways. The produce from the food gardens was either given to needy community members and learners parents on a day-to-day basis or used to complement feeding schemes. Vulekani Special School provided learners with tools and seeds to start their own home gardens and this resulted in improving parents’ confidence in the value of education to their children. Seven of the eight schools included in the Performance Evaluation had functional vegetable gardens in place at the time of the evaluation. However, the optimal utilization of the food gardens were limited by droughts; poor soil conditions; as well as the lack of parental and/or community involvement in maintaining the gardens. At two schools (Ladam Secondary and Buhlebezwe Primary) in Umzimkhulu Circuit experienced problems with soil conditions and were unable to obtain good quality vegetable produce. Informants at Buhlebezwe Primary indicated that throw utilizing traditional means such as using cow dung as manure the vegetable garden was has become a huge success and the produce is used to complement the school feeding scheme. Ladam Secondary has sent soil samples to the DOA for testing and is still awaiting results. “Our food garden was poor but with the support of this project we were able to upgrade the garden and the learners now get fresh vegetables.” (ISLT Member of Vulekani Special School). “Garden was done by the learners themselves – the young ones – they enjoyed it.” (Vulekani Special School). “Food garden was integrated into our curriculum. We did not buy seedlings but bought seeds for the learners to learn how to plant it and to take care of it. They had to bring along manure from their homesteads.” (Buhlebezwe School). iii. Conditional Grants The EOVC Project provided an opportunity for project schools to receive a conditional grant aimed to encourage schools to: i) take responsibility for school￾based care and support; ii) provide funding to implement care and support activities; iii) encourage sustainability of the work of the ILST; iv) enable schools to develop financial management skills and demonstrate that they can manage these funds. A conditional grants policy guided the schools on the use of and reporting on spending of the grant allocated49. Monthly conditional grant of R1,000.00 (US$139.32) or R5,500.00 49 The conditional grant. Participant Manual. ESO. USAID/Southern Africa. RTI International. MiET Africa. Department of Education Kwa￾Zulu-Natal. April 2010 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 27 (US$766.25) per quarter was distributed to each of the EOVC Project schools50. In total a conditional grant of R24,000.00 (US$3,343.64)51 has been paid to each of the EOVC Project school over the project period. Training in the use of the grants was aimed to ensure that school principals and ILST members were able to monitor, use and report on conditional grants appropriately. A conditional grants policy guided the administration of this grant52. In seven of the eight schools visited the utilization of the conditional grant was evident. In most of the schools the grant was used to establish and enhance more than just the school food gardens that were the focus of the initial tranches made to the schools. Grant funding was used to provide food parcels (limited number of schools visited); school uniforms (all except for one school), toiletries including sanitation pads for female learners; soap, tooth brush and paste, towels, etc.(majority of schools visited). In some schools, the Grant was also used to buy reading books and stationery for learners as a way to improve their performance or hired local community members to do garden work or assist learners with homework. Such stipends were also paid from the Grant. “First thing we did was to buy learners a full set of school uniforms including shoes; beanies; and raincoats as it is very cold in winter in an attempt to reduce school absenteeism” (School Principal Tafelkop Primary School). “Grant funding was used to buy uniforms for learners in need. They are now fully dressed – just as the other learners” “(ILST Members of Nombewu FSS) Members of the ILST of Thibane Combined School highlighted that they were able to provide emergency assistance to a learner through the grant. The learner fell asleep while studying and the whole room burned down including his books, uniform etc. Immediate assistance was rendered to learner to prevent him from being further disadvantaged while preparing for the final school examination. Stakeholders at the district, circuit and school level agreed that uniform provision made possible through the conditional grant has had positive impact on learners’ school attendance. The OTT and a number of community level informants at selected schools indicated that the school attendance has improved as a consequence. iv. Classroom Management, Teaching and Learning Support The classroom support workshops were to help educators to create learner–centered classrooms, to support learners whose individual needs and abilities are acknowledged and to prepare them to achieve success. This is a key component of the IE Strategy and essential for schools to address barriers to learning in practical ways. This workshop therefore provides educators with the opportunity to manage a classroom which accommodates all learners in their class. The managerial actions include: i) planning the classroom environment; ii) organizing the teaching environment; and iii) classroom administration. Teachers at Vulekani Special School indicated that the majority of their teachers were not trained in teaching learners with learning disabilities and the EOVC Project gave them more confidence and useful 50 Exchange rate of 7.1778 average over project period (30 April 09 – 31 August 11): http://www.oanda.com/currency/historical-rates/ 51 Exchange rate of 7.1778 average over project period (30 April 09 – 31 August 11): http://www.oanda.com/currency/historical-rates/ 52 The conditional grant. Participant Manual. ESO. USAID/Southern Africa. RTI International. MiET Africa. Department of Education Kwa￾Zulu-Natal. April 2010. p.4. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 28 material on multi-level teaching. Other teachers reported that these workshops were very helpful it enabled them to identify and to help slow learners through multi-level teaching in their classes. “It was useful. We got books to read as reference material and it resulted in learner performance improvement as we were taught how to do multi level teaching - we were not aware that we can do this even in the foundation phase” (ILST Teacher of Tafelkop Primary School). Teacher training also included a focus on literacy/language and numeracy/mathematics (maths) subject areas. The numeracy/maths workshop focused on a enhancing teacher knowledge and methodology and innovative ways to teach basic number concepts including space and shape through practical activities with general applicability across grades. In December 2010, 60 Maths Centre numeracy kits were distributed to each participant. Teachers were overwhelmed – one Umzimkhulu teacher said ‘Come and visit us next year, you will see us in action especially now that we have the resources.’ In response to a cry from Further Education and Training (FET) teachers for support in Maths, a Maths specialist was brought in to train FET teachers using a variety of methods and apparatus. A set of lesson DVDs and guidelines were given to each participant. Although there has been great excitement about this training, according to a MiET Africa informant, regular and more intensive training is needed to bridge the teacher knowledge gap before any significant impact is made to learner maths competency in the District. The literacy/language workshop aimed to promote reading – to get children into the habit of reading. The workshop provided ideas for generating vibrant reading environments in schools across all grades and coaches the use of a range of skills amongst others: reading aloud, guided reading, silent reading (especially in secondary schools) and how to start and sustain reading clubs. An important aspect of the training was to demonstrate how teachers can use language across the curriculum and how to organize and host literacy events such as “Festival of Books” and “Reading week”. Schools were encouraged to buy IsiXhosa readers using the EOVC Project conditional grant to promote reading in the Foundation and Intermediate Phases. EOVC Project schools received boxes of children’s books from ELITS that added enormous value to the project, through the initiative of the MIET’s Training Coordinator. “We had competitions at the school last year but this year we also participated in the reading completion of the district and some of our learners won some prices” (ILST member of Carl Malcomes School). v. Caregivers The role of the caregiver is crucial in a child’s success at school. The caregiver’s role can range from financial (paying school fees, providing uniforms); to allowing children to attend school—or better, encouraging and facilitating school attendance and strong performance; to providing a place and a time for children to study at home; to reviewing homework and school reports; to providing constructive feedback for strong and poor performance; to becoming involved at the school. Minimally, caregivers will need to play a role in structuring an enabling environment for learning, including sharing in-school children’s domestic chores; helping the children find time to study, and identifying relatives or peers who can lend assistance; discussing performance with the learners and organizing a time to keep in touch with the schools regarding progress on the learners’ support interventions; engaging with other caregivers to share experiences and collectively address challenges; and building on each others’ successes. Cluster workshops with caregivers and parents – including the SGB focused on assisting parents on how best to help children learn better. Ideas included the formation of support groups for learners. During 2011 five groups of parents/caregivers drawn from the various EOVC Project school communities were trained on how to care for and support children’s education. Only one of the parents/caregivers interviewed during this evaluation attended two of the parent/caregiver focused training, and she said that the workshops were very useful and enhanced her confidence to provide support to her children. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 29 3.2.3 IMPACT ON LEARNER ACCESS, RETENTION AND PERFORMANCE The current USAID Education Strategy53 is premised on the development hypothesis that education is both foundational to human development and critically linked to broad-based economic growth and democratic governance, shows a commitment to increasing access to quality and relevant learning. This has become one of the focal points in all programs funded by the USAID. The EOVC Project aimed to improve learner access to education and their retention within the education system. As the intended impact assessment study was replaced with the Performance Evaluation of the EOCV Project per se the intention – as outlined in the TOR was not to assess learner performance, but only to get an indication of the perceived impact on learner performance. The nature of information gathered during this evaluation provide some – tough very limited, insight into the enrolment patterns and attendance trends and experienced by the EOVC Project schools between 2009 and 2011 as well as the dropout – retention trends since the beginning of 2011 as summarized in Table 4 below that impacts on learner performance. This section discusses the manner in which the selected schools experience interventions on the impact it made in learner access, retention and performance. This is examined in the context of the school’s enrolment patterns, as well as the rate of learner attendance and dropout. Some of the questions that were asked to gather information on this section were: Since the intervention: how has learner access and retention changed in the two participated circuits? What aspects of the EOVC Project brought about that change? What aspects of the EOVC Project were least effective in influencing learner access and retention? Having asked these questions to different categories of informants provides a broader overview of how interventions were both perceived and experienced by people at various levels. Table 4: Learner Enrolment Trends, Dropouts Figures and Attendance Rates Name of the School Type of Location Total Enrolment 2009 Total Enrolment 2010 Total Enrolment 2011 Dropouts Jan-Aug 2011 Approximate Attendance Rate Tafe kop Primary Farm 26 27 24 0 81-100% Carl Malcomes High Urban 1590 1485 1781 3 41-60% Thibane Combined Farm 437 385 297 10 61-80% Xoloxolo Full Service Urban 854 803 757 81 61-80% Ladam Secondary Rural 382 406 303 10 61-80% Buhlebezwe Primary Rural 198 179 171 8 81-100% Vulekani Special School Rural 275 336 261 75 41-60% Nombewu Full Service Rural 469 405 445 4 81-100% Learner access to education is examined in terms of enrolment patters and approximate attendance rates. These are marked in red color in Table 4 above. Enrolment in three schools (two in East Griqualand Circuit –Thibane Combined and Xoloxolo Senior Primary FSS and Buhlebezwe Primary in Umzimkhulu Circuit) declined between 2009 and 2011, while it fluctuated in five schools (three in Umzimkhulu Circuit – Ladam Senior Secondary, Vulekani Special School and Nombewu FSS and two in East Griqualand Circuit –Tafelkop Primary and Carl Malcomes High. For the schools where enrolment figures were high, 53 USAID Education Strategy 2011-2015 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 30 Figure 3: Summary on Learner Performance the informants believed that this was a consequence of the EOVC Project, more specifically the provision of uniforms and food parcels that were made possible through the conditional grant. It was also reported in a number of schools that attendance increased (though still below the expected norm) as a result of the support that the schools were giving to learners – the ability of teachers to identify and provide support to OVC in need of care and support through the ILST and collaboration with other departments. It was indicated that this support have increased learners’ interest to come to school. On the same vein, informants reported that there was a noticeable drop in absenteeism rates. The selected schools had a varying experience of retaining learners, and this had very little to with enrolment sizes, but more to do with how the school had marketed itself to its clientele. The dropout figures are marked in blue color in Table 4 above. The table indicates that experience of learner dropouts in the eight selected schools between January and August 2011varied from 0 to 81 in the East Griqualand Circuit, and from 4 to 75 in the Umzimkhulu Circuit. Ensuring the retention of learners in some of the schools was said to be difficult to achieve because of teacher workloads and overcrowding in classrooms where learner teacher ratios was as high as between 1:90 and 1:135. For these schools, a suggestion to employ dedicated staff focusing on implementing projects such as the EOVC Project was made. In Umzimkhulu Circuit, the Vulekani Special School had the highest number of learner dropouts, while Nombewu FSS had the lowest. Informants indicated that through the hosting of Jamborees, visible support and additional services rendered to learners as a FSS, Nombewu FSS was noticed more by the community and as a consequence attracted more learners. The Vulekani Special School ISLT indicated that parents were using the school to apply for disability grants and once the grant commenced payments, they then remove their children from the school. To answer the question of whether learners benefitted from the EOVC Project in terms of performance, a total number of 69 learners (34 boys and 35 girls) from seven of the eight selected schools were rated by their teachers on Numeracy, Reading in Mother Tongue Language, Reading in the Language of Instruction, and in Writing using a categorization of above average (above set standards), average (meeting set standards) and below average (below minimum set standards). This was done purely to get a sense of the impact the EOVC Project on learner performance. Figure 3 shows that the majority (53) of learners were average in their performance in Numeracy, while a smaller number (11) of them were rated as below average, and only a tiny fraction (4) of them were rated as above average. Tables 5, 6 and 7 provide the performance ratings in terms of gender. An almost even number of girls and boys were rated as average. In Table 5 sizable number of boys (7) compared to that of girls (4) were rated as below average, and none of them were rated as above average, while 4 girls were rated as such. It was observed that even in Carl Malcomes High that rated all its learners as above average in three performance areas, learner performance in Numeracy was rated as average. This shows that the improvement of learner performance in Numeracy has been marginal, according to teachers. For those learners that performed on average, some teachers indicated that learners had difficulty in mastering simple counting, number knowledge, one-to-one correspondence between number symbols and objects; has trouble counting rapidly, learning multiplication tables, formulas and rules; difficulty learning and memorizing basic addition and subtraction procedures. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 31 Table 5: Gendered Performance Ratings in Numeracy Numeracy/ Mathematics Rating Total Below Average Average Above Average Gender Boy 7 26 0 33 Girl 4 27 4 35 Total 11 53 4 68 For those learners that performed below average, teachers in these schools mentioned three reasons for this, i.e. learners had difficulty remembering shapes of letters and numbers, and they did not enjoy doing school work and were often late for lessons. As a result, a lot of creative ways to address this were being done at the schools. These include encouraging more counting by giving learners extra work to improve their understanding of strategic counting principles, as well as to encourage early arrival of learners to school. In most schools, learners were given individual attention, while others ensured that parents signed learners’ homework and/or class work regularly for the same purpose. Improved the numeracy skills of learners by doing every day counting and also ended the use of calculators in the earlier grades. Table 6: Gendered Performance Ratings on Readings in Mother Tongue Language Reading in Mother Tongue Language Total Below Average Average/ Meeting Above Average Gender Boy 2 23 9 34 Girl 2 24 8 34 Total 4 47 17 68 There is general agreement among informants that the EOVC Project exerted a considerable emphasis on improving learner performance in reading. A slightly different scenario appears in the rating of learner performance in reading in their mother tongue language. In Table 6 it appears that though the majority (47) of learners was rated as performing on average in this regard, 17 of them were rated as above average, while a considerably smaller number (4) were rated as below average in their performance. In Table 6, the performance rating between boys and girls appears to be even, with marginal difference. Table 7: Gendered Performance Rating on Reading in Language of Instruction Reading in the Language of Instruction Total Below Average Average Above Average Gender Boy 3 24 7 34 Girl 4 26 5 35 Total 7 50 12 69 A similar pattern is evident in the rating of learner performance in reading in the language of instruction. Figure 3 above indicates that the majority (50) of learners were rated as performing on average in this regard, 12 of them were rated as above average, while a slightly smaller number (7) were rated as below average in their performance. Table 7 shows that though the difference in the rating for boys and girls in still marginal, girls were rated slightly higher than boys in average and below average ratings, and they were slightly higher than girls in the above average rating. Improvement in literacy was noted by the Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 32 language teachers and they attributed it to the reading competitions and identity (ID) workshop on reading and writing. In one school, learners were reported to do 30 minute reading every day. Table 8: Gendered Performance Ratings on Writing Writing Total Below Average Average Above Average Gender Boy 3 24 7 34 Girl 2 28 5 35 Total 5 52 12 69 The performance rating of learners in writing in Figure 3 is not so different from the rating in the other areas. The majority (52) of learners were rated as performing on average in writing, 12 of them were rated as above average, while a considerably smaller number (5) were rated as below average in their performance. Table 8 above further indicates that a slightly higher number of more girls than that of boys were rated as average in writing, while slightly more boys than girls were rated as below and above average respectively. In most schools, informants were in general agreement that performance has improved and the rate of learner dropouts has decreased considerably over the years since 2009. The resultant increase in performance has been some of the reason why enrolment in some schools increased dramatically between 2009 and 2011. Teachers in Carl Malcomes High, school with the highest learner enrolment figures and the lowest dropout rates, said that they were far better than Kokstad College and Mt Currie High, as they received the Education Member of the Executive Council (MEC) of the KZN province Award for learners obtaining more than four A’s in their Matric results. One of the ILST teachers said: “We had A’s in Accounting, Economics, History, and Maths”. 3.2.4 THE IMPACT OF THE EOVC PROJECT ON DEPARTMENTAL PROCESSES In addition to facilitating KZNDOE’s service delivery to schools through jamborees, the OTT said that the EOVC Project had a direct impact on the departmental processes in that it led to the establishment of Nombewu as a Full Service school, with far better support than other full service schools in the district. Though the criterion for its selection was that it had already obtained a Section 21 status, financial allocation has been made to the school, a Support Centre is being built and the Educational Human Resource Directorate (EHRD) is providing training to teachers at the school. The KZNDOE’s financial allocation to this school is in threefold, i.e. between R50,000 and R80,000 for Care and Support; R150,000 for ILST Training; and R285,000 for Career Centers. According to the OTT, Vulekani Special School was identified as a SSRC (Special School as a Resource Centre) through the EOVC Project. Even after the EOVC Project, the school will receive continuous support from the KZNDOE. The OTT sees that the concept of establishing full-service schools ensures that learners are benefitting directly with equipment and infrastructure from government institutions. Most ILST informants said that the new FSSs and special schools in the EOVC Project got more understanding of their role in the light of the IE Strategy. This was also an observation of the KZNDOE official interviewed, who said that the schools have informed the KZNDOE about the positive impact that the EOVC Project has made on the education system. The MiET Africa offered a number of training workshops that supported learning and teaching, and these are outlined in relevant sections above. Informants commented on the impact these workshops have made on schools. The MiET A informants said that the workshop on Classroom Management has enhanced teachers’ interest in promoting reading in their schools. Teachers in Vulekani Special School Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 33 said that MiET Africa also provided ongoing support through regular school visits. One ILST teacher said: “It made life easier for us and we will be using it from now onwards”. The OTT indicated that most schools have increased their classroom support where learners participate in reading through class libraries. The workshop on Curriculum Support improved teachers’ teaching methods. Even at Vulekani Special School, teachers indicated that most of teachers were not trained in teaching learners with learning disabilities but the EOVC Project gave them more confidence and useful material on multi-level teaching. However, they were concerned that the Workshop on Numeracy only focused on GET Grades 4 - 8, and not on Grades R – 3, and for this reason they felt that it was less relevant for them. According to the OTT, the SIAS, which is the tool used to identify vulnerable children, assess their level of vulnerability and provide them with proper support, has been used in EOVC Project schools. As a result, the OTT believes that the information on OVC is readily available in schools for other potential interventions beyond this EOVC Project. This was attested by one ILST teacher who said that the teachers liked the template, because it enabled them to check vulnerability and keep their statistics. There is general agreement among ILST members at various schools that through the EOVC Project, teachers are now better able to attend to learners’ specific needs on time, and that this has enabled them to contribute towards reducing learner dropout rates. OVC are identified on yearly bases by Life Orientation (LO) teachers and/or peer educators. At school level, orphans and vulnerable learners were identified through SIAS, and the project Conditional Grant was used to support them in various ways. Mostly, in full service schools, learners are also provided with spectacles and hearing aids. HIV Tests is done at one of these full service schools, and library membership cards were provided to learners and prize giving days held to encourage best performance and achievement. Some schools had hired, or were in the process of hiring, a local community member to work in the garden, and be paid from the Grant. Stakeholders at the district, circuit and school level agree that uniform provision and awareness has had positive impact on learners’ school attendance. The OTT and a number of parents informants at selected schools indicated that the school attendance has improved through uniform provision, and that the awareness has impacted positively on parents who are now encouraging their children to attend school. Most informants agree that the EOVC Project was of importance for both the educators and learners. While OVC are more likely now to seek help with confidence, teachers are able to properly diagnose their circumstance and provide relevant assistance. However, for those learners who are still shy to communicate their needs, the schools have devised different mechanisms to encourage learners. Some gives them food parcels and uniforms privately to needy learners to help minimize shyness. It was reported that in some schools, teenage pregnancy has dropped since 2009. Informants at Ladam Secondary Schools, teenage pregnancy dropped considerably from 14 in 2009 to approximately 3 or 4 in 2011, though the problem of dropouts remains a major concern for the school. 3.2.5 SCHOOL-COMMUNITY RELATIONS & PERCEIVED VALUE OF EOVC PROJECT The manner in which the EOVC Project was valued is examined here in the way in which various role players in it showed commitment and level of ownership of the process. School level informants demonstrated these in aspects of the EOVC Project, such as the ways in which they handled their food gardens. The value of the EOVC Project is seen in terms of the most significant change that has happened in most school contexts, and the nature of ownership to the EOVC Project that has created among the stakeholders involved. The Vulekani Special School learners were excited about the garden and were given the opportunity to extend them to their own homes. They were observed working in the garden and teachers said that they often do this on their own free will without being instructed. Most informants agree that the EOVC Project was of importance for both the educators and learners. While OVC are more likely now to seek help with confidence, teachers are able to properly diagnose their Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 34 circumstance and provide relevant assistance. However, for those learners who are still shy to communicate their needs, the schools have devised different mechanisms to encourage learners. Some gives them food parcels and uniforms privately to needy learners to help minimize shyness. It was reported that in some schools, teenage pregnancy has dropped since 2009. Informants at Ladam SS, teenage pregnancy dropped considerably from 14 in 2009 to approximately 3 or 4 in 2011, though the problem of dropouts remains a major concern for the school. Uniform distribution to OVC at various schools has been used as a form of community awareness, which provided a platform to mobilize support from key community structures. For instance, some traditional leadership involvement has been experienced in the Project. Here a traditional leader promised the KZNDOE that ukuthwalwa54 of young girls will no longer be tolerated in his tribe, and that he will meet with his Council to determine the suitable penalty for perpetrators of this practice. This typifies the change of attitude among traditional leadership regarding certain cultural practices that disrupts the education of children. Some of the changes mentioned were that the school dropout rates were lower than in previous years; teachers felt more confident to manage their classroom through multi-level teaching and provide necessary support OVC; learners reading skills and motivation to read improved. In a farm school, some learners were even confident to assist others with their reading. The school-community relations have improved through the use of jamborees and parents workshop. EOVC Project schools have become more seen as centers of care, support and community life. One of the schools won a district food garden competition. In Carl Malcomes High, up until 2008, learners have been involved in Arts as part of the curriculum, but since 2009, with the introduction of the EOVC Project, they started doing Arts as a hobby. In 2010, the school has obtained Position 3 in the provincial Arts competition, funded by the Department of Transport. Kokstad College and Mount Currie School often send their learners to this school during revision times of the year, because of the quality of learning. In terms of community ownership of the schools, it was reported in most schools that community involvement in school matters had increased over the years. Though in some of the schools community involvement, through parents was experience even before the EOVC Project, informants were in agreement that engagement with parents increased as a consequence of the EOVC Project. The schools differ in terms of the progress they have made to build and strengthen the relationship with parents and community in supporting learning for OVC. The community mapping exercise has worked for a number of school and they have managed to forged relationships with existing structures in the community that have resources to support OVC. According to a number of informants, a more involved relationship has been achieved at most selected rural schools in Umzimkhulu circuit. However, their experience of this relationship can be seen in a continuum that ranges from poor involvement (which was also linked to poor support from SGBs) to a much stronger involvement. Jamborees, especially the benefit it made to community members, appear to have played an important role in encouraging this strong relationship. It was reported that the parents’ workshops that were held on how to play a meaningful role in the education of their children had a positive impact on their general involvement in schooling, and this also strengthen the relationship between parents and schools. Urban schools in the sample appear to be better off in terms of parental involvement in school matters and the involvement of other stakeholders, such as community centers in supporting OVC. Such schools can now be seen as centers of community life. In this school-context, jamborees and the schools’ ability to mobilize community support have been cited as some of the contributing factors for this. However, the topographical factor that exists in farm 54 Originally this was a peculiar strategy within a courtship process where the lovers expedite their marriage process often against their parents’ approval. To expedite the process, the male removes the female from her home, and immediately send the lobola as a sign of his willingness to marry the girl. However, nowadays, the practice of ukuthwalwa is done without the girl’s consent and with no intention to marry her, and this amounts to violation of both culture and the Bill of Rights (Summative Evaluation of Education Centres Supporting Rural Development, p46, 2011). Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 35 school-contexts in the Greater Kokstad appears to be limiting parental and community involvement in school activities. Though in one farm primary school attendance at parents meetings had reportedly increased because of the support the EOVC Project was giving to their children, the involvement of the broader community was limited to the support provided by the farmer. In another farm combined school, SGB informants indicated that parents demonstrated limited or no interest in getting involved in the education of their children as the EOVC Project encourages. Project schools were sometimes clustered together and serving same communities, and thus jointly providing support to local OVC. This was reported to be another factor that made a positive influence to school-community relations in various school contexts. In addition to this, there were local community structures that worked hand-in-glove with the schools to support OVC. Such structures include local churches, community centers and other households which provided support to OVC in the form of food, clothes, and blankets. In one context, churches often conducted house visits to provide such support twice weekly to OVC, especially those who did not qualify for receiving social grants of any kind. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 36 4. ACHIEVEMENTS, CHALLENGES AND RECOMMENDATIONS In this section the main achievements; challenges and recommendations identified and proposed for each of the priority questions formulated will be highlighted for consideration in the planning of future related projects and or the roll-out of the EOVC Project. This section of the report is to be read with the report on the 360 degree review that was conducted with key stakeholders at the end of the EOVC Project and as attached as Annex G to this report. 4.1 THEME 1: PROJECT IMPLEMENTATION The key questions on this theme are outlined in par 2.3 above. 4.1.1 KEY ACHIEVEMENTS • The KZNDOE is highly appreciating the collaborative design of the project interventions on the basis of evidence, experience and real beneficiary needs.. • Though the interventions were designed to benefit OVC, the improvement in teaching or school management within the framework of the IE Strategy benefitted all learners in the school regardless of their vulnerability status. • Project activities implemented in partnership with departmental counterparts directly impacted positively on the implementation framework of the strategy. • Informants were very complementary about the success of the project management structures and there was strong evidence that they continue to function. Implementation was strongest at provincial/district level where there was evidence of project ownership. • Ownership of the EOVC Project has resulted in successful building of institutional capacity. This has had a positive effect at various levels: DBST is able to roll out IE Strategy to other schools in Umzimkhulu; ILSTs are able to identify and provide support to OVC; as well as teachers and caregiver, both of whom appreciated the training received. There was evidence that even the foundation phase children were reading better because of the EOVC Project. • In spite of the severe time constraints experienced a total of 8,593 OVC received services through the EOVC Project. After two years, and at just under $55 per person reached each year, a policy that appeared daunting and complicated has touched the lives of more than 22,000 beneficiaries in practical and meaningful ways55. 4.1.2 CHALLENGES • The lack of set standards and/or norms for the implementation of interventions makes it difficult to evaluate impact over time. • The limited timeline and implementation delays have prevented full project maturation. This concern was shared widely across all stakeholders, and it impacted negatively on intended project goal. The EOVC Project had an insufficient lifespan to do everything it set out to do, particularly in the area of improving learner performance. • The protocols and levels of authority within the KZNDOE sometimes compromised official’s attendance at project meetings. Representatives from some section felt that project activities did not relate to their core mandate or areas of responsibility; or because of a non-alignment between mandates e.g. the GET and the FET band. Some indicated that there was lack of available capacity as this project was not their only responsibility. • The current Sustainability Plan of the District provides no clarity on how challenges will be address should they arise in the process. The Plan is also silent on how to monitor the implementation and facilitate proper reporting procedures to enhance accountability. 55 Final Draft EOVC Report pp40-42 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 37 • The training strategy followed (cascade method) - due to funding and time limitations, seems not to be effective. The time set aside for training was too short to do follow-up engagements and revision activities to embed learning into operational practices. Training attendance was inconsistent and some targeted participants did not attend training at all. The envisaged advantage of collaboration between the EOVC project and the Link Community Development (LCD)’s Peer Education project implemented in the same district was not possible during the implementation period of the EOVC Project. This was due to the delays in the implementation of the Peer Education project • Record keeping particularly data on OVC in most schools is still a challenge. Only three of the eight schools visited could show documentation on who received what services. • Various concerns were raised by FSSs including the day-to-day management of funds deposited into their accounts and the decision-making involvement of the District in the utilisation of such; Implications of visits to other schools on Classroom Contact Time; What is or was the involvement of the Unions in the process? 4.1.3 RECOMMENDATIONS • Ensure that the roll-out is done in the District and other parts of the province and that the impact assessment is conducted as was planned. • Ensure sustainability and other rollout plans are realistic. • Provide ongoing support at especially ISLT levels to ensure seeds sprout and are strengthened. • Clarify and confirm protocols and alignments within the Department. • Take decisive actions and relevant activities towards the successful implementation of Sustainability Plan to sustain project. • Ensure that the Sustainability Plan show clear mechanism for contingency, monitoring, and accountability, including reporting procedures at various levels of implementation. • Enhance human capital enhancement through follow-up training; revising training curriculum; and providing special training focus for teachers in special schools. • Provide Inclusive Education workshops to more educators to enable them to better understand, and respond to, needs of OVC. • Provide ongoing support to ILSTs through school visits, especially farms schools to assess the conditions at these schools and how best to assist. • Put mechanisms in place to further encourage the multi-sectoral collaboration that already exists. Such a platform bring together different government and non-government stakeholders to strategically join forces towards addressing the OVC in at provincial, district, and school levels. • Establish a Full Service School Forum to discuss related challenges and possible solutions. 4.2 THEME 2: UTILIZATION, BENEFITS AND EOVC PROJECT IMPACT The critical questions in this theme are outlined in par 2.3 above. 4.2.1 KEY ACHIEVEMENTS • The baseline study documented the numerous socio-economic and cultural issues encountered in education and within the district. From this process, limited interventions were designed. • Schools were given the option to select and implement those interventions most suitable for their context and need. Most schools visited showed evidence of implementation, especially of food gardens, participating in jamborees, and using the conditional grant to support OVC. • A range of provincial initiatives, such as the Premier’s Flagship Sukuma Sakhe Programs provided a framework for the implementation of the EOVC Project. • The main benefit of the EOVC Project was learner exposure to an array of pilot interventions. The benefits were primarily material, i.e. it terms of food, clothing, etc. • The retention and access improved by school uniforms, books and stationary, toiletries including sanitary pads, food, as well the knowledge that learners and their families can ask schools for help. Most EOVC Project schools had become centres of care, support and community life. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 38 • The benefit to learners was visible in terms of school uniforms; food gardens etc. Some of the stories told were about learners’ homes getting burned down due to the overreliance on candles. In one case, the whole room of a learner burned down including learner’s books and uniform. Assistance was rendered to learner though the EOVC Project to prevent him from being further disadvantaged in preparing for the final examinations. 4.2.2 CHALLENGES • The EOVC Project did not address issues of child labor or other cultural inhibitions to learning and teaching such as forced marriages, cattle herding, and initiations. • At schools, stigma was still an issue and some learners were still shy to ask for help. • The Conditional Grant was distributed as a ‘one size fits all’ across different schools. • Primary schools mainly participated in reading competitions which are being sustained. It is difficult to say much that is meaningful regarding performance without pre-post testing and a comparison group, especially given the number of challenges that the education sector is faced with in this time period. With the constraints of time, qualitative inquiry (versus real learner testing, even a shortened version of learner testing), and no comparison group (e.g. either a pre-post design as originally planned and budgeted for or control schools) there is little that can be said with any validity about true performance and attribution. Teacher report results are not always reliable as they are influenced by a lot of factors linked to their expectations. 4.2.3 RECOMMENDATIONS • Conduct an assessment on the various pilot interventions implemented at school level to ascertain what worked best and why. • Distribution of funding to be calculated in terms of the number of learners in each school, and need not be a flat rate for all schools. • Follow-up / ongoing support to the learners from primary school to high school is required. • Schools to learn from one another regarding the implementation of various interventions and they need to exchange notes ideas on how best to implement such interventions. • Extend nature of interventions support to include a focus on psychosocial support to OVC. • Conduct an impact assessment study to assess the EOVC Project impact over a longer period of time with specific reference to the baseline study findings and to strengthen interventions to respond adequately to learner access, retention and performance needs. 4.3 THEME 3: SCHOOL-COMMUNITY RELATION AND PERCEIVED VALUE OF EOVC PROJECT The key questions on this theme are outlined in par 2.3 above 4.3.1 KEY ACHIEVEMENTS • The EOVC Project appears to be highly valued by the KZNDOE. • The schools valued the additional resources (training for the teachers, food gardens, and conditional grants) received as part of the EOVC Project. Parents and/or caregivers and learners appreciated the uniforms and food provided through the EOVC Project, as well as the jamborees where they could have access to various government services. • A more involved relationship amongst parents, community and schools to support learning especially for OVC was achieved through jamboree and parent’s workshops conducted as part of the EOVC Project, and the informants hoped that these would continue. 4.3.2 CHALLENGES • The accountability chain at various levels in the EOVC Project was weak, in that it was only significant in relation to conditional grants. • While a more involved relationship among parents, community and schools to support OVC was achieved, its sustainability was questionable considering the costs involved in organizing them. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 39 4.3.3 RECOMMENDATION • Create context-specific mechanisms where school-community relations can be strengthened and sustained in a cost-effective manner. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 40 5. OVERALL CONCLUSIONS The following were the USAID specified evaluation questions and responses: Theme 1: What processes were followed in the implementation of the EOVC Project? How was the EOVC Project implemented at province, district, circuit/ward, and school level? What worked and what did not work during the implementation of the EOVC Project? What are some of the socio-cultural factors influencing – enhancing or inhibiting – the implementation of the EOVC Project? The EOVC Project 24 month timeframe required a clear implementation strategy to ensure optimal use of existing resources. Due to personnel challenges, the need to ensure KZNDOE involvement, technical difficulties with the baseline, mobilization took eight months. During this time, project management and implementation structures were strengthened and/or established at provincial, district, circuit/ ward. This was to ensure that activities are aligned with district priorities and avoid duplication. Further delays were experienced due to the World Cup hosted by South Africa (where the winter break was extended to five weeks) and the three week teacher strike, which reduced implementation time even further. The four month no-cost extension was granted only on the final contract day, so while it enabled further activities, this late extension had a negative effect on the project. Institutional capacity building activities took the form of training and ongoing support of the DBSTs as well as the District Training Team (DTT), ILSTs, teachers, and community care givers by MiET Africa’s Project Coordinators in each circuit. Training and learning support were provided primarily on five focus areas. These were: the advocacy and training of district officials, school managers, and members of the ILSTs; caregiver training on how to care for and support children’s education; a multi-sectoral collaboration with relevant government departments and integrated service delivery events and systems; supporting the establishment of learner support mechanisms in each project school focusing on literacy/language and numeracy/mathematics; and working with circuit and ward managers to increase caregiver involvement in the education of vulnerable children. The baseline study documented the numerous socio-economic and cultural issues encountered in education. It highlighted the diversity in the district. From this process, a limited number of socio￾economic interventions were designed and schools were allowed to choose from a 10 intervention menu. However, the project did not address child labor or other cultural inhibitions preventing learner retention. EOVC Project interventions at school level included training that focused on supporting the establishment of learner support mechanisms; providing conditional grants to address OVC needs (e.g. uniforms, educational materials), establishing food gardens; and providing support services to OVC through multi-sectoral collaboration with relevant government departments. The stakeholders were very complementary about the project management structures and there was evidence that some of these structures would continue to function. All schools in the sample showed at least minimal evidence of implementation, especially of food gardens, participating in jamborees, and using the conditional grant to support OVC. Primary schools participated mainly in reading competitions which are being sustained. Implementation was noticeably strongest at provincial/district level where there was evidence of “owning” the project. However, ownership was much weaker at school level. Teachers and caregivers expressed appreciation for the training but the training lacked follow-up, attendance was inconsistent, and therefore it was too early to see evidence of other learning outcomes although there was limited evidence that foundation phase children were reading better. Generally, stakeholders strongly felt that the limited timeline and implementation delays prevented full project maturation. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 41 Theme 2: Did schools use the pilot interventions implemented under the EOVC Project in a way that helps reduce learning barriers amongst learners especially OVC learners? If so, in what way? If not explore the challenges faced during the continuation of the EOVC Project. Did the targeted learners benefit? If they did, how – outline the key areas in which the EOVC Project benefited target learners. There is general consensus among informants that the project interventions contributed towards meeting immediate material needs of learners, such as food and uniforms. However, there is still much room for improvement in terms of mitigating the social and cultural factors charactering the school-community contexts. All learners were considered vulnerable, and thus the Project did not address only OVC. Retention and access was reportedly improved by school uniforms; books, stationary; sanitary pads; food; and knowledge that learners and/or families could ask the schools for help. However, stigma was still an issue. During Jamborees, learners and their families received direct assistance from other governmental departments (e.g. birth certificates, ID documents, social development grants, seeds and tools, etc.). The main benefit was learner exposure to array of interventions were primarily material (food, clothing, etc.), but only three of the eight schools visited could show documentation on who received what services. There is not enough evidence to state that educational outcomes were improved. Theme 3: How is the EOVC Project perceived and valued? Was there a more involved relationship amongst parents, community and schools to support learning especially for OVC? The Project was highly valued by the KZN Province, but less so by the District, although still valued. Schools valued the additional resources (training, gardens, and conditional grants). Caregivers and/or parents and learners appreciated the uniforms and food provided, but accountability chain was weak, only real accountability was for conditional grants. The schools differ in terms of the progress they have made to build and strengthen the relationship with parents and community in supporting learning for OVC. The community mapping exercise has worked for a few schools and they have managed to forge relationships with existing community structures that have resources to support OVC. Such structures include local churches, community centers and other households in the form of food, clothes, and blankets. In one context, churches conducted house visits to provide such support twice weekly to OVC. So, though such a relationship has increased, it was not sustained, and there was still limited involvement. A more involved relationship has been achieved at most selected project schools, but their experience of this relationship appears to be in a continuum that ranges from poor involvement (which was also linked to poor support from SGBs) to a much stronger involvement. Jamborees, especially the benefit it made to community members, appear to have played an important role in encouraging this strong relationship. Theme 4: What can the KZNDOE take forward to scale-up the EOVC Project? First of all, it is important to note that KZNDOE is committed to implementing the IE policy and has participated in sustainability planning to carry on with project activities. However, it was clear that without an implementation agency (performing the joint role, played by MiET Africa and RTI International) it is unlikely to succeed. If replicated, it will be crucial to provide with more intensive training and support of ILST and link accountability for outcomes (particularly training results attainment). When implementing, it is crucial to differentiate between schools and their implementation ability and only provide resources and training to schools that are ready and accountable. KZNDOE needs to: Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 42  Clarify roles and responsibilities especially of the Full Service and Special Schools.  Strengthen sustainability plan (time, resources, M&E, reporting). The current Sustainability Plan of the district needs to show clear mechanism for resilience, evaluation, monitoring, and accountability, as well as reporting procedures.  Put in place at structures at provincial, district, and school levels to build on the multi-sectoral efforts to support OVC. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 43 APPENDIX A. EDUCATION PROGRAM PERFORMANCE EVALUATION STATEMENT OF WORK FOR THE EVALUATION OF THE EDUCATION SUPPORT TO ORPHANS AND VULNERABLE CHILDREN IN SOUTH AFRICA 2011 I. Summary This Statement of work sets forth guidelines for a performance evaluation of the USAID Southern Africa Education Support to Orphans and Vulnerable Children (OVC) in South Africa project implemented in KZN.  It is proposed that this evaluation take place no later than   August 1, 2011. The Education Support to OVC in South Africa project is implemented through RTI‐International from 30 April 2009 ending in August 2011 and support by USAID/Southern Africa.  The overarching goal of the project is to improve school performance and retention among vulnerable children and adolescents.  USAID/Southern Africa in collaboration with the KZN Department of Education (KZN DOE) is coordinating an education evaluation and design team to review the achievements of investments to date and to identify opportunities for future intervention in this sector.   This statement of work outlines the scope of work for the performance evaluation. The New USAID Evaluation Policy describes performance evaluations as evaluations that focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making.   Performance evaluations often incorporate before‐after comparisons, but generally lack a rigorously defined counterfactual.   The new evaluation policy can be accessed on this link: http://www.usaid.gov/evaluation. II. Background USAID/Southern Africa provided assistance for OVC in the Education sector in KZN since April 2009.   This project rests on the premise that: • although there is information on the barriers to education access among OVC, factors related to school performance among orphans and vulnerable children are not well documented;   • data related to barriers to learning among OVC will inform the development of targeted interventions to support learning among OVC and will add value to the Department of Education’s Inclusive Education Strategy and Schools as Inclusive Centres of Learning, Care and Support (SICLCS); and   • Embedding activities into the Inclusive Education strategy of the Department of Education will ensure success and sustainability and avoid redundancy. The project was designed to:   • identify the effect vulnerability among children has on learning and identify factors related to school performance among OVC and use this as the basis for dialogue and design of learning support interventions; Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 44 • provide different levels of assistance to identified schools in two circuits   in the Sisonke District of KZN to minimize barriers to learning; and   • Pilot intensive interventions in 42 schools across 2 circuits.   Key features of these interventions are their community orientation with   school‐wide and institution‐level application A baseline study was conducted in November 2009 through February 2010 to:  identify barriers and gateways to school performance among vulnerable children;    identify learner and guardian perceptions regarding the school;  identify perceived barriers to (and positive factors related to) school performance among vulnerable children as reported by learners, guardians and teachers; and  Establish a baseline on learner performance indicators.     Pilot Intervention Target Groups The pilot activities are being implemented in two school clusters namely in:  16 schools in Umzimkhulu Circuit where government’s Inclusive Education strategy (IE) had not been introduced when the project started and has not yet been implemented, and    26 schools in the Greater Kokstad ward where some elements of the IE strategy has been introduced.     In addition, 2 Full Service schools also form part of the project schools.  The program also provides broad level support on the IE strategy to the remaining 168 schools in the Umzimkhulu circuit.    III. Proposed Scope of Work Drawing on international literature and experience, the evaluation team will review existing documentation on USAID support for the Education Support to OVC in South Africa projects and meet with RTI, the Implementing agency, and government officials both at the national and local levels.    The evaluation team will review project documentation, to identify the critical decisions made during the life of the project, why these decisions were made and how the project’s original plans were adapted to address the reality of the situation and context on the ground.  This evaluation will emphasize but not necessarily be limited to key issues such as:  What processes were followed in the implementation of the EOVC activity?    What worked and what did not during the implementation of the EOVC activity?  Did schools use the pilot interventions implemented under the EOVC activity in a way that helps reduce learning barriers amongst learners especially OVC learners? If so in what ways? If not then explore the challenges faced in moving the program forward?  How was the project implemented at province, district, circuit/ward, school level?  How is the program perceived and valued?  Was there is a more involved relationship between parents/ community and schools to support learning especially for OVC?    Whether the interventions have had any positive influence on learner access and retention?  Did the targeted learners benefit? If they did, how – outline the key areas in which the program benefited target learners?  What can the provincial DOE take forward to scale‐up the program? Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 45  Other questions that are pertinent to the program design, management and operational decision making. It is acknowledged that the project goals of Improved Performance, Access and Retention are high level and long‐term aims that the project interventions cannot contribute towards within such a short life‐span.  Hence the project result and its outcome/output would not be measured by the project evaluation.  Accordingly, learner tests will not be conducted because the interventions have not had adequate time to take root and impact on learner performance. As such, the evaluation activity will also help to determine:‐    which interventions are being applied and how these are being applied; and  if they are having an impact on learners and, in particular, on the more vulnerable learners what can be learned from the pilot program for replication in other school in the area. IV. Team Composition The Evaluation Team will consist of two local education experts with experience in basic education and a strong evaluation background.  Combined, the team should have expertise in monitoring and evaluating large scale programs, and should fully understand OVC issues in the context the education system.  A representative from USAID and KZN DOE will join the team during field visits.    The Team Leader will be a consultant with extensive experience in education, children’s vulnerabilities with respect to HIV/AIDS, and orphan‐hood.  The Team Leader will hold meetings with the other core team members, key representatives from USAID/Southern Africa, and the KZN Department of Education staff prior to the commencement of the evaluation. The Team Leader will:  finalize the work plan for the assignment;  establish assignment roles, responsibilities, and tasks for the members of the team;  ensure that the logistics arrangements in the field are complete with assistance from RTI, and USAID Office;  facilitate the Team Planning meeting;  take the lead on preparing, coordinating team members’ input, submitting, revising, and finalizing the assignment report;  manage the process of writing the final report;  manage team coordination meetings in the field;  coordinate the workflow, team tasks and ensure that the team schedule works; and  ensure that the team field logistics are arranged. The other Education Expert should have an advanced degree in education, 5 years of experience in his or her specialties, and be local.    In addition, the team member should have, at minimum, the following skills and experience: 1. Demonstrated skill in written and oral communication. 2. Demonstrated knowledge of international education approaches, basic education strategies for education systems strengthening and promoting host‐country ownership of programs. 3. Have experience working in Southern Africa. 4. Work effectively in, and communicate with a diverse set of professionals. 5. Possess excellent English language skills (both written and verbal).    Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 46 V. LEVEL OF EFFORT (in # of days) Task Team Leader Team Member 2 Background Document Review 3 3 Team Planning Meeting 2 2 Meetings and Interviews with Key Stakeholders and Field Visits 15 15 Analysis and writing draft report 4 4 Debrief 1 1 Editing 5 0 Finalizes report for submission to USAID and KZN DOE 3 1 TOTAL 33 days 26 days VI. TIMELINE The entire review should be completed in five weeks.  This includes preparation days, and visits to the province, report writing and finalization.  The assessment will begin August 1, 2011 in Pretoria. VII. LOGISTICS The evaluation will be conducted over a 33+ day period with a start date of on/about August 1, 2011.  The evaluation team, in collaboration with the staff of RTI, KZN Department of Education and USAID/Southern Africa will arrange all meetings, interviews, site visits, in briefing and out briefing in advance.   USAID/Southern Africa and RTI will provide contact information for suggested interlocutors but in all other respects the evaluation team should be self‐sufficient. VIII. DELIVERABLES 1.   Pre‐trip Briefing: Prior to arrival, Team Leader and Evaluation team will review all relevant documentation and schedule a conference call or a face to face meeting with USAID/Southern Africa, and KZN Department of Education. 2.  Team Planning Meeting and Work plan:  A work plan will be developed during the team planning meeting and briefings with USAID/Southern Africa Program Office, Education Office, and the KZN Department of Education.  The work plan should include, but not be limited to, the following items: 1. Milestones and deliverables with due dates clearly established 2. Key interview questions that ensure quantifiable data, methods, and tools 3. Parameters for secondary analyses of existing data 4. Schedule of in‐briefing and formal debriefing presentations 5. Tentative schedule for informant interviews 6. Tentative schedule of travel to field sites Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 47 7. Timeline for drafting the assessment report, requesting feedback, and finalizing the final report 3.  Debriefing: The Evaluation Team will make a formal oral presentation to USAID/Southern Africa and the KZN Department of Education members in Pietermaritzburg.    4.  Draft Report: The Team Leader will submit a draft evaluation report to USAID/Southern Africa and the KZN Department of Education – as one hard copy and one electronic copy on a CD Rom or flash drive per recipient.   The report (not including attachments) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, Lessons Learned, Conclusions and Recommendations. 5.  Final Report: USAID/South Africa, KZN DOE and RTI have 15 working days to review the draft report and provide one single set of comments.  The Team Leader will submit the final report to the Program Office within one week of receiving comments from USAID/Southern Africa and the KZN Department of Education.    USAID/Southern Africa and KZN DOE requests both an electronic version of the final report (Microsoft Word 2003 or Word 2010 format) and 5 hard copies of the report.  The report will be released as a public document on the USAID Development Experience Clearinghouse.    IX. RELATIONSHIPS AND RESPONSIBILITIES (USAID and Consultants) USAID/Southern Africa will recruit and hire the two‐person Evaluation Team and will coordinate and manage the Evaluation Team.   The USAID/Southern Africa Program Office will provide overall technical leadership and direction for the Evaluation Team throughout the assignment and will undertake the following specific roles and responsibilities: Before Site Visit Work  Respond to all points included in the SOW, including the submission of the final report.     Consultant Conflict of Interest (COI).  To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors or NGOs evaluated/assessed and information regarding their affiliates.     Documents.  Identify and prioritize background materials for the consultants and provide them, preferably in electronic form.     Local Consultants.   Assist with identification of potential local consultants and provide contact information.     Site Visit Preparations.  Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in‐country travel and accurate estimation of country travel line items costs.     Lodgings and Travel.  Provide guidance on recommended secure hotels and methods of in‐ country travel (i.e., car rental companies and other means of transportation) and identify a person to assist with logistics (i.e., visa letters of invitation etc.) if appropriate.    During the evaluation  Mission Point of Contact.  Throughout the evaluation, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work.    Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 48  Meeting Space.   Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e.  USAID space, if available, or other known office/hotel meeting space).     Formal and Official Meetings.   Arrange key appointments with national and local government officials and accompany the team on these introductory interviews (especially important in high‐level meetings).     Other Meetings.  If appropriate, assist in identifying and helping to set up meetings with local professionals relevant to the assignment.  Facilitate Contact with Partners.  Introduce the Evaluation Team to implementing partners, local government officials, and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After In‐Country Work  Timely Reviews.   Provide timely review of draft/final reports and approval of the deliverables   X. MISSION CONTACT PEOPLE/PERSON Mission Contact for this assignment: USAID will provide a contact with the program office.   Final report shall be sent to this individual.    XI. FIRM FIXED PRICING Consultants shall provide firm fixed price which shall include all costs for performance of services, included, but not limited to,   1. Labor; 2. Administrative Costs; 3. Travel; 4. Accommodation;   5. Printing; 6. Communication; 7. Any other administration costs. This Firm Fixed Priced contract is payable on receipt of an approved final report. XII. REFERENCES (PROJECT DOCUMENTS) Reviewers will be provided with the following background documents in preparation for the assignment: Key Resource Documents: 1. Review the USAID Evaluation Policy. 2. RTI award document – plus related amendments. 3. Review existing EOVC activity reports as outlined below, review relevant Basic Education policies – e.g. White Paper 6 on Inclusive Education; and USG PEPFAR OVC Guidelines 4. The 2008 APS that funded the Education OVC project implementing partner, RTI   Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 49 5. The project quarterly and annual reports 6. The RTI PMP 7. Quarterly, Semi‐Annual, and Annual Progress Reports to USAID that summarize the achievements of the Education OVC project XIII. ADMINISTRATIVE AND LOGISTICS FUNCTIONS   The Program Office will make every effort to support the successful completion of assignments.   Consultants are expected to undertake the following tasks independently of Program Office assistance (unless otherwise stated in the scope of work): maintain individual calendars, set appointments, take notes, send emails, make phone calls, do photocopying, and other administrative functions necessary to implement the evaluation.   The Team leader is additionally responsible for maintaining the schedule and work plan for the team and for making logistical arrangements. The Program Office provides administrative and logistical support in the following specific areas:  Providing instruction in completing required forms (expense report, invoice, etc.)  Facilitating contact with USAID staff  Arranging for editing/layout of final report   All other tasks required to complete the scope of work will be done by the consultant, except where the scope of work designates specific tasks for USAID, or another organization.  Where other specific Program Office assistance is desired, consultants will make a request to the Evaluation Contracting Officer’s Technical Representative (COTR), who may provide staff if the request is deemed appropriate and staff is available.    XIV. Consultant Communication with USAID, RTI  and KZN The consultant/team leader reports to the Evaluation COTR or designee and is required to keep the COTR informed of any relevant updates, including deliverables, changes to schedule, and constraints/concerns in implementing the assignment.   If questions, problems, or concerns arise during the course of the assignment, the consultant will discuss those issues with the COTR or other USAID Education Officer, who will communicate them to USAID as appropriate.  If USAID requests the consultant to make any changes to the scope of work or undertake any activities that are outside of the scope of work, the consultant will ask the COTR or designee for authorization.  The COTR will ask periodically for updates on assignment status or a debriefing at the end of an assignment; the consultant will reply promptly and with the level of detail requested.  The consultant is required to submit a copy of all deliverables to the COTR, unless informed otherwise by USAID. XV. MINIMUM QUALIFICATIONS AND EVALUATION CRITERIA FOR PROPOSALS/QUOTATIONS USAID will evaluate proposals/quotations based upon the following criteria: 1. At least 5 years relevant experience performing evaluations for similar nature and scope; 2. Technical Skills/Capabilities; 3. Education/Professional Degrees;   4. Past Performance; and 5. Price. Please provide contact information for a minimum of three references which USAID may contact to obtain past performance information. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 50 APPENDIX B. Education Program Performance Evaluation of Support to Orphans and Vulnerable Children in KwaZulu-Natal, South Africa Work Plan for September - October 2011 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 51 INTRODUCTION Supported by USAID/Southern Africa, the Education Support to Orphans and Vulnerable Children in KwaZulu-Natal, South Africa project was implemented through RTI-International from 30 April 2009 until August 2011. The overarching goal of the project was to improve school performance and retention among vulnerable children and adolescents. A baseline study was conducted in November 2009 through February 2010 to inform the design of appropriate interventions. The pilot activities were implemented in two school clusters namely in: • 16 schools in Umzimkhulu Circuit within the Mkhatshana/ Mhlaba Ward where government’s Inclusive Education (IE) strategy had not been introduced when the project started and has not yet been implemented, and • 26 schools in the East Griqualand Circuit within the Greater Kokstad Ward where some elements of the IE strategy has been introduced. • In addition, there are two Full Service schools that form part of the project schools, while the program also provides broad level support on the Inclusive Education strategy to the remaining 168 schools in the Umzimkhulu circuit. The time has come for the program to be evaluated and USAID/Southern Africa is doing this in collaboration with the KwaZulu-Natal Department of Education (KZNDoE) in order to review the achievements of investments to date and to identify opportunities for future intervention in this sector. USAID/Southern Africa and the KZNDoE commissioned Khulisa Management Services Pty Ltd (Khulisa) to conduct a performance evaluation of the Education Support to Orphans and Vulnerable Children (EOVC) project implemented in Sisonke district. The primary purpose of this evaluation is to evaluate performance and ensure that project objectives were met through determining which interventions were applied and how they were applied. It will also provide some commentary on the impact that the Program had on learners and, in particular, on the more vulnerable learners. The evaluation will also identify lessons that can be learned by all stakeholders involved from the pilot program for possible replication in other schools in the area. This document outlines the Work Plan for the performance evaluation and it constitutes a key early deliverable as set out in the Scope of Work for the Education Program Performance Evaluation of Support to Orphans and Vulnerable Children (OVC) in KwaZulu￾Natal. The Plan discusses in some detail the methodological framework that the team has adopted for the evaluation, and some underlying principles that informed the type of the approach adopted. This framework is articulated within the scope of the evaluation provided in the terms of reference and it also introduces the team complement and the competencies they bring to this evaluation. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 52 MILESTONES FOR EVALUATION DELIVERABLES Milestone Date due 1. Team Planning Meeting & Work plan 9 September 2011 2. Field visit schedule (14-23/9/2011) 9 September 2011 3. Analysis; drafting of report and submit Draft to USAID 12 October 2011 4. Debriefing to KZNDoE and Implementation Partners 14 October 2011 5. USAID submit comments to Khulisa 19 October 2011 6. Presentation of Evaluation Findings to USAID 24 October 2011 7. Submit of Final Evaluation Report and Annexures 25 October 2011 METHODOLOGY FOR THE EVALUATION Scope of the Evaluation The evaluation team have adopted a qualitative approach that combines participatory and real world evaluation techniques for this performance evaluation for a number of reasons. A qualitative inquiry will enable the evaluation team to catch a close-up reality of the EOVC project and lived experiences, thoughts, and feelings of people involved in it (Cohen, Manion, & Morrison, 2000). This will allows for an empirical inquiry into the intervention process within its real-time context using multiple sources of evidence for better triangulation (Nieuwenshuis, 2007b). This is done in order to catch the complexity of the situatedness of their behaviour; and to present and represent reality in order to reflect people’s experience (Cohen, Manion, & Morrison, 2007). The strength of a qualitative inquiry lies on its holistic treatment of the phenomenon that generate in-depth and detailed data from a wide range of data sources (Cohen et al., 2007). The participatory nature of this evaluation entails creating a platform where all key stakeholders that are involved in the project are able to shape the process. Real world technique is classified as an “asset” based methodology that focuses on the positive changes that have occurred according to stakeholders. This allows for a more non-deficit where evaluators would select the stories which encapsulate the most important findings. In using the Most Significant Change (MSC) technique questions will be asked less directly e.g. tell us a story which illustrates the three major project accomplishments; if you were to be the project manager and replicate the project what would you do differently? If you were using your own money, what would have changed? These stories are complemented by structured interviews - key informant interviews conducted with principals, teachers and the parents from the school governing bodies (SGBs), caregivers and community members. When conducting school visits, the evaluation team will endeavour to compare baseline data with current school data to the extent possible. To date, the evaluation team has requested the school data but RTI International has yet to provide it to the team. This may end up being a limitation to the evaluation, if the data cannot be compared. When conducting the school visits Khulisa will assess whether Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 53 there is evidence that the project was working at the schools. These objectively verifiable indicators include, but are not limited to: a) Posters provided by the project; b) Royal Netherlands sponsored binders including all the relevant inclusion policy documents; c) Acquisition of books and other necessities identified with the Grant allocation provided. Most importantly are the lessons that can be learnt from the particular case. It is for this reason that interview questions are articulated in such a way that they are open enough to allow for depth in informant responses that can be analysed thematically. However, as a performance evaluation, findings and recommendations cannot address the longitudinal and higher-level impact of the program. The objectives of this performance evaluation are to:  Identify the critical decisions made during the life of the project;  Determine why these decisions were made;  Establish how the project’s original plans were adapted to address the reality of the situation and context on the ground;  Determine what interventions are being applied and the manner in which they are applied;  Determine whether or not the interventions have had an impact on learners, particularly the orphans and vulnerable children; and  Identify lessons that can be learned from the pilot program and how it could be replicated to other schools. The Evaluation is guided by ten critical questions as outlined below:  What processes were followed in the implementation of the EOVC activity?  What worked and what did not during the implementation of the EOVC activity?  Did schools use the pilot interventions implemented under the EOVC activity in a way that helps reduce learning barriers amongst learners especially OVC learners? If so in what ways? If not then explore the challenges faced during the continuation of the program?  How was the project implemented at province, district, circuit/ward, school level?  How is the program perceived and valued?  Was there a more involved relationship amongst parents, community and schools to support learning especially for OVC?  Whether the interventions have had any positive influence on learner access and retention?  Did the targeted learners benefit? If they did, how – outline the key areas in which the program benefited target learners?  What can the provincial DOE take forward to scale-up the program?  What are some of the socio-cultural factors influencing –enhancing or inhibiting –the implementation of the Education Support to OVC? Evaluation Team Composition and Top Line Schedule Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 54 The evaluation team is composed of two local South African consultants of Khulisa (Mr. Sandile Mbokazi, Team Leader and Mrs. Elna Hirschfeld supported by Mrs. Jennifer Bisgard). Each member of the evaluation team has more than 10 years of experience in conducting research and evaluations using an array of techniques including qualitative methodologies, including participatory and real world approaches to evaluation as applied within in the education sector. The evaluation team will work together in conducting the fieldwork during the week of 12 – 23 September in KwaZulu-Natal. Sample of Schools Selected A total of eight schools were purposefully selected from the list of 42 schools included in the pilot project based on the following stratification criteria:  A total number of three and four schools respectively were selected from the two circuits Umzimkhulu and East Griqua Land Circuit;  A total number of three schools as well as four schools were selected from two wards within each of these circuits, Mkhatshana/ Mhlaba and Greater Kokstad; and  Representative selection of different categories of schools within these circuits e.g. primary; secondary; combined schools; full service schools. However, on recommendation of the Sisonke District Office, Vulekani School –a school for learners with special education needs located in the Umzimkhulu ward within the same named circuit was included in the sample to bring the total of schools included in the sample to eight. Schools within this stratification were randomly selected. In-depth interviews and observations will be conducted to understand the ways in which the Program has impacted on school-communities. Data Collection Strategy There are a number of key stakeholders that were involved in the implementation of the project that will be interviewed as a critical component of the evaluation process:  Activity Manager: USAID/South Africa: o (one informant)  KwaZulu-Natal Department of Education o Provincial level: one or two informants) o District level: (two or four informants) and o Circuit level (two informants),  Project Management: RTI International o (one informant) and  Implementing Partner: MIET Africa and JET Education Services o The implementing partners (two informants, one from each).  Project Intervention Sites: 8 Sample Schools o At each of the schools, interviews will be conducted with principals, teachers engaged in the ILST, Institutional Level Support Team, SGB parents including caregivers, and learners in order to evaluate the manner in which each of these categorised schools experience and is impacted by the EOVC. The conversations with these informants will be voice recorded and some photographs will be taken linked to observations made for better capturing of information in each school. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 55 Interviews with each category of these stakeholders will be conducted in order to strengthen answers to critical questions and to develop useful recommendations for the Department of Education to better respond to the educational needs of orphans and vulnerable children. Data Collection Techniques The following data collection techniques will be applied in the evaluation process. These are:  Document review: USAID/South Africa, RTI International and other relevant national and international literature.  Key informant interviews (KII): These will be conducted with a range of representative stakeholders, including members of staff from USAID/South Africa, RTI International, KZNDoE provincial and District Officials and implementation partners (MiET Africa and JET Education Services) in order to obtain specific information that would strengthen the evaluation strategy and tools.  Focus group interviews: These will be conducted with district, circuit and school level informants. The idea is to get informants to discuss their experience of the EOCV Program.  A short school profile instrument to collect basic statistical information at the selected schools will be administered. This will also be confirmed with the corroborating information that will be obtained from the district and circuit levels.  As it relates to engaging the learners, in addition to adhering to basic ethical requirement of obtaining formal consent from principals, creative ways of facilitating discussion with learners will be used. These include fun activities and appreciative inquiry.  Techniques such as MSC and or Appreciative Inquiry will also be used when engaging parents from the school governing bodies, caregivers and community members. In using MSC questions will be asked less directly.  The team will ensure that disruption to the process of learning and teaching is avoided and where it is unavoidable, it will be kept as minimal as possible.  Monitoring data need to be assessed for the eight schools to verify it from the source as this will indicate a degree of validity and quality of reporting. Triangulation, analysis and report drafting The qualitative data will be analysed and reported in conjunction with the survey data, in order to produce the evaluation report. The analysis will draw largely and triangulate information from a number of data sources, i.e. findings from quantitative data; interviews with all key informants and documental analysis. The methodology is designed to enable triangulation of findings from a range of sources within the scope of this performance evaluation. Findings from the evaluation process will be compiled into a comprehensive report. Analysis of all findings will be a joint team effort, as will be the writing of the evaluation report. To this end daily team meetings will be conducted. This will enable initial analysis as per the report structure. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 56 Evaluation Tool Data Collection Logical Framework and Analysis Plan Overarching research question Desired outcomes Specific research questions Data source 1. What processes were followed in the implementation of the Education Support to Orphans and Vulnerable Children (EOVC) activity? Enable improvement in performance [and retention] of OVC in target school￾communities What processes were intended? What roles and responsibilities were assigned and to whom? What processes were actually followed and how, why? What strategic decisions were taken to inform the process, by whom and why were they taken? Documents, USAID, TRI, JET, MiET, DoE, District, OTT, Principals 2. What worked and what did not during the implementation of the EOVC activity? Enable improvement in performance of OVC in target school￾communities What aspects of the Program worked well or were easily implemented? What conditions or dynamics were at play that determined the successes and/ or difficulties in implementation? To what extent was the intervention tailor made for the circuits/ schools? RTI, JET, MiET, OTT, Principals 3. Did schools use the pilot interventions implemented under the EOVC activity in a way that helps reduce learning barriers amongst learners especially OVC learners? If so in what ways? If not then explore the challenges faced during the continuation of the program? Increase number of OVC in the target area participating in learning support programs. What school-level interventions were planned? What interventions were carried out? How did the school respond to the interventions? What school-level decisions were taken, by whom, and how did they inform the implementation of the Program at school-level? Now that the project is closed, what initiatives are still being implemented? Why? OTT, Principals, Teachers, Parents, Learners, and community members 4. How was the project implemented at province, district, circuit/ward, school level? Enable improvement in performance [and retention] of OVC in target school￾communities What are the specific roles played by KZNDoE, District officials, Circuit/ Ward officials, and principals in the process? What activities were intended? What activities were actually done, and by whom? KZNDoE, MiET, JET, District Officials, OTT, and Principals Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 57 APPENDIX C. Table 3: Evidence-based and Consultative Project Interventions Designed56 Proposed Interventions Project Activity Doable Interventions designed and implemented Daily Weekly Monthly/ Termly Training of teachers to organize and manage structured reading and writing time for learners Train FP teachers in Reading + Writing. Train teachers in basic Numeracy and Maths. Read aloud time: 30 min Group, Guided and Silent Reading: 30 min. At least one page writing work Counting: 30 min Times table: 20 min Inter-Sen Phase Maths activities : 30 minutes Peer to peer learning through story telling; reading and games: 20 min Reading Log for each learner to record home￾reading Each child able to take reading material home to read to a family member Festival of Books and reading/writing competitions Children compete for best counter and win a badge for that term. Speech Festival - learners speak on topic for 2 min in their grade. Award ceremony for most improved and highest achiever. Recognize learners who help others to learn. Facilitating links between schools on peer writing /secret pals writing program Each class has a girls / boys club to help peers, do social work and report back to the class Training teachers in career focus days, researching careers, inviting professionals to schools, involving career guidance services and career dress-up. This activity was not developed as it forms part of Link Community Development’s Peer Education Program with Secondary Schools in the district. Facilitate access to mobile libraries. Training of teachers integrating reading for Obtain access to mobile Library where possible. Establish a relationship with See above. 56 Education Support to OVC in South Africa. Final Report (Draft). 31 August 2011. p.22. This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. USAID/Southern Africa and KZN Education Department. Education KwaZulu-Natal. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 58 Proposed Interventions Project Activity Doable Interventions designed and implemented Daily Weekly Monthly/ Termly pleasure into teaching program. ELITS. Promote school food gardens in partnership with local government, DOA. Training of teachers to integrate gardens into the curriculum. Establish school food gardens in the 42 project schools with the help of conditional grants and support from DoA. Integrate school garden into class and home work Maintain food gardens with community, learners and teachers. Advocacy on Reading competitions ILST training on Reading and Writing. Organize Reading, Poetry and Drama competitions between schools. Training of teachers to integrate drama, poetry and art into teaching programs. ILST training Literacy. Organize Reading, Poetry and Drama competitions between schools. Conduct community resource mapping to support for OVC. Service delivery days in nodal schools to help schools become an integral part of a community service delivery agent. Strong relationship with local municipalities helped schools to access essential services. Essential government departments joined for community service e.g. legal services, access to ID documents, birth certificates, social grants, health check￾ups, HCT, food garden advisory, provide uniforms and shoes; help access scholar transport facility etc. Consolidate resources and services information available in the Ward and provide this information to each school. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 59 Proposed Interventions Project Activity Doable Interventions designed and implemented Daily Weekly Monthly/ Termly Train and support schools to manage child-to-child care and support programs Teacher training in SIAS helped to record and maintain OVC Registers which identified learners needs and addressed them. Parents/ guardians help children with homework or other learning activity Each ILST develops special learning activities to help OVC. Research and pilot various models of after￾school homework support interventions, including at drop-in centres where appropriate. Advocacy and training to help parents/ caregivers help children learn at home. Parents/ guardians help children with homework or other learning activity Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 60 APPENDIX D. Project Partner Role and Relationship Figure 3: Project Partner Role and Relationship57 57 Education Support to OVC in South Africa Final Report (Draft) 31 August 2011 p 12 This publication was produced for review by the United States Agency for International Development It was prepared by RTI International USAID/Southern Africa and KZN Education Department Education KwaZulu-Natal Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 61 APPENDIX E. Debriefing Presentation to USAID Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 62 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 63 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 64 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 65 Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 66 APPENDIX F. Matrix: Comments on Draft Report Addressed Institution Section of Report and Comment Description Response & Location in Application Section 1: Introduction COP The project budget was $2,399,016 + cost share of $596,000 or a minimum of 10% of funded amount. Refer to Section 1: Introduction: Done Par 1.1 Background USAID Amend based on COTR comments re APS. Refer to par 1.1: Background: Done USAID: COTR OVC detail: Highlight data refers to the status of OVC at the time of conceptualisation and does not reflect current status of OVC. Refer to par 1.1: Background: Done RTI Description of project goal not correctly formulated. Refer to par 1.1: Background: Done Par 1.2 Project Theory of Change USAID Insert theory of change here or move paragraph to section where the theory of change is discussed. Refer to par 1.2: Theory of Change: Done Par 1.3 Project Components COP Clarify par re main assumption identified as underlying the EOVC Project. Refer to Par 1.3: Par removed following further clarification received from USAID Par 1.4 Project Implementing Partners USAID Reference to counterparts. Refer to Par 1.4: Project Implementing Partners: Corrected Par 2.3 Priority Evaluation Questions and Themes Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 67 USAID Group questions according to theme categories identified and referred to in report findings. Refer to Par 2.3: Priority Evaluation Questions and Themes: Done Par 2.4 Evaluation Scope of Work USAID Remove reference to additional debriefing session requested and initial no cost extension provided. Refer to Par 2.4: Reference Period of Performance Evaluation: Done Par 2.5.2 Evaluation Limitations USAID Rephrasing of small sample size as a limitation. Refer to Par 2.5.2: Done RTI Clarify the use or not use of the available project baseline data. Refer to Par 2.5.2: Done USAID Pretesting of research tool. Refer to Par 2.5.2: Done Section 3: Evaluation Analysis and Findings USAID List the broad themes/categories in which the questions were grouped. Refer to Par 2.3: Priority Evaluation Questions and Themes: Done Par 3.1.1 Project Conceptualization USAID Amend based on COTR comments provided. Refer to par 3.1.1: Project Conceptualization: Done Par 3.1.2 Project Strategy USAID Amend based on COTR comments provided. Refer to par 3.1.2: Project Strategy: Done Par 3.1.3 Project Implementation Plan USAID Clarify negative impact on uptake of some learning support interventions; change in classroom practices; and especially activities planned with the secondary schools. Refer to par 3.1.3: Project Implementation Plan: Done Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 68 Project Partners USAID Diagram outlining the nature of interwoven implementation and management structure. Refer to par 3.1.5: Collaboration and Planning: Done Par 3.1.4 Management and Operational Structures USAID Include ILST poster / organogram Refer to par 3.1.4: Management and Operational Structures: Done USAID Clarify reference to Xoloxolo and Nombewu. Refer to par 3.1.4: Management and Operational Structures: Done Par 3.1.7. Collaboration and Planning RTI The Peer Education program was also funded by USAID and not KZNDOE. Refer to par 3.1.7: Collaboration and Planning: Done RTI Specify difference between Intervention Design Workshop and intervention design reflection meeting. Refer to par 3.1.7: Collaboration and Planning: Done Socio-Cultural and Contextual Factors of EOVC Implementation USAID Refer to comments received from the Sisonke District meeting on 14 October 2011. Refer to par 3.2.1: Social Factors Influencing EOVC Implementation: Done Some Benefits of the EOVC Project RTI It is a difficult to say much that is meaningful regarding performance without pre-post testing and a comparison group; especially given the number of challenges that the education sector faced in this time period. With the constraints of time, qualitative inquiry (versus real learner testing, even a shortened version of learner testing), and no comparison group (e.g., either a pre￾post desi gn as ori ginally planned and bud geted for or control Refer to par 3.2.3: Impact on Learner Access, Retention and Performance: Noted and indicated as such Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 69 schools) there is little that can be said with any validity about true performance and attribution. Teacher report results are in general high estimates of the real situation - when it comes to learner performance (and teacher performance for that matter) and may in some ways reflect the teachers’ expectations which are sometimes lower than what we would expect given age expectations in early grade learning. Classroom Management and Learning Support RTI Clarify reference made to additional funding received from KZNDOE. Refer to par 3.2.2.4: Classroom Management, Teaching and Learning Support Retention: The Experience of Learner Dropouts at EOVC Project Schools RTI Clarify reference to medical aid provision. Removed in the current versions of the document Conclusions: The schools have fairly achieved the goal to improve learner access and retention through this Program. The small sample that was rated by their class teacher shows that the majority of both boys and girls are performing on average in Numeracy, Reading in Mother Tongue Language, Reading in the Language of Instruction, and Writing. The performance of girls and boys was fairly even in all four areas. In terms of learner retention, schools varied in the way in which they experienced dropout in the past eight to nine months. Reasons for such variation included, among others, teenage pregnancy and poor parental support to their children’s education. In the absence of valid pre-and post- learner testing and/or a comparison group it is difficult to have attribution relative to performance. Teachers tend to rate high, but it could still be that a comparison group would have noted positive shifts in some schools based on the learner interventions. Noted and clarity is provided in par 3.2.3 The facilitation of the Program in Program schools was largely shaped by social-cultural factors, geographical location, size of learner enrolment, and the availability of infrastructural resources at their disposal. Thus the uniform Noted and addressed in par 3.2.1: Social Factors Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 70 approach to implementing the EOVC Project and relying on the ability of schools to establish their own innovative plans and mechanism to facilitate OVC support was not always effective. I don’t quite understand this and do not necessarily agree. Schools did difer in the effectiveness of implementing interventions and one of the intended outcomes of the evaluation (as originally planned) was to learn about some of the factors that different between schools with regard to their ability to implement learner support interventions. This would be extremely useful information for the MoBE KZN as they continue to adapt learner support under the Inclusive Education Project. Influencing EOVC Implementation There appears to be tensions inherent to this Project and these related to seeing schools both as unique, facing unique challenges, and handling them as homogenous entities in the process of implementation. Other tension related to funding the Project through an emergency fund that had very limited development focus. I have some trouble with this as the schools, indeed, developed and implemented their OWN SCHOOL-DEFINED learner interventions. This is really important as it is the essence and uniqueness of the approach and its success. Noted Unique school driven interventions Limited presentation/review of the unique school-driven learner interventions. One of the unique features of the design was to provide a more scientific approach to guiding interventions and at the same time to empower schools to learn from the initial assessment information and to develop (with guidance and support) interventions to support learning - according to their own needs and demands. This was considered and articulated as one of the underpinning strengths of the project per the reviews from beneficiaries and partners presented at the final project dissemination and awards ceremony. Noted In the conclusions section it was noted that the interventions were more or less universal or “one-size-fits-all”. Some of the training was indeed standardized across the schools. This was; however, based on one of two things: 1) universal demand; or 2) needs on the part of the MoBE KZN Inclusive Education Strategy. On the other hand, and most important, was that the schools all came to gether to learn about the outcomes of the Noted Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 71 assessment and developed school-driven learner support programs that were “data driven.” The important element of this was the ownership and pride in attending to specific needs of learners. and that teachers Challenges and Lessons Learnt: Please explain how this section should be read. Are USAID, KZNDOE, Implementing partners etc the respondents or responsible for the challenges listed. These are presented in conjunction with the Project’s achievements, and recommendations in Section 4 Strategy: Implementing Partners & KZNDOE: Lack of set standards/norms for the implementation of interventions makes it difficult to evaluate impact over time. Please explain this statement. Collaboration & Joint Planning: USAID & KZNDOE: Limited advantage taken on envisaged collaboration between the EOVC project and the LING project implemented in the same district. Did you mean Link? Please explain this statement. Refer to par 4.1.2 Theme 1: Project Implementation General Comments Report to be reviewed and revised to improve flow; standardize language reference; formatting including referencing of sources Done Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 72 “Provided a seed that was planted in the District which must and will be looked after carefully” Mr. Mdletshe, Sisonke District Director APPENDIX G. Draft Information Dissemination Seminar Report The purpose of the Education Support to OVC in South Africa Concluding Seminar was to provide an opportunity for RTI International and the full set of partners who worked together on this Project to share the highlights, lessons and outcomes of the project, to consolidate the recommendations for a way forward and sustaining the work within the Department of Education. Another key objective of the Seminar was for RTI International to publicly recognize the accomplishments of the pilot schools in providing school-directed learning support programs orphans and vulnerable children. The opening remarks of Mr. Mdletshe, Sisonke District Director, underscored the true ownership that the Provincial Department of Education and Sisonke District had in the Education Support for OVC in South Africa Project. He stated that the project provided “a seed that was planted in the District which must and will be looked after carefully. Highlights according to KZN Department of Education Officials and Educators Highlights of the project were provided in the opening remarks from District and school officials: Mr. Sikhulu Mkula, East Griqualand Circuit Manager; Mr. Vincent Dlamini, Xolo Xolo Full Service School; Mrs. Victoria Sibutha, Nombewu Full Service School; and Mr. Mncedisi Dlamini, Vulekani Special School. These are summarized below. Literacy and numeracy workshops and parent workshops were all considered a key to reducing barriers to learning among vulnerable children. According to Mr. Vincent Dlamini, Xolo Xolo Full Service School, as a result of these workshops teachers “became better practitioners,” and parents “changed and now play a more active role in the learning of their children.” Mr. Dlamini explained that in Xolo Xolo approximately 90% of the learners are vulnerable and thus every opportunity to support the vulnerable learner was critical. The literacy and numeracy workshops also helped to mobilize additional teaching resources and from conditional school grants, to purchase supplemental reading materials for the schools. Mr. Dlamini also mentioned that the workshops on finance and budgeting were extremely helpful to the School Management Committees (SMCs), especially in managing the conditional grants in support of learning initiatives. With the conditional grant, the schools were empowered to collectively decide on how to best support the learning initiatives. Though the conditional grants did provide some of the more conventional support to buy school uniforms or to provide feedings for needy children and seeds for the garden, for the first time the schools used the grant funds to support learning directly, including: supporting after school learning support; reading competitions; providing supplementary materials supplemental learning materials. All opening presenters stressed the value of the project in its support for school gardens. In Mr. Mkula’s words, “The food gardens have made an incredible impact on the lives of vulnerable children as gardens provide these children and their families with produce.” Mr. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 73 Project hosted Jamborees “mobilized true cross-sector collaboration to support the school with cooperation and contributions from the Departments of Social Development, Local Government, and the South Africa Police as well as the Basic Education” Mr. Vincent Dlamini, Xolo Xolo Full Mncedisi Dlamini from Vulekani Special School reported that the school had – on their own initiative –expanded the school garden activity that was “seeded” by the Education Support to OVC Project to provide a micro business where the produce were sold as income generation for the school. The garden was also leveraged as an opportunity for teaching and learning. Mr. Dlamini shared a heartfelt story of how the food garden helped to save a life. There was one young girl at the school who was borderline suicidal and the school grant, income generated from the garden and the garden vegetables themselves helped to provide shoes, uniform and food for the child to take home. As a result her self-esteem was strengthened and this life “was saved.” The school garden activity also helped to mobilize additional inputs from the Department of Agriculture who provided seeds, information and school visits/training in support of the activity. The Jamborees were also reported as an important “unexpected positive outcome” of the Project. Mr. Vincent Dlamini and others informed that these activities mobilized true cross￾sector collaboration to support school, including inputs from the Department of Social Development, Local Government, and the South Africa Police. According to Mr. Dlamini the project, and especially the Jamborees, helped to mobilize new partnerships between stakeholders and education such as partnerships between youth groups and schools; health NPOs and schools. Finally, Mrs. Victoria Sibutha and others underscored the importance of the orientation training on the Province’s Inclusive Education Strategy. The assistance to the Institutional Learning Support Teams (ILST) helped the newly formed ILSTs consolidate and manage support for vulnerable learners, which was noted as invaluable. According to Mr. Mncedisi Dlamini from Vulekani Special School, “the management team is very strong now and directly gives support to the school to ensure that vulnerable learners receive the attention they need.” The opening remarks from the Department of Education partners were followed by brief summary of the project activities given by Ms. Saeeda Anis-Prew and Mr. Chris Ramdas. Project Review: Saeeda Anis-Prew, Chief of Party and Chris Ramdas, MiET Ms. Prew shared the overall project objectives, design and documented outcomes. The 28 month 28 month project USAID funded project (April 2009 to August 2011) worked toward three overall objections, to: 1. Reduce barriers to learning among OVC 2. Contribute to the current knowledge about real learning challenges facing OVC 3. Support and add value to government’s Inclusive Education strategy in practical and cost-effective ways The following highlights the key elements of implementation, which, at the core was: to work with and through the KZN Department of Education and District officials – as partners; and 2) to design interventions that were informed by both rigorous study of the barriers and catalysts to learning; and 3) to implement interventions within the context of the KZN Inclusive Education strategy. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 74 The following outlines the outcomes of the project within the context of the drivers for change. What we found! What we did! Average scores among 3rd graders tested was 30% for language Average scores among 3rd graders tested was 30% for language Average score of 10th graders tested was 31% for language Average score of 10th graders tested was 31% for language Learners who read aloud at home every day tended to have higher scores. 65% of responding learners had no books to read at home. Learners who read aloud at home every day tended to have higher scores. 65% of responding learners had no books to read at home. Learners performed best in literal comprehension and worst in writing. Learners performed best in literal comprehension and worst in writing. Average numeracy score for 3 rd graders is 24%. Trained teachers on how to teach basic numeracy and Maths. Supplied working kits to each participant. 10th graders scored an average of 25% for maths. Emphasis on basic numeracy exercises daily in class. Learners performed worst on questions requiring application. Promoted Maths competitions. 54% of the least vulnerable and only 1.4% of the most vulnerable said that their parents helped them with homework. Continuous advocacy with care-givers stressing importance of helping children to learn at home. Average numeracy score for 3 rd graders is 24%. Trained teachers on how to teach basic numeracy and Maths. Supplied working kits to each participant. 10th graders scored an average of 25% for maths. Trained teachers on how to teach basic numeracy and Maths. Supplied working kits to each participant. School practices especially pertaining to uniforms, late arrivals, and late fee payment were not aligned with policy. 130 District Officials and over 1000 school managers/ teachers trained on the Inclusive Education Policy. School practices especially pertaining to uniforms, late arrivals, and late fee payment were not aligned with policy. Correct application of key policies stressed to support OVC at school and District levels. Teachers felt that individual lessons and academic competitions, helped OVC performance. Organized academic competitions. Teachers felt that provision of uniforms, food, stationery and extramural activities helped OVC in class. Established and enhanced school food gardens to supplement school nutrition. Parental involvement was lacking but many parents are willing to help at school. Trained primary care-givers on the importance of family support in children’s education. Some schools solicited community help for after-school homework support. Held service delivery events at schools to provide essential services to OVC and vulnerable households through inter-sector collaboration. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 75 The overall accomplishments are provided in the table below. Accomplishments – Over 17000 children across 42 schools benefit from the project. 2011 Targets Identified and reached nearly 8000 OVC (half are girls) with a range of support including learning support, nutrition, legal, social and psycho-social care. 1000 40 project schools established productive food gardens. 42 Trained 150 District and Circuit Based Support Teams (DBST+CBST) on Inclusive Education strategy and school support 20 Trained and supported 1200 Principals, teachers and members of the Institutional Level Support Teams (ILSTs) on Inclusive Education Strategy, academic training in literacy, numeracy and maths as well as in the management of conditional grants. 100 Trained 130 SGB members in providing care and support to OVC. 100 Reached 1500 parents/ caregivers on providing care and learning support to OVC at home. 50 Facilitated productive relationships amongst essential government departments including Social Development, Health, Home Affairs, Agriculture and the local municipality through 5 Service Delivery Days/ Jamborees held at project schools. 4 Chris Ramdas discussed the role of our MiET Africa partner, as presented in the schematic below: Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 76 Highlights of the MiET Africa experience summarized by Mr. Ramdas included: • Relationships – established (RTI, JET, DoE, Schools, Communities, multi-sectoral partners) • Learning opportunities – as we explored new ground • Rich experiences that have emerged from our involvement • Witness the development of people – our team and partners within the District • Satisfaction in the positive first steps taken to mobilize learning support toOVC • Witnessing the change • Enthusiasm of “champions” within the District • Leadership shown by the District Manager • The role of the managing agent RTI, professionalism of staff of RTI, in particular Saeeda Prew Mr. Ramdas shared a relevant and pointed question about a young Standard 3 reader who was required to rehearse a poem for class. He chose a poem that was well above his reading level and insisted on it in spite of the challenge. Dr. Ramdas ended his discussion with the following poem, and a title “Fulfilling” our roles: Strategy review and lessons learned by Dr. Jonathan Godden Dr. Jonathan Godden conducted a strategic review and study of lessons learned though desk research and interviews with school and district beneficiaries. He studied the outcomes of the project within the context of its goals and assumptions. The following summarizes Dr. Godden’s presentation. Goal Improved school performance, access and retention among vulnerable children and adolescents Assumptions  Factors relating to school performance of OVC not well-documented  Data on barriers/catalysts to learning among OVC would inform the targeted interventions I must go down to the seas again, to the lonely sea and the sky, And all I ask is a tall ship and a star to steer her by, And the wheel's kick and the wind's song and the white sail's shaking, And a grey mist on the sea's face, and a grey dawn breaking. I must go down to the seas again, for the call of the running tide Is a wild call and a clear call that may not be denied; And all I ask is a windy day with the white clouds flying, And the flung spray and the blown spume, and the sea-gulls crying. I must go down to the seas again, to the vagrant gypsy life, To the gull's way and the whale's way, where the wind's like a whetted knife; And all I ask is a merry yarn from a laughing fellow-rover, And quiet sleep and a sweet dream when the long trick's over. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 77  Learning support interventions will add value to KZNDOE’s IE strategy and SICLCS  Embedding activities in IE strategy will ensure success & sustainability, avoiding redundancy. Lessons Learned from Dr. Godden’s review  Project Design and Time Frame  A simple project management structure with limited number of partners may be preferable  May be better to exclude the M&E agency from partnership and contract externally  Collaborative design of interventions lead to high levels of acceptance  Project inputs, including TA and time, must be commensurate with the changes anticipated in goals, objectives and targets  Baseline Study  Evidence-based approach of the baseline study appreciated, and well worth the delay  Interviewing parents/caregivers laid strong basis for community acceptance  District  Participation of relevant Ward Managers from the outset was critical  Weaknesses in the training approaches of District Curriculum Advisory Services were exposed  Challenge will be to sustain the high-quality training to which teachers have been exposed  Full Service schools will require much more resources, including professional staff  Inter-departmental collaboration is possible and can be both beneficial and stimulating  School and Community  Schools are not helpless in addressing the effects of poverty and vulnerability  Generating significant improvements in learner performance is not as impossible as it seemed before  Limited exposure to high-quality training and practical ideas can be multiplied internally in a school  Find simple and practical ways to make sense of complex or vague policy directives  What may first seem like just more unnecessary work, could turn out to be fun and make your work more enjoyable  Education Policy  Often a vast gap between policy adoption/ declaration and actual implementation at school level  Isolated rural schools are more challenged in implementing new policies than their urban counterparts  Too many policy changes/directives are dependent on the same intermediaries (ward managers &curriculum advisors)  Non-governmental partners can be valuable in bolstering mediating capacity  Policy implementation always has a cost, and should be budgeted for  Inter-departmental Cooperation  Inter-departmental collaboration is feasible around concrete focused issues in which all participants have a shared and equal stake.  Practical collaboration is easier to arrange at a more localised level  The project initiated collaboration around meeting the diverse needs of OVC met both of these requirements Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 78  The jamborees provided opportunities for representatives of diverse agencies/departments to showcase their work and earn esteem  Successful collaborations can become self-sustaining and develop beyond the initial purpose Sustainability plan presented by Mr. Sipho Mdletshe, Sisonke District Director Mr. Sipho Mdletshe presented a way forward in sustaining and expanding the activities of the project. The principle goals of the Sisonke Sustainability Plan were: 1. Increase number of OVC participating in learning support programs 2. Improve school performance among vulnerable children and adolescents The principle strategy presented for sustaining the project good works was to empower and advance the existing school support structures of the District and the newly established and strengthened Institutional Learning Support Teams (ILSTs) at each school. According to Mr. Mdletshe, an important goal will be to establish fully functional ILSTs in all schools. A second strategy was to further build school-community-parent relationships and to further support and facilitate parent involvement in their children’s learning. This sustainability plan presented by the Mr. Mdletshe outlined the specific activities that will be undertaken to continue the intervention activities with the aim of achieving maximum benefit for the target population, namely vulnerable children so that overall learner performance will improve. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 79 A SUSTAINABILITY PLAN FOR THE ESO-SISONKE PROJECT Goal: Improved school performance, access and retention among vulnerable children and adolescents Objective Activities Respon￾sibility Target group Time frame Milestones Indicators Resources Budget Increase number of OVC in the target area participating in learning support programs Train schools to identify vulnerable children and devise strategies to address vulnerabilities (Training in the I E Manual) SDSS, DTT, ILSTs, SMTs and teachers Ongoing All pilot schools have fully functional ILST’s, all teachers trained in the IE Manual except Pholela Schools Increased no. of OVC identified, referred and assisted. I E Manual, ILST Registers & Statistics R60 000 Establish and support structures such as ILST, CBST, DBST to function effectively and strengthen referral procedures SDSS, EHRD & G&M ILSTs and SMTs of all schools & Pilot schools End of March 2012 ILSTs at schools and CBST at Circuit, DBST at district levels established (including Sch. based L/Counselors & Peer Educators Structures established and names of committees displayed; regular meetings held and minutes kept. L/C, P.E. and I E Manuals, wall chart, policy documents, Minute books Hold service delivery events in collaboration with Flagship Programme DTT SDSS, and Schools OVC and their parents & caregivers Ongoing At least one Service Delivery Event held by end f the year, and one every six months OVC and their families accessing services from and supported by the relevant depts. & NGO’s Flagship Programme, stakeholder’s meetings & database Classroom Support Training for Educators GET, FET, SDSS, LSE’s ILSTs and SMTs of pilot schools Ongoing All schools at NMZ & the pilot schools have been trained. Educators will be able handle learners with barriers & keep records Classroom Support Manuals R47 000 (I.E. Budget) Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 80 A SUSTAINABILITY PLAN FOR THE ESO-SISONKE PROJECT (Continued) Objective Activities Respon￾sibility Target group Time frame Milestones Indicators Resources Budget Enable improvement in performance of vulnerable children and adolescents in the target schools and community Hold Advocacy to mobilize youth against teenage pregnancy, Drug Abuse & HIV/AIDS as per My Life, My Future programme SDSS, GET & FET (Lifeskills & L.O. Advisors) Learners, Educators, Parents & Community members Ongoing ML,MF has been launched, some schools have been trained in Peer Educ. Programme and some have Lay Counselors ML,MF will be launched in schools, Peer Educ and Lay Counselling will be provided ML,MF document, Peer Educ and Lay Counseling manuals. Train parents and Caregivers to offer learning support to their children. G&M, Counselors & Social work parents and Care givers in all schools Ongoing Parents and Caregivers trained in supporting their children. Learners present completed homework regularly; improved quality of learner work (reading & numeracy). Parents and Caregivers will be able to build better relationship with their children. Parental & Caregiver materials. R50 000 Train, monitor and support teachers in literacy & numeracy TLS & GET Teachers in all schools March 2012 All primary school teachers trained in content & methodology – literacy & numeracy Teachers using new methods and materials; learners performing better in Literacy & Numeracy Foundations for learning, other materials used in the pilot schools Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 81 USAID Address – Mathata Madibane Ms. Madibane underscored spoke to the passion of the Department and USAID that was a driving force behind the success of the program, and recognized the important support provided through the Presidential Emergency Plan for Aids Relief (PEPFAR). Ms. Madibane explained that USAID recognized the importance for vulnerable children to stay in school, thus complimenting the set of OVC support initiatives funded by PEPFAR. Recognizing a lot of vulnerable learners were struggling, it was decided that that a component to support learning, specifically, was critical. All of USAID’s programs are aligned with South African Government Priorities and this program, as discussed throughout this topic, is aligned with Kwa-Zulu Natal Province’s Inclusive Education Strategy. Ms. Madibane spoke passionately about how Sisonke District and the province have helped USAID to get where they are today with this program – impressing on the audience the importance that the District took ownership and reinforced USAID’s understanding about how significant the development of this partnership is. Ms. Madibane indicated that she has collected all the input from the seminar to and will be taking it back to USAID. She indicated that she would use this information to work with the department to design a program addressing these issues and stated that “We will be able to achieve what we want to achieve”. She reported that with all levels of government engaged in a multi-sectoral approach – not only the health flag is flown, as education is also a key priority. This is important considering the broad focus of foreign assistance on health. She underscored the importance of education, even in health, “If people are better educated, they lead a healthier lifestyle.” Key Note Address – Dr. H.P. Gumede (KZN DOE) Dr. Gumede from KZN Department of Education greeted the audience and asked everyone to give themselves a round of applause, expressing the great appreciation for support and hard work from all. D. Gumede expressed gratitude to all those who gave of themselves selflessly by going out to the rural areas to attend to the needs of our disadvantaged learners, quoting Che Guevaro - “a revolutionary is someone who has a love for his people in his heart” – and stated that these are the qualities needed to go out and make sure lives are improved. Dr. Gumede stated, on behalf of the Department, that he believes that learners benefitted immensely from all aspects of the project. As he sat and listened to all the presentations of the morning and the different aspects of the project were highlighted, the benefit to the learners could not be denied. In following up, the department will need to sit down and look at the recommendations, to prioritize, move forward to implement those that can be implemented relatively easily and to conduct an in-depth analysis for the set of recommendations and consider their respective implications in supporting vulnerable learners. Around these recommendations, Dr. Gumede was glad that some of the issues raised were ably addressed by Ms. Sifunda and Dr. Faith Khumalo – access, retention and learner performance. Ms. Sifunda also addressed how the Inclusive Education Strategy is going to ensure roll out encompasses all issues that were raised. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 82 Various pieces of legislation and policies which enforce the basic human right to education such as the South African Schools Act and Education White Paper 6. In KZN, this policy intention is being briefed through the strategy that schools are inclusive centres of learning, care and support. Dr. Gumede also discussed the relevance of this project under Action Plan 2014 which is the national plan to realise the objective or vision of schooling in 2025. It is the conviction of the department that by identifying barriers to learning and mitigating them early, the department will not only be able to increase access to basic education but also to deliver on the learning outcomes and targets as contained in the Action Plan 2014 towards the realisation of schooling in 2025. Dr. Gumede expressed that the budget that the department has been given is huge and demands an integrated approach, suggesting that most areas are best be tackled by collaboration within the department; collaboration between the Department and other government departments and between the Department and other sectors. According to Dr. Gumede, the KZN Department takes pride in the way they have managed to successfully work in an integrated fashion, when it comes to the delivery of the Inclusive Education Mandate. The Department realizes that when talking about inclusive education, they are talking about a real transformation of the education system, and in the end that cannot be done by a single section of the department, but is a responsibility that needs to be carried by everyone in the department as well as partners outside. The only way to do anything, at any level, requires people working together, and collaboration. Partnerships between the various directorates have been solidified, and so have partnerships between various government departments. Dr. Gumede expressed pride in the partnerships formed with other sectors outside of government, partnerships they have with civil society partners, funders and business. According to Dr. Gumede, with this collaboration they have managed to pilot the Inclusive Education Strategy and are presently at a point where they are field testing various aspect of the strategy; e.g., determining norms and standards and cost implications of establishing full service schools, special needs schools, resource centres, etc. In closing, Dr. Gumede expressed, “The partnership between DBE, NDoE, USAID, RTI, JET and MiET is what we are about here today. This partnership is intended at linking barriers to learning, to learning outcomes. The department was keen, through the OVC project, to establish the extent of correlation between barriers to learning, to learning outcomes or performance”….as shown from Dr. Jonathan Godden’s presentation. In addition, it was noted that the project enabled the department to fast track the delivery of some of the crucial elements towards the establishment of an inclusive education and training system. Though all the benefits cannot only be enumerated, Dr. Gumede was pleased that project schools that did not have food gardens now have it, leaners that did not have uniforms have had these purchased for them from funds set aside for this. This project has also played a significant role in supporting provincial government policy which is one home, one garden, one school, one garden. These 42 schools have demonstrated to us that it is possible for each school in the province to have its own garden because a school garden, a food garden, is not Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 83 there to decorate a school and make it look beautiful but it is there to produce food that will result in the promotion of the health of the learner so that their performance in the class can improve. Dr. Gumede expressed that the challenge going forward will be to ensure that work accomplished by the project is sustained. He pointed to the leadership in Sisonke District that will be the underpinning success in this and commended Mr. Mdletshe for his tireless work and commitment. He stated that the Department is also committed as the lessons learned and new good practices will be introduced as part of the Inclusive Education Strategy. Importantly, as part of our commitment, the Department will work toward ensuring that examples of good practice emerging from Sisonke will get exported to other 11 Districts in Sisonke. He proposed to all that, together, we should commit ourselves to ensuring that all schools: have a food garden - in line with provincial policy; a functional ILST; 30 minutes of silent reading per day. Dr. Gumede indicated that the Department “shall also intensify the relevant and appropriate empowerment of teachers” and is committed to “aggressively driving care and support strategies to support parental involvement. Dr. Gumede’s closing remarks of appreciation, “the department would like to extend its appreciation to USAID for taking the initiative and funding the Education Support to OVC in Sisonke. We hope that the will be there for the long haul, when we take the results to the street, and that they won’t leave us alone out in the cold, but will always be there next to us, as that has been demonstrated to work,” inviting a round of applause for RTI, MiET Africa, JET and the partners in KZN, DBE, Sisonke District Manager for his leadership, program managers, circuit managers, ward managers, principals, educators of participating schools, SGBs, parents and learners. Vote of Thanks – Mrs. Thobile Sifunda (KZN DOE) Ms. Sifunda opened by stating that at the beginning of the seminar she almost shed a tear because she couldn’t believe where they are now as a province. She then went on to say that she wanted to thank all the people and organisations that made KZN proud and made a difference, including: USAID Mathata Madibane who she described as “an ambassador of what care and support is all about.” According to Ms. Sifunda, Mathata comes and presents programs that USAID funds, with a very, very humble attitude and this is what is appreciated. “She really has had a listening ear and look what has happened with this project because of this.” RTI International Dr. Liz Randolph, Regional Director, thanked for her very humble and bottom-up approach to service delivery. Saeeda Prew – has become a sister in the delivery of service for our people. We have worked with you over the years and you have remained the same -very humble, very approachable, and Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 84 very soft spoken and have not changed. She makes a point and makes sure that she carries through whatever she laid at the table in the beginning to the finish, and we really appreciate. Thank you so much for everything you have done for the project, even the goodies that you brought here today – that took us by surprise as we did not anticipate you would have done this for us. The certificates, the goody bags – it’s much appreciated. It shows a motherly love that you have inside of you. Thank you so much. Michelle Layte and Viloshnee Reddy – for handling the logistics and working tirelessly, liaising with each and everyone one of us, making sure everyone was here today and comfortable, sorting out the program, transportation of the delegates, and all that you did for making this day a success – we really appreciate it. MiET Africa It’s very difficult to say exactly how thankful we are to have MiET on board, not just for Sisonke – they are a partner in inclusive education and we count on MiET to be with us for the long haul. They are a tried and tested partner, and have been with us through thick and thin. We say thank you to Lyn van der Elst, CEO, and her team, those who have made a huge difference in terms of thos project: Chris Ramdass, Vusi Mbube, Israel Mdladla, Ndeni Mjoli, Thuli Dlamini and Nonhle Dlamini. We also have to bid farewell to some of the MiET team as they will have to leave now that the project has ended. We wish you well in the endeavours you will pursue going forward. We are proud that MiET lives by their vision statement, because not only have they cared for OVC and many other children in the province, but they have actually employed someone from one of their projects. We appreciate that your actions speak louder than words. JET Godwin Khosa – thank you for guiding the process around the research Carla Pereira – the head of research – thank you so much for all the robust debate M&E, that was for the good of the project. Resource Organisations UKZN - Lecturer Elson Khambule Jonathan Godden – who presented and facilitated today SADTU Partner in education. Serve in the Provincial Coordinating Committee on Inclusive Education – highest structure in KZN. Has been with us through it all. Are visible and there to ensure the partnership thrives and projects deliver on their objectives. Department of Education, NDoE Dr. Faith Khumalo – made such good input today. She is at the helm of the Care and Support for Teaching and Learning Strategy. Thanks for her leadership. Dr. Moses Simelane – Director of Inclusive Education - leads by example. Thank you for providing direction and listening to us – we really appreciate that. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 85 Dr. Mbokazi – has been there to chair some very big and sensitive meetings, to give direction and the shoulder the whole implementation of inclusive education in the province. We really appreciate him, and even though he couldn’t attend today, he is here with us in spirit. Dr. H.P. Gumede – the “Father of Inclusive Education”. It started out with him in 2001. What you are seeing today is the result of the seeds he put out and we are proud that he is seeing the fruits of his toil. His leadership is much appreciated. In closing, “Thank you to the following individuals who have made a huge difference and who are drivers of the Inclusive Education Strategy: Kamsilla Naidoo; Dr. S.E. Chonco; Mam Zondi District District Manager – Mr. Mdletshe – very upfront and open. His team was always there 100%, and we look forward to going forward. Circuit Managers, Ward Managers, Program Managers and all other sections that were involved at Sisonke – thank you for making a huge difference. Principals of Participating Schools – without you none of this would have been possible and another big round of applause to you all. SGBs – have been a vibrant group Learners – beneficiaries Presenters Thank you to all the presenters – it takes guts to prepare and present to such an educated audience and you have done well. Councilors - thank you for the work you’ve already begun. The learner support educators are already making a difference. Thank you also to the municipalities, partners in government and all the various government departments that work with us. From Ms. Sifunda, “Thank you in advance to USAID, RTI and MiET for the roll out.” Award Ceremony Participating schools were provided with a certificate for schools and individual students for succeeding in the reading, poetry, and drama competitions and gift boxes for schools. School boxes included reference materials (Atlas, Bi-lingual Dictionary, Encyclopedia) and First Aid Kit. Final Program Performance Evaluation Report: Education Support to Orphans and Vulnerable Children, October 2011 86